In a recent webinar, DistillerSR customers Rosie Morland, AI and Data Science Specialist at Excerpta Medica and Remon van den Broek, Head of AI Strategy and Data Science at Adelphi Group were joined by Mark Priatel, VP of Software Development at DistillerSR. They shared their journey with AI Classifiers and how it significantly expedited and improved the accuracy of their systematic literature reviews.
Q: Can you share your experience with using classifiers in literature review projects, Remon?
A: About 10 years ago, I did a large systematic review with about 20,000 records. It was a challenge, and when we finally completed the project, the evidence we gathered ended up being out of date. We then started developing algorithms with IBM Watson, but moved to DistillerSR for its flexibility in training classifiers. This allowed us to adapt classifiers based on specific project needs.
Q: What was your experience, Rosie?
A: Our first project using DistillerSR and trained classifiers was delivered four months early. We screened about four times as many references. The process of training classifiers is integrated into the project, which is a key advantage. We’ve also developed over 100 classifiers, tracking them in Excel to understand their evolution and suitability for different questions.
Q: Remon, have you seen time savings with the implementation of classifiers?
A: Yes, there’s significant time savings. We created a live database pulling data from PubMed and Embase, which gets automatically classified. This allows for multiple projects to be integrated into the same database, saving a lot of setup time.
Q: Rosie, what has been your experience?
A: Absolutely, we’ve seen time savings and efficiency. Classifiers allow us to focus on specific questions and streamline the screening process significantly.
Q: How do you decide on and create classifiers for your projects, Rosie?
A: The classifiers are based on need and project requirements. We started with disease-agnostic classifiers and then moved to more bespoke ones for specific needs. We prioritize questions that are straightforward for both people and classifiers.
Q: What about you, Remon?
A: It depends on the project’s output and the questions we need to answer. Some classifiers need thousands of abstracts for training, while others need much less. It’s often a trial-and-error process to find what works best.
Q: What are your thoughts on Large Language Models (LLMs) compared to traditional classifiers, Remon?
A: There’s plenty of interest in LLMs like ChatGPT, but the reliability issues cannot be underestimated. LLMs are a black box and can’t provide the transparent explanations needed in systematic reviews. AI-enabled Classifiers, however, offer transparency and more reliable results.
Q: Your thoughts on the subject, Rosie?
A: LLMs are exciting but volatile for systematic reviews. Classifiers are much more reliable for specific data extraction tasks. LLMs can often produce overwhelming amounts of data that still need sifting.
Q: Any best practices or recommendations for those starting with classifiers, Remon?
A: Make sure the question for the classifier is simple and clear. Multiple reviewers should be on the same page regarding the question. It’s important to ensure that the classifier’s training is transparent and the output is trustworthy.
Q: Rosie, any recommendations to get started with classifiers?
A: Pay attention to the data used for training classifiers. Broader training sets can be beneficial, especially for disease-agnostic classifiers. It’s crucial to have clarity on what you want from the output.
Watch the recording of the webinar for all the insights, including an introduction to AI Classifiers by DistillerSR Machine Learning Senior Engineer, Rushdi Shams.
