The demand for evidence-based research is reaching a tipping point. Across almost all industries, the need for clear and concise evidence is starting to overwhelm the researchers whose job it is to find it. Systematic reviews (SRs) are still widely considered to be the best way to collect, analyze, and synthesize relevant research about a specific topic. Yet, today’s researcher needs a solution to complete their reviews faster, without sacrificing precision so that the validity of their reviews is not compromised.
For years, systematic review tool developers have been working on solutions to automate some of the more repetitive and mechanistic review tasks, thereby allowing researchers to focus on the science. Earlier on the blog, we looked at natural language processing, a subfield of artificial intelligence that has the power to benefit researchers by expediting certain SR tasks. This week we’re taking a closer look at automation in systematic reviews and covering the specific type of algorithm that makes it all work: the classifier.
What Exactly is a Classifier?
A classifier is a statistical model that uses NLP to process information and classify it accordingly. Classifier algorithms are used in many automated processes that we use daily such as spelling and grammar checkers, spam filters, speech recognition tools, and more.
There are several types of algorithms that are considered classifiers. They can range from simple linear classifiers that look at singular features and classifies them based on their presence or absence in a data set, to extraordinarily complicated and futuristic models like neural networks, which use a nonlinear function on input data to move information through multiple layers resulting in an output that answers a specific question.
How do Classifiers Work for Systematic Reviews?
Automation of systematic review tasks has long been of interest to the evidence-based research community. In particular, classifiers have been developed to sort and categorize information. The ability to classify references automatically is an incredibly valuable tool for researchers who are reviewing hundreds or sometimes thousands of references. Technological advances are increasing our ability to do this faster and with more precision.
For example, DistillerSR’s Artificial Intelligence System uses classifiers to make the review process faster for researchers. Specifically:
- Enables users to create custom classifiers from references they have already screened to be used across multiple projects
- Can be used as a second screener or a QC screener, and compare responses from human reviewers
- Can automatically answer form questions with a full audit trail
- Can test the precision and recall of classifiers to ensure accuracy
DistillerSR builds classifiers using training data. Essentially, the AI is fed references that have already been classified by humans and learns from past decisions to make future decisions.
Classifier Challenges
One of the biggest challenges of using classifiers in systematic reviews is getting access to them. Manually building a classifier is an arduous and complicated task that is best left to data scientists. Building a classifier without relying on the training approach is incredibly time-consuming, and is therefore a costly process that most research organizations have neither the time or budget to tackle.
The training method for classifiers is easier and cheaper. However, the idea of classifying references strictly to use to train the classifier is not productive. DistillerSR solves this problem by using references from past projects that have already been screened by humans to train the classifiers. This method is extremely efficient because you aren’t doing any extra work, but still reaping the benefit of training the AI with ‘good’ data.
A “good” classifier will work the way it’s expected: with accuracy and sensitivity. Good classifiers will properly place references in the appropriate classes to which they are meant to be assigned and will correctly exclude references from categories to which they should not belong.
How Much Can We Trust Classifiers?
Classifiers are a valuable tool for researchers. Sorting and classifying references is one of the most time-consuming parts of a systematic review if done manually. By automating select tasks, researchers can save time on their review and ensure they meet critical deadlines. Automation in systematic reviews is something many people want, but are hesitant to adopt because they worry that trusting a robot to do their work could compromise the validity of their reviews.
Originally, classifiers were used in screening to group references into one of two piles: references that look similar to ones that have already been included, and references that do not. The challenge with approach is that because the classifier is only looking at overall similarity, the users cannot tell specifically why a reference was included or excluded.
This approach applies classifiers to specific screening questions (e.g. “Is this reference an RCT?”, or “Was this study conducted on human subjects?”). By having DistillerSR use question-specific classifiers to complete a screening for the way a human would, researchers are now able to determine precisely why a reference was included or excluded. Moreover, in situations where two screeners are used, one can now be DistillerSR and discrepancies between humans and DistillerSR decisions can be automatically flagged for review, resulting in a more accurate screening process with less manual work.
The way we see it, the classifiers are only as good as the data used to train them. So it’s essential to test your classifier to understand its performance. The best way to test is to compare your results with the current gold standard. And consider that using classifiers in systematic reviews is akin to developing any other diagnostic test. It’s important to test and understand their performance so you can make informed decisions for your specific needs.
The Future of Classifiers in Systematic Reviews
So what does the future hold for using classifiers in systematic reviews? At this point, the potential is endless. It remains to be seen exactly how the use of AI in SRs will unfold, but what we do know is that technology will continue to move in the direction of automating or semi-automating the time-consuming tasks involved with systematic reviews. It may eventually help with developing search strategies, identifying relevant records, data extraction steps, and more. We’ve peeked into the future of systematic reviews, and it’s automated!
