Artificial intelligence in systematic reviews is undoubtedly the way of the future. With an overwhelming amount of data powering today’s evidence-based research, it’s becoming more difficult for researchers to keep up with the amount of manual work required to manage and maintain systematic reviews.
Automating manual tasks is the best way to improve efficiency and proactively keep up with the demand for evidence-based research in today’s fast-paced world.
But with the adoption of AI, there are bound to be some growing pains. As with any new, innovative idea, there will be early adopters and those who are hesitant to try the latest technology. A recent webinar moderated by HealthEconomics.com featured Kimberly Ruiz from Xcenda and DistillerSR Inc. CEO, Peter O’Blenis on a panel about AI in systematic reviews. In the webinar, they discussed the current state of the art in AI-enabled reviews, best practices for using AI and where they see things going as we move forward.
Here are a few of the topics discussed:
What is an acceptable accuracy rating?
The short answer: it depends. An acceptable accuracy rating relies almost entirely on your comfort level and how you actually use AI, but keep in mind that if you are looking for 100% accuracy, you most likely won’t find it. Recent studies have shown that error rates in human reviewers also range. If we consider that dual independent screening is the “gold standard” do we need our AI to be as good as a pair of screeners, or just one?
Additionally, some of the feedback we’ve received from early adopters who are integrating AI into their processes is that it’s critical to not get into a situation where the AI needs to be perfect. There are a number of ways in which to leverage AI on your reviews that would not have catastrophic consequences should the AI make a mistake.
What are some of the best practices for the use of AI?
While best practices dictate dual reviewers for screening and data extraction, the same can be applied to AIs. Treat the AI as you would any other reviewer: don’t let it make important decisions without a second set of (human) eyes. While the AI may serve as a second screener, having it make screening decisions autonomously carries risk.
Food for thought: Systematic reviews follow rigorously defined scientific process that must be reproducible. Transparency is critical, so issues can arise from not knowing how or why an AI made a specific decision.
Classifiers, the advanced algorithms that power this type of automation, look for re-occurrences of features with text and make decisions by assigning weightings to them. Classifiers are massive statistical models that can be difficult to dissect. Since determining exactly what an AI based its decision on is extremely challenging, researchers typically fall back to the logical alternative: test and retest the AIs decision making. Do you get the same result every time you use it? Long term validation of AI decision making is one valid approach to developing confidence in its results.
What tasks are currently amenable to semi-automation?
Artificial intelligence in systematic reviews is best used when it’s automating mechanistic, time-consuming tasks. In keeping with the best practices discussed above, several systematic literature review tasks jump out immediately as good candidates for semi-automation. Some of these tasks include:
- Literature sorting/reordering by potential relevance during the screening process
- Citation screening (one of two reviewers in a dual reviewer setting)
- Post screening audit for screening errors
- Simple data extraction/data collection/categorization
By semi-automating these tasks, research teams can save considerable time and effort in their systematic reviews while reducing error rates.
Artificial intelligence in systematic reviews is the future. But it’s not a magic button that will do our reviews for us. A pragmatic approach and an open mind are two things you will need when applying AI to your processes.
