When conducting a review, there are a number of different approaches to reference screening. While the basics are solo screening and dual screening, there are many variations on these that you can configure to best suit your needs.
DistillerSR gives you the freedom to design your own screening forms and to decide what inclusion/exclusion consequences you use. In addition, DistillerSR allows you to configure how the responses your screeners select are used in determining the fate of a screened reference.
This reference provides quick overview of what happens to a screened reference depending how a reviewer or reviewers select inclusion, exclusion and neutral responses in your screening forms.
When to use Setting 1
When teams use one reviewer (human or AI) to screen or classify each record.
Benefits:
- Less resource-intensive than dual human review with consensus.
- Neutral responses will be considered conflicts. However, in DistillerSR, they can be configured so that neutral responses can move to the subsequent review stage or remain “in conflict” until resolution. This could be useful to “tag” or identify records that require further review. (e.g. challenging records to be reviewed by another member of the team or when more information is needed.)
Challenges:
- Chance of errors is higher with a single reviewer over dual review.
- Teams using this approach should ensure reviewers are experienced and knowledgeable of the topic.
- Additional measures such as quality control or review of “neutral” responses by another reviewer could also be beneficial.
When to use Setting 2
Often referred to as “liberal inclusion”, this setting requires at least two reviewers (human or AI).
Benefits:
- May accelerate the process of screening at the level to which it is applied as it requires only one reviewer to indicate if a record is relevant or potentially relevant, but two to indicate if a record should be excluded.
- Dual review for exclusions may decrease exclusion errors compared to a single reviewer.
Challenges:
More records will generally move forward in the review process than with dual independent review with conflict resolution. This could potentially require additional resources for subsequent levels (e.g. full-text retrieval and screening).
Note: Most groups using this level setting will not require conflict resolution for inclusion/exclusion status. Instead, they will choose to allow conflicts to pass to the subsequent review stage.
When to use Setting 3
Dual human review is often considered the “gold standard”.
Benefits:
- May decrease inclusion/exclusion errors over single reviewer screening or liberal screening.
Compared to liberal or accelerated review settings, it may also result in fewer records passing to the next level. - If you are using DistillerSR’s AI features such as continuous automated reprioritization, this type of cautious screening approach may also improve the performance of the algorithm.
Challenges:
- Requires more resources than single reviewer screening.
- Requires additional time to review conflicts and reach consensus over a liberal screening process. This may or may not be offset by fewer records passing to subsequent levels.
Note: Most groups using this type of level setting will require conflict resolution for that level (e.g. consensus between reviewers). If conflict settings dictate that conflicts are passed on to further stages, this setting will have the same consequences as liberal screening shown above (Setting 2).
When to use Setting 4
This setting can be employed when review teams need to collect more information during screening. For example, it can help with classifying records for triage or prioritization for subsequent review stages (e.g. RCT vs. observational study), collecting key information from excluded studies, tagging records as additional sources, or using neutral response options to flag records where additional information or expertise is required.
Benefits:
- The neutral option allows review teams to include both consequential responses (e.g. inclusion/exclusion responses) and neutral responses (screening questions or data variables with no consequence).
- Enables teams to collect additional information during screening without consequence.
Challenges:
For teams opting for an unconventional workflow, it is recommended that review teams pilot their settings prior to the review.
For all Settings: The level setting flexibility allows your workflow to be configured to the unique needs of each project. We recommend piloting your settings prior to the review, especially for teams considering an unconventional workflow. DistillerSR allows teams to easily configure, test and, if needed, adjust the settings of their workflow so you can create the optimal workflow for your review.
