Whether you’re choosing your next car or replacing your worn out sneakers, it’s generally a good idea to do your research before you pull out your wallet.
The same applies when looking for software to help with your systematic review process. The tools available in this space vary greatly and it can be overwhelming to try to understand their strengths and weaknesses. Some tools focus on one stage of the review process, such as screening or reporting, while others are used throughout the review life cycle. Some tools are specifically designed for reviewing randomized controlled trials (RCTs), while others can handle a variety of study types. How can you figure out which is best for you?
The trouble with the feature matrix
The feature matrix is one of the most common approaches to comparing products to help make a purchasing decision. It typically takes the form of a chart with a number of desired features and requirements along one side and all the product options along the other, with a check mark to denote which requirements are met by each product.
The feature matrix tends to work well for comparing products that essentially do the same thing, but fails epically when comparing substantially different solutions.
To illustrate this point, here’s an example of a feature matrix comparing several different “transportation solutions”:
Transportation Solutions Buyers Guide
Looking at the table above, you might conclude that a bicycle and a boat can be used interchangeably. Since that’s obviously not the case, what’s wrong with the feature matrix? Simply put, the feature matrix fails to capture context.
‘Fast”, for example, means different things to different people. Similarly, the speed of a bicycle really depends on who is riding it. The same can be said for “Ease of use”.
And if you’re traveling within a city, “Gets you from A to B” may be valid for a bicycle, but if “A” is North America and “B” is Europe, the validity of each option changes.
The devil is in the detail
Most systematic review tools available today have very little in common in terms of what they do, where they fit into the process, and how they work. To create a useful feature matrix, you have to make sure you’re comparing apples to apples – but when it comes to systematic review tools there are also oranges, grapes and strawberries. Now what?
Let’s take a look at the example feature matrix again from the perspective of systematic review tools to see why it doesn’t work very well when the available solutions are not really comparable.
As you can see, the granularity and specificity of the questions you’ll need to ask will vary widely from one use case to the next. For instance, the terms used to describe each stage of the process may mean different things to different groups (I can think of at least five different types of “screening”). What is important for one group may be irrelevant to another. Depending on your requirements, you may even benefit from using a combination of systematic review tools given that they vary so widely in purpose and implementation.
Systematic reviews are typically much more complex than most people anticipate. Because of that, we don’t always know what we don’t know, so asking the right questions ahead of time is difficult.
This is why using a “standard” feature matrix to evaluate tools can be misleading. When you actually conduct your review, you may discover that you need more features than you initially thought. Doing a live review tends to surface unexpected challenges and shed light on how certain tools may help. Once you’ve lived it, you may need to leverage your experience to find a systematic review tool that better fits your needs.
If you are new to systematic review software and don’t know what to look for, consider trying the different options while conducting a real review, using your specific protocol, workflow, and reference set to yield the most useful data to guide your final selection.
