Consider whether unpublished results were considered.
For summaries of research evidence to be reliable, unpublished results should also be looked for and considered. This is because many fair comparisons are never published, and in published reports, some outcomes are sometimes left out. Comparisons that are published are more likely to report favourable results leading to possible overestimation of the benefits of health actions and underestimation of the harms.
Explanation
It is important that systematic review authors look for and consider unpublished results and reports of comparisons of health actions.
Many randomized trials (fair comparisons of health actions) are never published. Researchers have found that trials that favour a particular health action (“positive findings”) were nearly twice as likely to be published as trials with “negative findings”, and that trials with positive findings were published more quickly than trials with negative findings. This bias is called publication bias. When not recognised and addressed in systematic reviews, publication bias can sometimes result in overestimation of the effects of health actions.
Outcome reporting bias occurs when researchers select for publication only some of the originally recorded outcomes after they know what the results are. Outcomes are more likely to be selected for publication when there is a clear positive result. When not recognised and not addressed in systematic reviews, outcome reporting bias can result in overestimation of treatment effects.
Requiring researchers to register protocols of trials in registries such as clinicaltrials.gov can help to reduce publication and reporting bias.
Example
Researchers found that over half of all trials submitted to the U.S. drug licensing authority, the Federal Drug Administration (FDA), between 1998 and 2000 remained unpublished five or more years after FDA approval of the drug. Selective reporting of trial results was found for commonly marketed medicines. Submitted trials of drugs for treating depression (anti-depressants) that reported positive results and larger effect sizes were more likely to be published in medical journals afterwards.
Remember: Consider whether the authors of systematic reviews have addressed the possibility of misleading underreporting of fair comparisons.
- Video: Prof. Steven Goodman on “publication bias”. This is a Stanford Medicine video lasting 1:39 minutes.
- Blog: All fair comparisons and outcomes should be reported. Students 4 Best Evidence.
- Blog: Publication bias. Catalogue of Bias.
- Essay: Dealing with biased reporting of the available evidence. The James Lind Library.