Consider whether outcomes were assessed reliably in the people being compared.
Some outcomes are easy to assess, such as births and deaths. Others are more difficult, such as depression or quality of life. For comparisons of health actions to be meaningful, outcomes that are meaningful to people should be assessed using methods that have been shown to be reliable.
Explanation
In a comparison of health actions (things people do for their health), unreliable outcome measures (how the outcome is assessed) result in measurement error. This can lead to underestimation or overestimation of the effects of health actions. To protect against being misled by measurement error, outcome measures should be precise. That is, random errors in measurement due to the play of chance should be unlikely. This is sometimes called ‘reliability’. However, to be shown to be ‘reliable’ (trustworthy), systematic errors (bias) should also be unlikely. This is sometimes called validity or accuracy.
Outcome measures include diagnostic tests, investigator-reported outcomes and patient-reported outcomes. Patient-reported outcomes are often used to measure outcomes that are important to patients. However, patient-reported outcome measures do not always reflect what is meaningful and important to patients. It is important to ensure that patients understand them and that they capture all aspects of what is important (relevant) to them.
In addition, self-reporting of an outcome can be influenced by the desire for approval, especially when anonymity and confidentiality cannot be guaranteed. For example, self-reporting of behaviours such as diet, smoking, sexual behaviours, drug use, or compliance with a prescribed treatment can be influenced by the study participants’ perceptions of what the researchers or healthcare providers view as good or bad behaviour. This can result in over-reporting of “good behaviours” and underreporting of “bad behaviours”.
One way of improving the reliability of the outcome measures used in a comparison of health actions is to create a “core outcome set”. This is a standardised set of outcomes, agreed by participants, healthcare providers, and researchers. Greater use of core outcome sets could improve evaluations of the effects of health actions and systematic reviews of those evaluations.
Example
The effects of treatments for a mental health condition called schizophrenia have been studied using both outcome measures that had been formally evaluated (and shown to be reliable) and ones that had not been formally evaluated. Treatment comparisons that used an outcome measure that had not been formally evaluated were more likely to report that a treatment was superior to the comparison treatment than comparisons of the same treatments that used an evaluated outcome measure. This was because use of unevaluated outcome measures resulted in misleading overestimation of effects.
Remember: Be careful about relying on the results of comparisons of health actions if outcomes have not been assessed using methods that have been shown to be reliable.
- Video: Reliability and validity. This is a Northeastern University video lasting 4:13 minutes.