Average effects

Be cautious of average effects.

In comparisons of health actions, sometimes outcomes are measured on a scale, and the effects are reported using the difference between the average in one treatment group (e.g., the average weight or score on a questionnaire) and the average in a comparison group. Scores can be difficult to interpret, and average effects can be misleading because not everyone experiences the average effect.

Explanation

Outcomes assessed using scales (continuous outcomes), such as pain or quality of life, are often difficult to interpret. Scales are sometimes reported in ways that make them meaningless (e.g., not reporting the upper and lower ends of the scale or saying whether higher numbers are good or bad). In addition, in studies with continuous outcomes, effects are sometimes described using average differences (average effects). These are calculated by finding the difference between the average score in a group of participants in a study and the average score in a comparison group of participants. Average effects can be misleading because the average score does not apply to everyone in a comparison group. Also, the average difference does not make it clear how many people experienced a big enough effect for them to notice it, or that they would regard as important. Even if the average effect of a health action is less than the smallest effect that is important to people, the health action may have an important impact on some people.

Sometimes continuous outcomes are converted to yes/no outcomes (dichotomous outcomes). This can make it easier to interpret the results. For example, pain can be measured on a scale from 0 to 100. Most people experience a 50% relative reduction in pain as important. So rather than reporting an average difference in pain scores, which can be misleading, researchers can report the difference in the number of people who had at least a 50% relative reduction in pain.

In systematic reviews it is sometimes difficult to make sense of continuous outcomes especially when different scales are used in different studies.

Example

Scores on questionnaires are often used to measure quality of life. In one quality-of-life questionnaire, researchers found that the smallest change that is important to people on a seven-point scale is 0.5. Even if the average effect between a treatment and a comparison treatment is much less than 0.5, the treatment may have important impacts (change greater than 0.5) on many patients.

Remember: Be cautious of outcomes measured on scales and of average effects. Don’t assume that everyone experiences the average effect.

Back to Top