This is a table from Late Lessons From Early Warnings Vol. 2 (European Environment Agency 2013), summarising the main direction of error from methodological weaknesses in research.
Source: European Environment Agency 2013, p.635. Click to enlarge.
This is important: when assessing the methodological quality of studies in the course of conducting an evidence review in a risk assessment, it is not enough to determine how robust the studies are – it is essential to also determine if the flaws found in the studies are likely to lead to an over- or under-estimate of effect size.
To understand why, imagine five studies were to report a small toxic effect from exposure to a particular compound. All studies are methodologically flawed. It is tempting to give them little weight in the analysis of risk to health posed by the compound – but the temptation should be resisted until the direction of effect of the flaws on estimated size of effect can be determined.
If the error leads to consistent underestimation of size of effect, then rather than discount the studies, as a risk assessor one could conclude that while the evidence may be weak, it is actually more rather than less likely that there is a toxic effect. Conversely, if the flaws result in overestimation of size of effect, then one could potentially conclude that risk to health really is limited.
Either way, from a risk assessment perspective, it is not sufficient to accord value to evidence simply on the basis of study quality, but it is necessary to also anticipate whether the flaws lead to under- or overestimation of effect size.
- Further reading: Grandjean, Bailar et al. (2004).