We managed to submit a few last-minute comments to EFSA’s consultation on the use of weight-of-evidence methods in conducting scientific assessments. We weren’t able to go into much detail but the general gist was that, when conducting scientific assessments, the following should be ensured:
1. A full process is followed for ensuring that the right questions are asked in a scientific assessment, in a manner amenable to systematic review (in particular ensuring that questions are answerable with available time and resources) and which address stakeholder needs.
2. Methods for identifying, appraising and synthesising evidence in a scientific assessment are planned and published in advance of the assessment, in the form of a protocol, which should then be followed in the assessment process. Deviations from the protocol should be made only when necessary and always be justified.
3. All evidence of potential relevance is found via a systematic search strategy and screening/selection process.
4. All evidence is assessed for internal validity using appropriate risk of bias assessment methodologies. These methodologies should replace the ambiguous requirements around assessing “reliability” of the evidence.
5. Suitably comparable evidence is synthesised and the credibility of the results of the synthesis are interpreted in the context of structural limitations in the evidence base. This should take into account the overall risk of bias of the evidence, and also its precision, heterogeneity, risk of publication bias, and its consistency.
6. Where multiple streams of evidence of limited direct comparability have been included in a scientific assessment, understanding of how these streams of evidence support or contradict each other (the integration of evidence) should be developed according to a priori inferential models developed in the review planning process.