What we did
We analysed two of EFSA’s Scientific Opinions on BPA to determine whether the methodology they followed for evaluating evidence of BPA’s toxicity are consistent with scientific standards for literature review, as defined by Cochrane Reviews in medicine.
We found that they are not consistent with these standards. Therefore, if in future we are to be confident that the decisions made in chemicals policy are those which are best supported by the evidence, the processes which EFSA follows for reviewing toxicological evidence need to be radically overhauled.
We make some suggestions as to how this might be achieved.
The report advocates the use of systematic review techniques first developed for use in medicine as a new approach to reviewing evidence in the conduct of chemical risk assessment, in order to strengthen the connection between the decisions made in chemicals policy and the evidence base which supports them.
Chemicals policy is increasingly characterized by controversy rather than consensus. For chemicals such as BPA, we see a range of opinions as to its safety, from EFSA’s position that it poses no threat to health at current exposure levels, to Swedish regulators even banning its use in thermal paper. This diversity of opinion exists in spite of everyone having, at least in theory, access to the same evidence base.
The same problem has been faced in medicine, where there are many examples of how decisions made in healthcare have failed to match those which were best supported by the available evidence. The cause of the problem was determined to be a general failure to use scientific methods for identifying, appraising or synthesising information when conducting reviews of the literature. The solution? To develop systematic review techniques, the application of the basic scientific principle of using a reproducible methodology to the process of reviewing evidence.
In medicine, systematic review techniques cover six basic elements: a clearly-stated objective; the use of a pre-published protocol defining the methods to be used in the review; a systematic search for evidence; clear criteria for electing evidence for inclusion in the review; an assessment of the methodological quality of the included studies; and a systematic synthesis of data and presentation of results.
A comparison between EFSA’s recent Scientific Opinions on BPA with a scientific approach to reviewing evidence produces a similar result as to that which was seen in medicine: review objectives are not sufficiently clearly stated; there are no pre-published protocols; methods for locating data are not consistently given; the criteria for selecting data for analysis are incompletely stated; how studies are evaluated for quality appears to be neither transparent nor consistent; the synthesis and presentation of results is unclear.
Without a transparent, reproducible method for evaluating toxicological data, it is not possible to be confident that the decisions made in chemicals policy are with those which are best supported by the evidence. As was the case in medicine, the solution is to develop systematic review techniques for reviewing toxicological data (part 4).
On this basis, we recommend the following measures to strengthen Scientific Opinions:
1. In advance of developing Opinions, EU Agencies should publish and publicly consult on a review protocol, to cover: review objectives; methods to be used in searching for evidence; criteria for including and excluding evidence in the analysis; criteria for appraising the quality of the evidence; and the method for synthesising the evidence.
2. As a precondition of achieving a scientifically-robust review process, a toolkit for appraising the methodological quality and directness of evidence needs to be developed and validated.
3. Guidance should be issued for Working Groups and Scientific Committees on the structure and writing of Opinions, in order to enhance usability for stakeholders in chemical regulation.
4. Controls on the interests of the authors of Opinions need to be tightened, restricting direct financial conflicts of interest and developing further policies for the management and declaration of all interests which could be perceived by a user to have influence on the conclusions of an Opinion.
5. All decisions made in the review process need to be sufficiently documented so as to be transparent; all methodological considerations, such as criteria for inclusion and quality assessment of studies, must be consistently applied throughout.
6. An editorial and peer-review process for revising and accepting Opinions needs to be instituted, to ensure that published Opinions meet the quality criteria outlined above.
There are a number of research initiatives in addition to the Policy from Science Project which can contribute to this process, with particular progress on review protocols being made by the Navigation Guide (University of California San Francisco, US), the Evidence Based Toxicology Collaboration and the US National Toxicology Panel. EFSA has also begun work in this area.
As a timetable for change, in the short term all imminent Scientific Opinions, including the next Scientific Opinion by EFSA on BPA (the hazard component of the overall risk assessment) should be structured to maximize ease of understanding. They should also include a comprehensive declaration of interests, present the full results of the evidence search and selection processes, and have a clear description of the methods used for appraising and synthesizing the studies included in the Opinions.
To support this programme, EU scientific staff and experts should receive training in systematic review techniques. In addition, researchers should be piloting more systematic reviews.
In the medium term, Scientific Opinions should be conducted according to pre-published protocols, developed in an open consultative process. Funding should be made available for education and research in systematic review methods.
In the long term an organization of similar function to medicine’s Cochrane Collaboration needs to be established, to facilitate the production of and set the standards for systematic reviews of toxicological evidence.
4 Responses to New report: The future of evidence in chemicals policy