New report: The future of evidence in chemicals policy

New report: The future of evidence in chemicals policy

What we did

We analysed two of EFSA’s Scientific Opinions on BPA to determine whether the methodology they followed for evaluating evidence of BPA’s toxicity are consistent with scientific standards for literature review, as defined by Cochrane Reviews in medicine.

DOWNLOAD THE REPORT HERE

We found that they are not consistent with these standards. Therefore, if in future we are to be confident that the decisions made in chemicals policy are those which are best supported by the evidence, the processes which EFSA follows for reviewing toxicological evidence need to be radically overhauled.

We make some suggestions as to how this might be achieved.

Executive Summary

The report advocates the use of systematic review techniques first developed for use in medicine as a new approach to reviewing evidence in the conduct of chemical risk assessment, in order to strengthen the connection between the decisions made in chemicals policy and the evidence base which supports them.

Chemicals policy is increasingly characterized by controversy rather than consensus. For chemicals such as BPA, we see a range of opinions as to its safety, from EFSA’s position that it poses no threat to health at current exposure levels, to Swedish regulators even banning its use in thermal paper. This diversity of opinion exists in spite of everyone having, at least in theory, access to the same evidence base.

The same problem has been faced in medicine, where there are many examples of how decisions made in healthcare have failed to match those which were best supported by the available evidence. The cause of the problem was determined to be a general failure to use scientific methods for identifying, appraising or synthesising information when conducting reviews of the literature. The solution? To develop systematic review techniques, the application of the basic scientific principle of using a reproducible methodology to the process of reviewing evidence.

In medicine, systematic review techniques cover six basic elements: a clearly-stated objective; the use of a pre-published protocol defining the methods to be used in the review; a systematic search for evidence; clear criteria for electing evidence for inclusion in the review; an assessment of the methodological quality of the included studies; and a systematic synthesis of data and presentation of results.

A comparison between EFSA’s recent Scientific Opinions on BPA with a scientific approach to reviewing evidence produces a similar result as to that which was seen in medicine: review objectives are not sufficiently clearly stated; there are no pre-published protocols; methods for locating data are not consistently given; the criteria for selecting data for analysis are incompletely stated; how studies are evaluated for quality appears to be neither transparent nor consistent; the synthesis and presentation of results is unclear.

Without a transparent, reproducible method for evaluating toxicological data, it is not possible to be confident that the decisions made in chemicals policy are with those which are best supported by the evidence. As was the case in medicine, the solution is to develop systematic review techniques for reviewing toxicological data (part 4).

On this basis, we recommend the following measures to strengthen Scientific Opinions:

1. In advance of developing Opinions, EU Agencies should publish and publicly consult on a review protocol, to cover: review objectives; methods to be used in searching for evidence; criteria for including and excluding evidence in the analysis; criteria for appraising the quality of the evidence; and the method for synthesising the evidence.

2. As a precondition of achieving a scientifically-robust review process, a toolkit for appraising the methodological quality and directness of evidence needs to be developed and validated.

3. Guidance should be issued for Working Groups and Scientific Committees on the structure and writing of Opinions, in order to enhance usability for stakeholders in chemical regulation.

4. Controls on the interests of the authors of Opinions need to be tightened, restricting direct financial conflicts of interest and developing further policies for the management and declaration of all interests which could be perceived by a user to have influence on the conclusions of an Opinion.

5. All decisions made in the review process need to be sufficiently documented so as to be transparent; all methodological considerations, such as criteria for inclusion and quality assessment of studies, must be consistently applied throughout.

6. An editorial and peer-review process for revising and accepting Opinions needs to be instituted, to ensure that published Opinions meet the quality criteria outlined above.

There are a number of research initiatives in addition to the Policy from Science Project which can contribute to this process, with particular progress on review protocols being made by the Navigation Guide (University of California San Francisco, US), the Evidence Based Toxicology Collaboration and the US National Toxicology Panel. EFSA has also begun work in this area.

As a timetable for change, in the short term all imminent Scientific Opinions, including the next Scientific Opinion by EFSA on BPA (the hazard component of the overall risk assessment) should be structured to maximize ease of understanding. They should also include a comprehensive declaration of interests, present the full results of the evidence search and selection processes, and have a clear description of the methods used for appraising and synthesizing the studies included in the Opinions.

To support this programme, EU scientific staff and experts should receive training in systematic review techniques. In addition, researchers should be piloting more systematic reviews.

In the medium term, Scientific Opinions should be conducted according to pre-published protocols, developed in an open consultative process. Funding should be made available for education and research in systematic review methods.

In the long term an organization of similar function to medicine’s Cochrane Collaboration needs to be established, to facilitate the production of and set the standards for systematic reviews of toxicological evidence.

4 Responses to New report: The future of evidence in chemicals policy

  • Who can you believe when it comes to chemical safety? | Health & Environment
    [...] reading: Systematic Review and the Future of Evidence in Chemicals Policy. A new report from the Policy from Science Project, looking at how systematic review techniques [...]
    Reply
  • Douglas Cross
    The ‘Policy from Science’ Project’s latest review, ‘The future of Evidence in chemicals policy’, provides a much-needed challenge to the process whereby science is regularly misrepresented by current policy, not only in the field of regulation of chemicals but in related areas as well. The methodology employed by the Cochrane Collaboration does indeed provide an excellent example of best practice in reviewing evidence-based decision-making. But its widespread adoption and implementation would nevertheless still not result, of itself, in resolving the greater problem posed by ineffective regulation where it is most urgently needed - ensuring that scientific concerns are swiftly and properly reflected in enforcing actual policies themselves. There are many examples in which well-founded conclusions of bodies such as the Cochrane Collaboration are simply ignored, even by closely-following reviews by other apparently independent and reputable bodies. For example, in 2004 the Collaboration examined the evidence on which the UK State policy of providing fluoridated milk to primary school children in ‘deprived’ areas is founded. It concluded that the evidence base was of too low a quality to endorse the practice. Yet only four years later the World Health Organization carried out a similar review, and enthusiastically supported milk fluoridation, despite over 90% of the evidence base on which its conclusion was founded having been already discarded by Cochrane as being unreliable. Yet amongst those promoting the continuation of milk fluoridation, it has been the unreliable conclusions of the WHO review that are endorsed and presented to politicians, allowing the policy to persist regardless of the uncertain, but allegedly ‘best available’, evidence. Scientific evidence may change as the research on which it is based is constantly challenged, but political policies, once embraced and implemented, are far less liable to modification or abandonment. To some extent this is the inevitable result of relying on precisely that same ‘transparent, reproducible method for evaluating toxicological data’ that this latest review advocates for evidence appraisal. In virtually all cases, reappraisal of existing policies depends on the existence of an established methodology of testing the evidence. This is itself founded on a prior ‘body of proof’ that has been compiled using a standardised methodology. When a review, even using the best available method of appraisal, finds that an established practice appears to be sound, this may simply be telling us that it is consistent with a previously-embraced belief. But even if such evidence is internally consistent, this does not ensure that it is sound - if the same bias exists in both the established research and the new, then consistent evidence may simply be consistently in error. Some of the most challenging evidence against an existing policy may come - indeed, usually comes - from exploratory research that reveals unexpected issues caused by unexpected or ignored factors. The reluctance of regulators to accept that Thalidomide posed a very high risk in pregnancy led to hugely damaging delays in restricting its use, for example. But if there is no ‘established body of evidence’, gathered by repeatable (and repeated) confirmatory research, that is contrary to established belief, then the policy is all too often allowed to persist long after at least caution in its implementation could be, and should have been, implemented. There is a pervasive bias in many expert groups in favour of the supposed scientific consensus - indeed, many members of such groups have been appointed precisely because they support that consensus. Exploratory research, no matter how severely it challenges the accepted view, is almost invariably dismissed until a sufficient body of confirmatory research on the anomalous issues raised by the first study emerges, Only then does such pioneering work force an often reluctant change in the established policy. There is unquestionably an urgent need to improve the quality assurance methodology recommended by these authors, and aimed at improving the reliability of formal reviews of all evidenced-based decision-making on chemicals policy. But if this is to have the maximum practical effect in protecting public health and safety, it is imperative that we also establish a more effective capacity to assess the reliability, implications and significance of new research findings. This is vital if such new findings reasonably challenge established assumptions; in such cases it is imperative that there is provision to enforce application of the precautionary principle, to suspend or modify existing policies until reasonable certainty - one way or the other - can be demonstrated. Until decision-makers in health policy can be made to react far more quickly to emerging threats than at present, no amount of careful study will ensure that when swift action is justified it will have any discernible benefits in the real world. Having observed at first hand how policy can indefinitely transcend evidence, even in the most extreme public emergencies, I have little confidence that good science alone will have much real effect in protecting the public from rogue decision makers.
    Reply
  • Report back from EU Parliament report launch | The Policy from Science Project
    [...] >> Download the report here. << [...]
    Reply
  • Everyone’s research problem | Health & Environment
    [...] This might sound like a niche research problem, but in fact it is a problem for everyone: we are constantly presented with pieces of conflicting research, which we are then challenged with trying to decide which one is the more credible. Yet we are clearly still at a very early stage of understanding how to do this – as the recent Policy from Science Project report showed, not even the European Food Safety Authority has a robust method for distinguishing better research from worse when it evaluates the safety of compounds such as BPA (Whaley 2013). [...]
    Reply

Leave a Reply

Your email address will not be published. Required fields are marked *