A new analysis from the PFS Project identifies major methodological obstacles to accepting that EFSA’s draft 2014 Opinion on BPA is a valid synthesis of the evidence of BPA’s toxicity.
> Click here to download the full report (PDF) <
This new critical appraisal has been conducted according to a set of general principles derived from the Cochrane Collaboration’s approach to systematic reviews, developed to minimise bias and maximise reproducibility of reviews of medical trials. These principles are outlined in a previous report on the use of systematic review methods in chemical risk assessment (Whaley 2013).
The nine domains under which we analysed EFSA’s draft 2014 Opinion on BPA (slide from PW’s presentation to EFSA stakeholder workshop on 23 April 2014)
Primary concerns relating to the validity and utility of the Opinion
1. The search and selection methods for finding evidence only retrieved a partial snapshot of the current literature relevant to evaluating risks to health posed by BPA. Documentation of search methods is insufficient to evaluate the extent to which this might have affected the results of the Opinion.
2. The appraisal of the methodological quality (reliability) of the literature in the hazard assessment of BPA is not valid, failing to in fact measure the methodological quality of research.
3. The weight-of-evidence analysis is insufficiently documented to allow its validity and consistency of conduct to be evaluated.
4. There appears to be two parallel, distinct risk assessment methodologies in the one document: the initial objective of characterizing risk is apparently superseded by a second process in which the Panel is concerned with identifying studies which would warrant a change to the current TDI for BPA. It is therefore ambiguous as to whether or not the Opinion fulfils its terms of reference.
Discernible improvements relative to previous Opinions on BPA
The extra documentation provided in this Draft Opinion compares favourably with previous Opinions, particularly in the level of detail given in the appraisal of the methodological quality of studies included in the review.
However, this improvement in transparency also reveals a lack of clarity with regard to best practice in finding, selecting, appraising and synthesising evidence of the toxicity of BPA before interpreting this into a risk assessment.
This should not be a surprise: best practices in these areas are not yet established. EFSA’s commitment to transparency in documentation and improving risk assessment methodology is therefore a useful contribution to strengthening the scientific foundations of chemical risk assessment.
Recommendations for the 2014 Opinion
1. In consultation with stakeholders, choose a single objective to either:
- evaluate existing studies to determine if there is a published study which would force revision of the current TDI for BPA (does not require a hazard assessment), or
- conduct a full risk assessment based on synthesis of all available data (does require a hazard assessment).
The following steps should be taken regardless of a choice between (a) or (b):
2. Review all the data relevant to the risk assessment of BPA within a specified time scale, either cutting off at 2012 or including all 2013 data.
3. Fully document the results of the search process and list the excluded studies, giving the primary reason for exclusion of each.
If (b), the subsequent steps should be taken:
4. Evaluate each relevant study on its own merits rather than treating previous Opinions as definitive appraisals of the state of the evidence relating to the toxicity and risk assessment of BPA.
5. Present results for the appraisal of the reliability and weight of studies on a study-by-study basis (as done in the neurotoxicity lines of evidence) rather than grouping several studies into single judgments of reliability and weight.
6. For appraising the reliability of studies, use a tool which better measures the credibility of a study than the guidance currently given in the document.
7. Fully document the method for (a) interpreting study reliability into weight of evidence, and then (b) interpreting weight of evidence into likelihood of hazard, ensuring that the same method is followed for each end-point.