Last week saw the Royal Society of Medicine host a conference, “Association or causation in miasmas and mixtures: current reflections on Bradford Hill’s 1965 contribution to public health“. I co-chaired a session on how we incorporate the great ABH’s thinking on causation into the systematic review methods we are developing for accurately summarising bodies of evidence.
Here is the presentation I gave to open up discussion (it’s a bit sketchy because it had to be a fairly off-the-cuff response to the panellists’ presentations). I wanted to emphasise three things when it comes to systematically reviewing evidence, as there seemed to be some misunderstanding in the room about the processes we are developing:
- When appraising evidence, there is always a process which is being followed. The issue is whether or not that process is transparent.
- If the process is not transparent, they cannot be audited or improved.
- If transparent process and the use of “algorithms” is rejected, then there is no way of ensuring that the BH “criteria” are not being misused.
What I didn’t really hear in the room was how, although everyone was very concerned about people gaming the process of appraising evidence in order to come up with favoured rather than true answers, creating transparent and algorithmic (probably not an ideal word) procedures which render most decision-making open to view wasn’t a step forward in this regard. To me, it just seems obvious that processes should be documented and evidence-based, rather than secret and intuitive. But apparently not.
What I would also liked to have said, with hindsight, is to repeat the old Chalmers, Hedges and Cooper quote: “Science is supposed to be cumulative, but scientists only rarely cumulate evidence scientifically”.