Why Science is Mistrusted

Why Science is Mistrusted

9
SHARE

Recently, the Harvard Chan School of Public Health, in their press release, reported about the effect of surgical checklists in South Carolina. The release was titled, “South Carolina hospitals see major drop in post-surgical deaths with nation’s first proven statewide Surgical Safety Checklist Program.”

The Health News Review, for which I review, grades coverage of research in the media. Based on their objective criteria, the Harvard press release would not score highly.

The title exudes certainty – “nation’s first proven.” The study, not being a randomized controlled trial (RCT), though suggests that checklists are effective, far from proves it. At least one study failed to show that surgical checklists improve outcomes.

The press release’s opening line is “South Carolina saw a 22 percent reduction in deaths.” It reports relative risk reduction (RRR). Reporting RRR is now considered a cardinal sin in healthcare journalism, because RRR inflates therapeutic optimism by making the intervention sound more efficacious than it is.

A good press release gives the reader the whole picture of competing evidence, particularly where there’s controversy. The press release mentioned one other study which showed an even higher mortality benefit of surgical checklists.

The press release goes onto say,

“Adoption of a safe surgery checklist has been demonstrated to reduce deaths in controlled research studies since 2009. But the ability to produce improved outcomes at large scale has remained questioned.”

In other words, the science is settled. But there is no mention of the Canadian study which failed to show surgical checklists improve outcomes. The reader is lulled into a false sense of certainty about the benefits of surgical checklists.

If this was a press release for a statin, or device, which, ironically, subjects itself to greater methodological rigor, it would have been taken to task. Yet, no one batted an eyelid about the press release about surgical checklists. Why does a sensational press release about an inferior quality study about surgical checklists not induce the same ire as a sensational press release about an RCT of a drug or device?

Future anthropologists might better answer this question. I’m not an anthropologist, let alone a future anthropologist. But I will speculate. In our rational age, we’re still indefatigably prejudiced, self-righteous creatures. Witnessing pharma make big bucks on false therapeutic optimism gets under our skin. Money corrupts. Nothing else quite corrupts to the same degree, we believe.

The proponents of surgical checklists don’t make money off checklists. That the proponents of checklists might advance their academic careers despite the therapeutic optimism of checklists doesn’t perturb. The nobility of their intent – they wish to make surgery safer – shields them from the scathing scrutiny reserved for drug and device industries.

It’s not easily apparent that the advocates of checklist might have the “this is my bright idea” bias. In this regard, the reaction of Atul Gawande, the author of Checklist Manifesto, who is self-evidently a true believer of checklists, to the Canadian study is instructive. He “wished the Ontario study was better.” He felt the study was underpowered and the implementation of checklists was weak.

When you don’t like the results of a study, the next step is to find weaknesses in the study design. The astute reader will note that I’ve done exactly what I’m accusing Gawande of doing. Guilty as charged. I have my biases. And there is no way to determine if one’s search for methodological weaknesses in a study is actuated by intrigue or bias. But my point isn’t whether surgical checklists work or not. My question is why do we not see the same skepticism for surgical checklists, and other policy measures, that we see for drugs and devices.

Of course, there are differences between checklists and drugs. Simple checklists cost nearly nothing to implement. If you’re wrong about the efficacy of a drug, that could be costly. If you’ve overestimated the efficacy of a surgical checklist, that’s no big deal.

But the differential standards, as justified as they may be, may lead to a deep mistrust of science and of dabblers in science. A casual observer, unfamiliar with epistemic nuances, may suspect that ideology is at play. The observer may see, in the sliding scale of methodological rigor, blatant double standards. The observer may lose faith in statistics seeing that it can be fine-tuned, like the thermostat of a shower, to prove and disprove what we wish proven or disproven.

When Anne Case and Angus Deaton showed that middle class white men were not reaping the mortality benefits of other demographics, the demolition of their analysis was swift.

https://twitter.com/StuartBuck1/status/847104688562548736

 

In a sense, this is good because science needs refutation. But would their study have inspired the same statistical scrutiny if it had shown that Hispanic or South Asian men were dying sooner than white men? Skepticism loses credibility if seen to be applied selectively. And if we can’t trust skeptics, who can we trust?

Statisticians can debunk or defend a study using their deep knowledge of statistics by selectively highlighting the strengths and weaknesses of a study – no study design is perfect. The degree to which the weaknesses of a study can be exposed, is arbitrary – there’s no normative frame. Thus, statisticians have become the new lawyers. I wonder how long before we have a “right to a statistician.”

The ideologue with a deep knowledge of statistics is the most dangerous ideologue. The ideologue who claims God or history is on their side is easy to identify. The ideologue who claims to have science, which really means statistical technique, on their side is more difficult to identify and refute. Few have deep enough knowledge of statistics to call BS on the callers of BS. The result is that people mistrust science.

Science has become a substitute for morality. It has become a weapon to fight social injustice (whatever that term means), to reduce inequality, to mock incompatible world views. This is a blow for both science and morality. Historically, our species never resorted to science to do the right thing. Slavery was bad because it was morally bad. The Civil Rights Movement was right because it was morally right. The suffragettes were on the right side of history, not because an academic proved the net benefits of women voting in an election (p <0.0005), by analyzing a secondary database. The suffragettes were right because they were morally right. India’s freedom struggle was not initiated by a macroeconomic analysis, but because independence was morally right. If your moral intuitions can’t tell you that same sex marriage should be permitted, I doubt any scientific analysis will convince you.

It may be morally right to be politically correct. Then we should say so, and keep science away from political correctness. Science loses most credibility when seen to selectively address the favored social injustices of the time. It is not fashionable in elite circles to be perturbed by the plight of the working class white man. In this circle, which I sometimes frequent, a physician seeping with social justice from every pore, once said that it was imperative that we developed a vaccine for malaria, because the Africans have suffered enough from imperialism. He was right and wrong. He was right that a malaria vaccine will reduce the suffering of Africans tremendously, and that it was imperative that it be developed. But he was wrong. The vaccine is no more, or no less, important because of the history of the African continent. Alexander Fleming didn’t discover penicillin because he wanted to fight capitalists.

We may discourage reporting RRR for drugs and devices to curb therapeutic optimism and encourage reporting RRR for surgical checklists to encourage compliance. This is understandable, because how we frame a message, and reporting relative risk reduction is just a way of framing, depends on what we wish to achieve. But here lies another problem. Note, science is not informing us what to do. We’ve already decided what we want to do. Science is helping us implement, helping us engineer, helping us design, not a bridge, not a spacecraft, not a computer, but ourselves and society. This was once the dominion of religion. That this is now the dominion of science is ironic. But more troubling than irony is that science is no longer value-neutral. And when science ceases to be value-neutral, science ceases to be science.

(Saurabh Jha MD is a contributing editor to THCB. He can be reached on Twitter @RogueRad)

Leave a Reply

9 Comments on "Why Science is Mistrusted"


Member
rmcnutt
Apr 25, 2017

Wow. You have a gift for making a point. I loved your piece. Let me try some controversial additions.

A pre-post study where everyone knows the hypothesis and does the analysis of the data is NOT science. It is a marketing experience.
An observational trial is NOT science. Don’t be seduced by the occasional large OR findings of some observational trials.
When a data set has been used for the same question over and over, this is NOT science.
A large randomized trial looking for minuscule differences in outcomes is NOT science. It is too easy to get spurious false positives, statistically more likely than neutrality, and can’t be replicated.
Using a convenience sample of patients for a study is NOT science.
Any analysis done by those doing the study, no matter how well trained, is NOT science.
A study, even a RCT, which cannot blind the intervention or the outcome is NOT science.

Maybe we need to debate clinical science standards, raise the bar, and be willing to stand behind findings because we think the study was scientific in the first place. If we don’t, don’t call it science. False positives are many; negative studies most likely to be correct? Refutation is meaningless if the science is bad in the first place. Science needs to get better to cure mistrust. We are partially to blame for the mistrust. If any study is science, the term is meaningless.

Member
Allan
Apr 25, 2017

Excellent! A lot of stuff out there is data mining. Throw a set of dice 11 times and it is more likely than not to get two of one number. The data miners think the dice are weighted towards that number because they don’t understand what they are doing and have no idea of probabilities. Unfortunately many of them are the same people that are destroying healthcare today and making life for physicians impossible.

Member
meltoots
Apr 23, 2017

Lets go over this checklist.
1. The questions will NOT decrease death, “do we all know each others names/roles, did we mark the site, etc.” In fact, the “sign your site” actually INCREASED wrong site surgery.
http://www.amednews.com/article/20101101/profession/311019963/2/
2. These ridiculous checklists are yet another click box nightmare for staff and are already overburdened with the advent of EHRs. Do we actually read all these popup alerts? 750 a day? I’d still be at work from 2 days ago if I actually read them all. Every time someone has a codeine allergy (more like sensitivity), every single OTHER narcotic ordered generates a popup alert about this allergy. We learn VERY quick how to ignore/dismiss the window. That leads to us ignoring and dismissing ALL popups. Get it?
3. As a surgeon, reading over this “magical death reducing checklist”….I see a few things that are good to communicate that are key to this list, so we could drop 20 out of the 23 things. How long do we think this is going to take, how much blood loss we anticipating, and are there probable difficulties we are going to run into. Those are good thing for anesthesia and the surgeon to discuss. But as I said, we do ALL these things already. To make yet another laundry list check box nightmare, it will be strategically ignored and clicked and dismissed, like I am SURE they were in the study’s operating room cases. What they should really look at is if the clicking the boxes was REALLY performed or were they just clicked. Antibiotics within 60 min prior -check? Was it 61 min? Or is that close enough? 65 Min? Was the essentials imaging displayed? Did it to go screensaver or log out after a specified time? Did it have to be re-looked up? See where Im heading? Many of these “Self” reported check boxes are never done correctly or at all. Its among 1100 other check boxes we are expected to do in the next hour. Comparing checklist hospitals to non check list hospitals is ridiculous. Further, there is NO way that the questions asked prior to these surgeries makes ANY difference in outcome. No way, They could employ a “how are you feeling checklist” and show the same reduction in deaths. Meaningless.

Member
Steve2
Apr 23, 2017

From your cited article on wrong site surgery, which is consistent with the cases I have been asked to review.

“Yet the physician narrative reports analyzed in the new study showed that in 72% of wrong-site procedures, the required timeout was not performed.

Not sticking to protocol
Patient safety experts said the problem is not with the safety protocol, but with failure to follow it every time. Sometimes physicians and other health professionals rush through the checks, and in some places, the surgeon may not even be in the operating room when the timeout discussion is done.

“I’m not aware of a case we’ve ever had where the procedures were followed correctly and we still had an incorrect surgery,” said James P. Bagian, MD, former chief patient safety officer at the Veterans Health Administration, an early leader in establishing presurgical safety checks.”

Also, I don’t really think that death is necessarily the best metric.

“But as I said, we do ALL these things already. ”

Many places do not, or have not done so historically. You keep saying you have been in practice a long time. How can you not be familiar with the places where staff were never allowed to question a doctor, especially a surgeon?

Steve

Member
William Palmer MD
Apr 23, 2017

I know nothing about how checlkists have been received by the operating team. Are they a giant hassle? It they require trivial work to accomplish, the issue seems moot. It is hard to see much to object to, if they don’t have opportunity costs.

Member
Steve2
Apr 23, 2017

At the institutions where I work, or for which I have responsibility, there was a lot of resistance at first, though not as bad as that with the central line initiative. It took at least a year until we had real compliance with several staff needing to be called in for individual talks. Talking with chairs from other programs, I think our experience was within the norm. (This is part of why I don’t think the natural experiment in the Canadian study is as good as some might think. It would be more meaningful if you took data a year or two before the change and a year or two after, but then there could have been other changes that caused a change. Just not a perfect way to do this kind of study, but probably your best chance is to have fairly large studies.)

Steve

Member
Seth
Apr 23, 2017

Outstanding!!!

The thing about checklists is they are not objective devices. It matters less what is on the list than participants attitudes towards safety. A great checklist will not trump a crew who doesn’t exude safety through their pores, a crew who is not morally engaged.

Could a checklist study cohort be biased, not by the list, but by engagement in the process?

Comm. SEL

Member
Apr 22, 2017

Well done, Saurabh.

Science at the service of the sociopolitical agenda is not new. In fact, it is the implicit framework under which we have been operating for the last hundred years–regardless of political affiliation.

https://en.m.wikipedia.org/wiki/Progressivism

Thanks for highlighting its inherent contradiction.

Member
Steve2
Apr 22, 2017

Your point is taken, but I think that in general public reporting on science issues is not very well done. (I also thought that the Canadian study was underpowered and had other issues, but it is really hard to do a “natural”study of this sort.)

Steve