Uncategorized

Is There Something Wrong With the Scientific Method?

A recurring them on this blog is the need for empowered, engaged patients to understand what they read about science. It’s true when researching treatments for one’s condition, it’s true when considering government policy proposals, it’s true when reading advice based on statistics. If you take any journal article at face value, you may get severely misled; you need to think critically.

Sometimes there’s corruption (e.g. the fraudulent vaccine/autism data reported this month, or “Dr. Reuben regrets this happened“), sometimes articles are retracted due to errors (see the new Retraction Watch blog), sometimes scientists simply can’t reproduce a result that looked good in the early trials.

But an article a month ago in the New Yorker sent a chill down my spine tonight. (I wish I could remember which Twitter friend cited it.) It’ll chill you, too, if you believe the scientific method leads to certainty. This sums it up:

Many results that are rigorously proved and accepted start shrinking in later studies.

This is disturbing. The whole idea of science is that once you’ve established a truth, it stays put: you don’t combine hydrogen and oxygen in a particular way and sometimes you get water, and other times chocolate cake.

Reliable findings are how we’re able to shoot a rocket and have it land on the moon, or step on the gas and make a car move (predictably), or flick a switch and turn on the lights. Things that were true yesterday don’t just become untrue. Right??

Bad news: sometimes the most rigorous published findings erode over time. That’s what the New Yorker article is about.

I won’t try to teach here everything in the article; if you want to understand research and certainty, read it. (It’s longish, but great writing.) I’ll just paste in some quotes. All emphasis is added, and my comments are in [brackets].

  • All sorts of well-established, multiply confirmed findings have started to look increasingly uncertain. It’s as if our facts were losing their truth: claims that have been enshrined in textbooks are suddenly unprovable. … In the field of medicine, the phenomenon seems extremely widespread, affecting not only antipsychotics but also therapies ranging from cardiac stents to Vitamin E and antidepressants: Davis has a forthcoming analysis demonstrating that the efficacy of antidepressants has gone down as much as threefold in recent decades.
  • “This is a very sensitive issue for scientists,” [Schooler] says. “You know, we’re supposed to be dealing with hard facts, the stuff that’s supposed to stand the test of time. But when you see these trends you become a little more skeptical of things.”
  • [One factor is] publication bias, or the tendency of scientists and scientific journals to prefer positive data over null results, which is what happens when no effect is found. The bias was first identified by the statistician Theodore Sterling, in 1959, after he noticed that ninety-seven per cent of all published psychological studies with statistically significant data found the effect they were looking for.
    • [The point here is that naturally all you see published is a successful study. Lots of useful information can come from failed studies, but they never get published.]
    • [The problem is that anything can happen once, at random. That’s why it’s important that a result be replicable (repeatable by another scientist): like that light switch, if someone else tries it, you better get the same result. But the article points out that most published results are never tested by another researcher.]
  • In recent years, publication bias has mostly been seen as a problem for clinical trials, since pharmaceutical companies are less interested in publishing results that aren’t favorable. But it’s becoming increasingly clear that publication bias also produces major distortions in fields without large corporate incentives, such as psychology and ecology.
  • [But publication bias] remains an incomplete explanation. For one thing, it fails to account for the initial prevalence of positive results among studies that never even get submitted to journals. [By this point, this article was driving me nuts.]
  • [Re another cause of this problem,] In a recent review article, Palmer summarized the impact of selective reporting on his field: “We cannot escape the troubling conclusion that some—perhaps many—cherished generalities are at best exaggerated in their biological significance and at worst a collective illusion nurtured by strong a-priori beliefs often repeated.”
  • [We had two posts in October here and here in October about an Atlantic article by Dr. John Ioannidis, who is quoted in this article:] “…even after a claim has been systematically disproven”—he cites, for instance, the early work on hormone replacement therapy, or claims involving various vitamins—“you still see some stubborn researchers citing the first few studies that show a strong effect. They really want to believe that it’s true.”
  • The current “obsession” with replicability distracts from the real problem, which is faulty design [of studies].
    • In a forthcoming paper, Schooler recommends the establishment of an open-source database, in which researchers are required to outline their planned investigations [before they do them] and document all their results. [Including those that fail!]
    • [Note: Pew Research publishes all its raw data, for other researchers to scrutinize or use in other ways.]

The corker that caps it off is John Crabbe, an Oregon neuroscientist, who designed an exquisite experiment on mice sent to three different labs with incredibly uniform conditions. Read the article for details. When these mice were injected with cocaine, the reactions of the three groups of relatives were radically different. Same biology, same circumstances, seven times greater effect in one of the groups.

What?? (There’s more; read it.)

If you’re a researcher and this has happened, and it’s time to “publish or perish,” what do you do? What is reality?

The article winds down:

The disturbing implication of the Crabbe study is that a lot of extraordinary scientific data are nothing but noise. The hyperactivity of those coked-up Edmonton mice wasn’t an interesting new fact—it was a meaningless outlier, a by-product of invisible variables we don’t understand.

Implications for e-patients

Wiser people than I will have more to say, but here are my initial takeaways.

  • Don’t presume anything you read is absolute certainty. There may be hope where people say there’s none; there may not be hope where people say there is. Question, question, question.
  • Be a responsible, informed partner in medical decision making.
    • Don’t expect your physician to have perfect knowledge. How can s/he, when the “gold standard” research available may be flaky?
    • Do expect your physician to know about this issue, and to have an open mind. The two of you may be in a glorious exploration of uncertainty. Make the most of it.
  • Expect your health journalists to know about this, too. Health News Review wrote about this article last week, and I imagine Retraction Watch will too. How about the science writers for your favorite news outlet? I can’t imagine reporting on any finding without understanding this. Write to them!

Mind you, all is not lost. Reliability goes way up when you can reproduce a result. (Rockets, light bulbs, chocolate cake.) But from this moment forward, I’m considering every new scientific finding to be nothing more than a first draft, awaiting replication by someone else.

Dave deBronkart better known as “e-Patient Dave,” is one of the leading spokesperson for the e-Patient movement. A high tech executive and online community leader for many years, he was diagnosed in 2007 with Stage IV kidney cancer, with a median survival of just 24 weeks. He used the internet in every way possible to partner with his care team and beat this unbeatable disease. Today he is well. In 2008 he discovered the e-patient movement, and began studying, blogging, and speaking at conferences, and in 2009 was elected founding co-chair of the new Society for Participatory Medicine. In 2010 he released his first book: “Laugh, Sing, and Eat Like a Pig: How an empowered patient beat Stage IV cancer (and what healthcare can learn from it).” He blogs frequently at e-patients.net.

Livongo’s Post Ad Banner 728*90

17
Leave a Reply

17 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
15 Comment authors
Jane SanchezBrandon HullRegina HollidayTina Harris, MDJanice Reverson, RN Recent comment authors
newest oldest most voted
Jane Sanchez
Guest
Jane Sanchez

There is NO SUCH THING as a common, universal, singular “method” that all scientist must use or follow. No such thing exists. Science textbooks promote GARBAGE. Little, if any, science is done that way. We all learned it completely wrong in 6th grade…..

Brandon Hull
Guest
Brandon Hull

http://www.theatlantic.com/magazine/archive/2010/11/lies-damned-lies-and-medical-science/8269/
Fantastic companion article in the current Atlantic on the same topic. Highlights work of John Ioannidis, who is making a career out of challenging the durability of medical study results. He basically concludes that there is nothing wrong with the scientific method per se, but society harbors unrealistic expectations about the magnitude and pace of it’s results. Somewhere in the piece he responds, when asked what the average person should do upon reading about some new study: “Ignore it.”

e-Patient Dave
Guest

Tina, I have no idea where you got your attitude, but you’re barking up the wrong straw man. If you’ll read my posts about my testimony to the meaningful use hearings, particularly those that cite Ross Koppel’s concerns about safety (especially my final testimony), you’ll see that I specifically talk about the value of patients serving as a second set of eyes for errors in the record – as Regina did. I know you have more things to do than subscribe to everything I do, but please also see slides 52-55 of my talk at AHRQ’s 2010 meeting of grantees… Read more »

Regina Holliday
Guest

This is Regina Holliday, data access evangelist, artist and widow. I thought I should chirp in here as our data access story got wrapped around Dave’s article reviewing measurable outcomes and the scientific method in peer reviewed journals. It does seem quite unrelated on first glance, but it really is not. If you listened to many of my speeches including two in front of Secretary Sebelius, you might have noticed my conclusion: “I eventually got a copy of Fred’s record and – despite its many errors– it became a virtual bible that we used to guide Fred’s care for the… Read more »

Tina Harris, MD
Guest
Tina Harris, MD

David,
Please provide the scientific evidence, if any, proving safety and efficacy of HIT systems as devices to control and distribute all care of hospitalized patients.
Sincerely,
Tina

Janice Reverson, RN
Guest
Janice Reverson, RN

Propensity can not spell but her points are valid. Having attended, I recall Regina’s comments in front of Secretary Sebelius at the roll out of the meaningful use rule in July 2010. Point of information, she stated that the doctors would not talk to her, that his care was run by an EHR, and that he died and that she could have saved him for a few months if she could see his EHR. Why wouldn’t the doctors not talk with her? If EHRs are so good, why did she need to tell the doctors what to do? Are you… Read more »

Peter
Guest
Peter

“As for patients, it’s ludicrous to expect them to be intelligent consumers of healthcare. Most of them are just not smart enough to understand the data. A huge percentage of patients lack the reading skills to follow the instructions printed on their damn medicine bottles, they simply are not going to be able to say “hey doc, I thought the n in that BMJ study on beta blockers was too small, what’s your take?” At their best, most people cannot understand this crap–and sick people are, by definition, not at their best.” +++Great points especially after the posts on consumer… Read more »

anon
Guest
anon

The scientific method is not supposed to be immutable. At all. I think this mis-interpretation comes from the whole “science vs religion” debates, but science on its own changes quite frequently and profoundly as we learn more. For instance, the Newtonian physics that was used in ballistics was completely uprooted by quantum mechanics–though, generally, it’s still a good enough approximation of reality at our scale to put a man on the moon. What we are seeing in science as applied to medicine is a whole bunch of things. First of all, our working models, unlike Newtonian physics, are not a… Read more »

e-Patient Dave
Guest

Hi Propensity – we haven’t met, I think. Some sloppy thinking here, and I’d welcome a chance to sort it out, since you seem passionate and outspoken here. What kind of work do you do? Feel free to drop me a note to discuss – my contact info is on epatientdave.com. (Really – I’m happy to gave a good discussion, and I see no reason for reasonable people to be anonymous.) I presume when you say “NATTA” you mean nada, Spanish for “nothing.” Not sure what natta is. And I’m glad you feel I’m to be walked-on-air (exulted), though I… Read more »

John H. Ryder
Guest
John H. Ryder

Some wrong with the scientific method NO it is the determaniation of the most proable.

Graham Chiu
Guest

The old adage, “Publish or Perish” also has a lot to do with this problem.

propensity
Guest
propensity

+++”But from this moment forward, I’m considering every new scientific finding to be nothing more than a first draft, awaiting replication by someone else.” Funny you should say that when you are promoting HIT instruments of care, especially CPOE gear, that have not any proof of safety and efficacy (ZERO, ZILCH, NATTA). Your electronic libraries are ok but often promote errors. I guess medical care managed by CPOE and HIT is so bad now that the sick patient needs to boot up the computer while lying in hospital bed filled excrement and see if he/she is dying while the nurses… Read more »

Gilles Frydman
Guest

It has been said that the scientific method is the worst form of knowledge discovery except all the others that have been tried.

e-Patient Dave
Guest

Thanks for the cross-post, and for the thoughtful comments.
I’d encourage people to see the meaty comments on the original post.
p.s. In my bio here, please remove “MD”. I’s just a patient.:-)

Stephen J. Motew, MD, FACS
Guest
Stephen J. Motew, MD, FACS

The scientific method is a mechanism to ‘optimise’ the search for ‘truth’. Indeed, the primary focus of a particular study is to support or refute a hypothesis. The definition for hypothesis states: “a proposed explanation for an observable phenomenon.” Note the word proposed. Over time, the finding that a study may ‘erode’ may simply support the advancement of knowledge or results of additional studies attending to the same hypothesis. Appropriate research indeed is a continued quest to identify new or expanded information that by definition may dispel previous information. Those studies that do not erode under such scrutiny are likely… Read more »