Medical Research We Never Hear About: The Problem of Unpublished Studies

Every day, there is another medical study in the news. There’s another newspaper or TV story telling us that X can cure depression or make you thinner or cause autism or whatever. And since it’s a medical study, we usually think that it’s true. Why wouldn’t it be?

But what most people don’t realize, let alone really think about, is that there might be other studies that show that X does none of those things — and that some of those studies might never have been published.

Just this week, the journal Pediatrics released an article that perfectly demonstrates this problem. There have been a number of studies that have shown that a certain type of medication, selective serotonin reuptake inhibitors (SSRIs), can help stop the repetitive behaviors of autism, like hand-flapping or head-banging. If you were to do a search of the medical literature, as doctors and parents and patients often do, you’d think that using SSRIs is a good idea. But when researchers dug deeper, they found that there were just as many unpublished studies that showed that SSRIs didn’t help. If they had all been published (they were all good enough to be published), that same search of the medical literature would have shown that using SSRIs isn’t a good idea.

This is bad. We rely on studies to guide our decisions. What is going on?

The journals that publish articles certainly play a role. After all, it’s cooler to publish a study that has a grabby headline, that promises an answer or a cure. That’s much more likely to get readers than a study that says that something doesn’t do anything at all. But it turns out that the researchers themselves play a bigger role.

Some researchers don’t even write up their studies or try to publish them. You can’t blame them, to some extent. If you set out to show that Wonderdrug cures Bad Disease, and it turns out that Wonderdrug does squat, that hardly seems interesting or important. Or, if you start studying Wonderdrug and then have to stop the study because of side effects or other problems with giving it, you might not write that up or try to publish it either — after all, it wasn’t finished. So the study goes into the trash bin, and you move on to the next idea. No big deal, right?

Well, if other studies are being published that say that Wonderdrug does cure Bad Disease, and don’t mention any side effects or problems, it is a big deal — because a whole other side of the story would be missing.

A companion study in Pediatrics looked at clinical research involving children and found that results weren’t available for more than half the studies involving children — because they weren’t completed or weren’t published.

As a pediatrician, this freaks me out. I’m making medical decisions for my patients based on less than half of the information out there?

There is a database, ClinicalTrials.gov, where researchers are supposed to “register” their studies before they even start. That way, there is a record of it — and even if they stop it, or if it never gets published, there is a way to at least know it existed and find out more about it. Unfortunately, not all studies get registered. Many journals, including Pediatrics, won’t publish a study unless it was registered — if all journals would do that, maybe all researchers would register their studies.

This doesn’t mean that you can’t trust medical research. Medical research has brought us antibiotics and heart surgery and drugs that help many cancers; medical research saves and improves countless lives every single day. But it does mean you have to be a savvy and skeptical consumer of health information. It means that you need to ask questions, learn about the sources of the information, read widely and — always — talk to your doctor before you make a health decision based on a study.

It means that you always have to remember that there just might be another side to the story.

Claire McCarthy, M.D., is a pediatrician and Medical Communications Editor at Children’s Hospital Boston. She and her husband are raising five children ranging in age from 21 to 6. She blogs for Thriving, the health and parenting blog of Children’s Hospital Boston, and for Boston.com as MD Mama; you can follow her on Twitter at @drClaire.

Spread the love

5 replies »

  1. I agree with you, Dr. McCarthy and Matthew above. There have been several instances where journals have refused to publish a study just because they didn’t believe it or due to academic rivalry. One cannot say that the researchers alone are responsible.

  2. Dr. McCarthy, it’s very true that many studies go unpublished and unfinished. But, sometimes the journals themselves prevent a completed study from being published. What if a scientist proves something but can’t get a journal to publish his studies?

  3. Dr. McCarthy, thank you for article, as it brings up a fascinating aspect of the ‘modern age of data’, where the gatekeepers to information distribution have been mostly eliminated. As frustrating as it is that splashy headline studies are more likely to make their way into the common consciousness, having access to clinicaltrials.gov is a giant leap forward.
    Now, with this amazing opportunity to collect and share big data on trials, the barrier is achieving engagement from the researchers in all circumstances and not just when big headlines are a possibility.
    This is a topic which is exciting and important to us as we continue to build out services for healthcare providers and contemplate how to become a part of the solution —– What do you think could be done, other than journal policy changes you mentioned, to encourage researchers to register and thoroughly document their unfinished, inconclusive, or negative study results?

  4. “The journals that publish articles certainly play a role”

    They play THE ROLE

    Publication bias has been around for ever, it doesn’t have much to do with scientists and it has a lot to do with Journals. Why would scientist spend time and effort writing a paper that proves the null hypothesis to be true knowing that it will never see the pages of a reputable journal?

Leave a Reply

Your email address will not be published. Required fields are marked *