Science Is Leading Us to More Answers, but It’s Also Misleading Us

Be careful what you wish for. That is the unexpected lesson of the past decade of biomedical research, which has been characterized by an overwhelming abundance of interesting things to study and powerful ways to study them. A pioneer of this era, MIT geneticist Eric Lander, speaks eloquently of the “global view of biology,” meaning that scientists now have extraordinary tools to study not only individual genes, but also multiple genes at the same time. Rather than immediately investing all their resources in a few favorite genes (the traditional approach), modern researchers first can survey thousands of initial candidates, then identify and ultimately direct their attention to the most important players and pivotal networks.But we are increasingly discovering that this global perspective comes at an unexpectedly steep price: We’re making a lot more mistakes. Or, at least, we seem to be having a lot of trouble picking out the rare, meaningful signal from the deafening noise in the background.

Typically, scientists accept a result as significant if there is a 95 percent chance it is real rather than random. But the catch is that as you start to make a large number of comparisons by examining thousands of genes, the possibility of a result appearing by chance becomes progressively more likely, to the point where such false positive results are all but guaranteed.The consequence has been a boon for scientists — most experiments yield enticing (read: publishable) results — but a bane (of sorts) for science.Scientific journals are littered with studies reporting “disease genes” or “molecular signatures” that are likely red herrings. To make matters worse, these results are typically packaged together as a tidy narrative, a post-hoc rationalization explaining how the newly identified genes fit perfectly into the biological process under investigation.A recent review of 85 published genetic mutations proposed to be associated with heart attacks demonstrated a validation rate of zero: There was insufficient evidence to suggest that any of the originally published associations reflected more than chance alone.Some genetic researchers (including Lander, who trained as a mathematician) recognize these pitfalls and apply appropriately rigorous statistical correction, leading to more-durable results that have been independently replicated and verified (the gold standard of science). But other investigators, particularly in other areas of biology, are still lagging.The problem of errors associated with multiple comparisons turns out to afflict much of contemporary science: the study of thousands of proteins or metabolic factors, even the study of brain activity using MRI. Compare enough discrete brain regions, and differences are bound to emerge.Sexy? Yes. Publishable? Often. Valid? Not necessarily.The multiple-comparison problem may also be afflicting science as a whole.There are now more scientists in the world than ever before, publishing more papers in more journals. Many scientists work on related problems, and it is likely — inevitable, even — that similar experiments are being done in many labs at the same time. Thus, even if the odds of an individual false positive result are only one in 20, the probability rockets upward as more researchers do the same study.Now, you might think that multiple researchers pursuing a common question would provide a safety net of sorts, a healthy counterbalance to the problem of false positives, but often this isn’t the case. Scientists (and scientific journals) have little appetite for negative results, and this pervasive bias effectively buries much of this important information.While the truth might eventually emerge as researchers meet at conferences and engage in informal discussions, erroneous results are almost never retracted, even though recent estimates suggest that more than half of published results are not reproducible.Of course, this tendency of random results to be reified by publication bias is not unique to science; the popular press routinely extols the latest hot hedge fund or brilliant money manager, rarely considering whether this success could simply be the result of chance.In a slightly different way, the pharmaceutical industry (where I have experience) also has fallen victim to the seductive charms of overabundance. For years, drug companies worked on a relatively small number of targets; as a result, there was a large knowledge base around these targets and in-depth understanding of how they functioned in the body. After the genetic revolution, companies have found themselves awash in possible drug targets, many of which initially exist only as a DNA sequence and a database identification number.According to a 2001 study by Lehman Bros., the average drug target was associated with only eight publications, down from several hundred a decade or so before. This shallow biology represents a significant challenge in the development of novel drugs (especially if, as expected, at least four of these eight reports are inaccurate or unreliable).Equally challenging for the industry, emerging data suggest that many complex diseases (such as Type 2 diabetes) are likely to result from the integrated effect of hundreds or even thousands of subtle genetic variants, rather than from a single causative mutation, making the selection of suitable drug targets an even greater problem.Before we throw the baby out with the bath water here, let’s be clear about a few things: Scientifically, we are better off now than we were four years ago and much better off than we were four decades ago. The powerful new techniques of global biology have opened up captivating new vistas and permitted a level of analysis and insight our scientific predecessors could not have imagined. As we used to say in graduate school, “More is more.”But we also must come to terms with the unique challenges associated with these powerful modern approaches. Just as researchers must adopt rigorous new methods of statistical analysis and temper their enthusiasm with caution, all of us, as consumers of scientific information, must balance the hope we place in global biology with the skepticism this field has surely earned.In the words of another era: Trust, but verify.David A. Shaywitz, an endocrinologist and stem cell scientist by training, writes frequently about health and science. He is a management consultant in New Jersey and a member of the healthcare practice of the Boston Consulting Group. This article originally appeared in the Washington Post.

6 replies »

  1. David, I hope you read this. What is needed is to restore and conserve the genius of divided government in health care government. The problem with executive organizations whether they are oligarchic bureaucratic corporations or oligarchic bureaucracies is that way too much power is invested in the managers. The way to reduce this effect is to improve the organizations with the democratic principles of separation of powers:
    Any health care reform should include Quasi Judicial functions in Judicial organizations, and Quasi Legislative functions in Quasi legislative organizations. In short Membership organizations made up of Providers and Scientists should select commissions that weigh in on Malpractice and best practices scientifically, publish results, and debate best practices.
    Specifically health care Congresses in each locality would provide its income by being allowed to keep the fees that doctors pay to keep their licenses. And they’d choose commissions who’d participate in arbitration reviews when malpractice is alleged or errors occur. They’d also choose delegates to Congresses at State and Metropolitan level and to the National Government; who would in turn select commissions for appeals and State and National issues.
    Let the Doctors set guidelines. Let anyone who experiments outside those guidelines be the only ones who experience malpractice risk. And let these boards examine options.
    Managers can then use this information for their own purposes, but will no longer be allowed to be judge, jury, legislature and executioner.

  2. Mistakes are of epidemic proportion. Useles data is out there. Useful data is misrepresented in the interests of profits.
    This will become worse as the government in a joint venture with industry has its way, forcing the use of poorly designed and coordinated HIT systems.

  3. David,
    Science does not mislead….It is we who can not interprete and make judgments.
    “Nature does not reveals its secret, it just answer to a line of questioning”
    In case of bio, there is not enough understanding to look at system level issue. We try to understand problems at local level and which is good but body is more that just brain, heart, etc. It is combination of all. That is the reason, our medicines have so strong side effects.
    When we figure out the total body interactions, the medicines would be far more better.

  4. Why do you think it is misleading us?there is a lot of research going on on health and medicine and there is a lot of help too.

  5. Dear Dr. David, It is nice to read about the truth. I wish I could share with you my work. My work is about prevention from origin. please, e-mail me your e-mail address, and I would be happy to send you a transcript of my work. ajarimba@hotmail.com Thank you very much.