By ROSS KOPPEL and STEPHEN SOUMERAI

A few days ago, we wrote an editorial for US News and World Reports on the scant or dubious evidence used to support some healthcare policies (the editorial is reproduced in full below).  In that case, we focused on studies and CMS statements about a select group of Accountable Care Organizations and their cost savings. Our larger point however is about the need to reconsider the evidence we use for all healthcare-related decisions and policies. We argue that an understanding of research design and the realities of measurement in complex settings should make us both skeptical and humbled.  Let’s focus on two consistent distortions.

Evidence-based Medicine (EBM).  Few are opposed to evidence-based medicine.  What’s the alternative? Ignorance-based medicine? Hunches?  However, the real world applicability of evidence-based medicine (EBM) is frequently overstated. Our ideal research model is the randomized controlled trial, where studies are conducted with carefully selected samples of patients to observe the effects of the medicine or treatment without additional interference from other conditions. Unfortunately, this model differs from actual medical practice because hospitals and doctors’ waiting rooms are full of elderly patients suffering from several co-morbidities and taking about  12 to 14 medications, (some unknown to us). It is often a great leap to apply findings from a study under “ideal conditions” to the fragile patient. So wise physicians balance the “scientific findings” with the several vulnerabilities and other factors of real patients.  Clinicians are obliged to constantly deal with these messy tradeoffs, and the utility of evidence-based findings is mitigated by the complex challenges of the sick patients, multiple medications taken, and massive unknowns. This mix of research with the messy reality of medical and hospital practice means that evidence, even if available, is often not fully applicable. 

Relative vs. Absolute Drug Efficacy:

Let’s talk a tiny bit about arithmetic. Say we have a medication (called  X) that works satisfactorily for 16 out of a hundred cases, i.e., 16% of the time.  Not great, but not atypical of many medications.  Say then that another drug company has another medication (called “Newbe”) that works satisfactorily 19% of the time. Not a dramatic improvement, but a tad more helpful (ignoring how well it works, how much it costs, and if there are worse side effects).  But what does the advertisement for drug “Newbe” say?   That “Newbe” is almost 20% better than drug “X.” Honest. And it’s not a total lie.  Three percent (the difference between 16% and 19%) is 18.75%, close enough to 20% to make the claim legit. Now, if “Newbe” were advertised as 3% better (but a lot more expensive) sales would probably not skyrocket. But at close to 20% better, who could resist?   

Policy:  So what does this have to do with healthcare policy?  We also want evidence of efficacy with healthcare policies but it turns out that evaluation of these interventions and policies is often harder to do well than are studies of drugs. Interventions and policies are introduced into messy pluralistic systems, with imprecise measures of quality and costs, with sick and not-so-sick patients, with differing resources and populations, with a range of payment systems, and so on and so on. Sometimes, randomized controlled trials are impossible.  But sometimes they are possible but difficult to effect. Nevertheless, we argue they are usually worth the effort. Considering the billions or trillions of dollars involved in some policies (e.g., Medicare changes, insurance rules) the cost is comparatively trivial.

But there’s another question: What if a decent research design is used to measure the effects of a large policy in a select population but all you get is a tiny “effect?”  What do we know? What should policymakers do? Here’s what we wrote in our recent editorial in the US News and World Report.

In an ideal world, health policies are based on solid evidence. But what if influential research is flawed? What if policymakers can’t distinguish between weak and trustworthy studies? And what if the resulting policies dramatically affect our health care system’s quality and costs?

Two recent studies in top medical journals, the Journal of the American Medical Association and the New England Journal of Medicine, concluded that “Pioneer Accountable Care Organizations” – which receive monetary incentives in attempts to improve Medicare’s efficiency and quality – saved money or reduced spending increases. Last week, the U.S. federal Centers for Medicare and Medicaid Services noted these findings as a reason to expand the program.

While we also want Accountable Care Organizations to succeed, we question whether the scant evidence reported in the studies justifies such a large public investment in time and money. The studies may have unfairly compared pre-selected, high performing pioneer Accountable Care Organizations with less experienced and less selective controls. This is akin to comparing baseball’s all-star teams to a team made up of the players not even considered for that honor.

To make matters worse, the research found only tiny differences (for example, 1.2 percent medical cost savings in one study) that may be dwarfed by error, bias and unmeasured variables, such as implementations costs. Statistics can’t fix research designs with major bias. Moreover, the JAMA study that measured a second year of Accountable Care Organization experience found declining “savings.” In the words of the co-editor-in-chief of the Incidental Economist, “Saving a couple hundred million out of the Medicare program is a decimal point.”

Our larger concern is how to interpret research on our costly, impossibly complex health system for policymakers who lack the scientific skills to evaluate the conclusions. Apparent cost savings often reflect coding irregularities, unreliable data (for example, insurance claims that may reflect what the companies will pay for rather than the patient diagnosis), unacknowledged biased comparisons, data omissions, arbitrary end points to studies or unrealistic time constraints.

Our worry is that as a research community, we are losing our humility and our caution in the face of declining research funding, the need to publish and the need to show so-called useful findings. Perhaps it’s becoming harder to admit that our so-called big data findings are not as powerful as we wish or are, at best, uninterpretable.

Probably, the most common weakness in such research is selection bias – for example, comparing volunteers to non-volunteers; programs chosen by the funding agency to those that weren’t; and, of course, comparing those who are healthy and wealthy to those who are too sick or poor to participate. Dead patients are also well known for not responding to requests for call-backs, responses or data.

The sociologist W.I. Thomas offered what is now called the Thomas theorem: “If men define situations as real, they are real in their consequences.” In this example with pioneer accountable care organizations, a finding that may be mostly measurement error becomes a truth to support a questionable policy.

What can we learn from this experience? Comparing haves and have-nots often generates exaggerated successes. Ideally, we can hope for randomized trials – in this example, by randomly dividing eligible, advanced health care organizations to accountable care organization participation or to non-participation. (The non-participating comparable controls might receive the program in a year or two after the results of the trial are in.) And when randomized trials are not possible, there are better designs than self-selection and administrative fiat.

We’ve had many randomized trials of government programs. When we are spending billions (or even trillions) of dollars, and the policies affect our social system, our economy and peoples’ well-being, we should require rigorous research methods and even more humility.

A portion of this essay appeared in the May 18th edition of US News and World Report.

Ross Koppel teaches research methods in the sociology department at the University of Pennsylvania and is a senior fellow at the Leonard Davis Institute of Health Economics. Stephen Soumerai is professor of population medicine and research methods and director of the Drug Policy Research Group at Harvard Medical School and the Harvard Pilgrim Health Care Institute.