An Ode to Evidence-Based Health Policy

A recent kerfuffle ensued when a CDC analyst leaked details of a meeting that noted a list of banned words and phrases that included ‘evidence-based’ and ‘science-based’.  This most recent assault on reason from the Trump administration was lapped up by partisans as yet another example of the dangers of having reality stars occupy the White House.

Unfortunately no one apparently told the director of the CDC, who took to twitter to respond:

Details are sparse.  A meeting took place.  Words were discussed.  No Trump administration official has come forward to take ownership of the meeting.

Regardless, we should all be relieved that we can now get back to the business of implementing evidence based health care policy.

How has that been going anyway?

Ten years ago this month Atul Gawande wrote a widely read article in the New Yorker called The Checklist.  In it he related a masterful riveting story of a 3 year old girl in Austria who slipped into an icy pond, and was underwater for 30 minutes.  On arrival to the intensive care unit, she required massive support – a heart lung machine, treatment and monitoring of brain swelling.  The end result in this case was nothing short of miraculous.  Two years later, she was like any other than 5 year old.  The point Gawande goes on to make is that the complexity of patients in the critical care setting is overwhelming.  In these sickest of sick patients even one mistake could be the difference between life and death.  Humans can’t do it – and the error rate that inevitably results is not one that society should bear.

The solution was the checklist.  Derived from the airline industry to avoid errors, porting this innovation that allowed the flying of planes safely to medicine, would make medicine more manageable and therefore safer.  Peter Pronovost, an intensivist at Johns Hopkins, started making use of a checklist when placing central lines to lower rates of infections related to the presence of these lines.  Remarkable drops in the rates of infections were noted and the era of checklist mania was upon us.  Dr. Gawande was sold, and as a young doctor, just finishing my training, I had no real reason to argue with him.  But as the years passed and I flailed about, losing hair and sleep at the bedside of the sickest of sick patients in intensive care units, the whole  concept started to seem absurd.  Yes, patients were immensely complicated.  Yes some patients died.  But as I went over what we could have done differently – I began to realize that prognosis in most of these patient had little to do with the actions of the team in the ICU.  The patient with a ruptured aneurysm in their brain related to an infection on a heart valve died for reasons that had nothing to do with better processes.  One patient’s infection was controlled by antibiotics, and the strands of bacteria flipping around with every heart beat stayed put or regressed.  The other patient treated in the same manner showered emboli, developed multiorgan failure as a result, and died.  The initial presenting condition was paramount to prognosis.  Beyond that, stochastic events – not expertise – seemed to guide patient outcome.  This doesn’t mean no errors ever took place, but that it was a rare event that could directly link an error to a bad outcome.

So ten years later it comes as little surprise that checklists have failed in a variety of arenas.  Just last week, the closely watched Gawande-inspired Better Birth Project to use checklists to improve maternal and fetal mortality in India was found to improve adherence to best practices, but was found to have no affect on mortality.  Apparently checklists weren’t enough to wipe out the disadvantages inherent in communities that subsist on $2/day.

Checklists didn’t even succeed in solving simpler problems that seemed ideal for the process like the “wrong site, wrong procedure, wrong patient” medical error.  There are few medical errors worse than surgery on the wrong breast, or even worse the wrong breast of the wrong person.  But it happens. Making this a never event became a mission for all sorts of agencies and checklists proliferated in hospital surgical staging areas.  As a boy growing up in India I had been accustomed to hearing the daily muslim call to prayer ringing out from the local mosque.  Who knew that as an adult walking through the hospital I would get used to hearing the health system prayer as nurses dutifully put doctors in ‘time-out’ to call out items from checklists. (This was especially loud when the Joint Commission visited).

If I had only chosen to dig a little deeper than a New Yorker article I would have found that this whole endeavor was doomed from the start.   As newsworthy as these events were, they were already almost never events.  The seminal study on the topic found a rate of errors of 1 in 112,000 procedures.  That means any one hospital would experience this event once every 10-15 years.  That’s a rate of disastrous complications the aviation industry would be envious of.  None-the-less, the pursuit of perfection in this arena brought us the universal protocol and its checklist from the aviation industry.  And it may have made things worse.


According to the Joint Commission, the era of checklists and Universal Protocols resulted in a higher rate of never events.

Joint Comm-fig

To be fair, I don’t know this for sure.  Perhaps the reporting improved, and the rate have been the same the whole time.  No one really knows.  But I think it would be reasonable to conclude that checklists didn’t result in these rare errors becoming never events.

The scale of the health policy evidence based blunders only get larger from here.

Remember The Cost Conundrum?  Also written by Dr. Gawande in the New Yorker, this was another powerful tale that built on data from the Dartmouth Eliot Fisher group that had mapped Medicare spending by county.  McAllen, Texas had the distinction of having the second highest per capita Medicare spending in the country.  A trip from Gawande to this little town uncovered a profit driven enterprise of doctors and hospitals milking the system.  The helpful solution was provided by a trip to the Mecca of healthcare in the United States that was low Medicare cost, but – the low cost, but high value Mayo Clinic.  A visit to a surgeon’s clinic there told of an hour long discussion with a patient followed by a cardiologist materializing within 15minutes from another floor to help ready a patient for surgery the next day.   How did they do this?

“..decades ago Mayo recognized that the first thing it needed to do was eliminate the financial barriers. It pooled all the money the doctors and the hospital system received and began paying everyone a salary, so that the doctors’ goal in patient care couldn’t be increasing their income. Mayo promoted leaders who focused first on what was best for patients, and then on how to make this financially possible.

No one there actually intends to do fewer expensive scans and procedures than is done elsewhere in the country. The aim is to raise quality and to help doctors and other staff members work as a team. But, almost by happenstance, the result has been lower costs.”

The answer to the health care cost problem lay in this elegant article.  The plan as initially forwarded by Eliot Fisher from Dartmouth and now gracing the pages of the New Yorker was to create Accountable Care Organizations in the image of the Mayo Clinic.  Convert McAllen, Tx to Rochester, MN and the nations problems would be solved.

I never stopped to think, of course, exactly how Mayo was operating in this manner.  How could a surgeon at Mayo afford to spend a whole hour with a patient?  How exactly does a cardiologist have time to run down in the middle of the day to discuss a complicated patient?  If the cardiologist doesn’t bill the consultation, how is the cardiologist being paid?

These details were not provided, and these questions were never asked.  The Cost Conundrum was required reading for the framers of the ACA, and so health care was reimagined and jiggered to make winners out of large health care systems.  Cuts from CMS targeted private practice reimbursement.  Regulations that required reporting of practices through an electronic health record were applied.  The incentives quickly melted away to become penalties.  Private practitioners faced a choice : accept the lump of coal or join a hospital.  Most fled to hospitals, dotting the landscape with soup to nuts health care systems and realizing the dream Gawande had written about.

Except, Gawande and his adoring readers (that would include me) had been hoodwinked.  The secret sauce for this high value care being provided to patients by the very best in the field wasn’t in the Medicare data that Eliot Fisher’s group in Dartmouth had put out.  The drunk looking for keys under the lamp post doesn’t find his keys for a reason.  The keys in this case was where no one was looking – payments from private insurers.

Just down the road from where I grew up at Carnegie Mellon University came a paper based on claims data from private insurers that showed a much more complex Savannah than the Eliot Fisher data had lead anyone to believe.Avg hosp reimbursement

The dollars paid by private companies was multiple of what was paid by medicare.  A knee MRI paid by private insurers was $1331, Medicare paid $353.  Even more startling was how Rochester, MN ranked relative to its peers in per capita cost.


While Rochester, MN was a bargain when it came to Medicare spending per beneficiary, it was one of the most expensive markets when it came to private spending per beneficiary.  The other large vertically integrated health systems (Grand Junction, CO – La Crosse, WI) that Gawande had highlighted? Also some of the most expensive on the private market.

Apparently, creating large integrated health system created a monopoly that could effectively name its price for the services it was rendering.  Medicare gets to set its prices – the private insurers have to negotiate with providers.  The fewer health systems in a county, the higher the prices negotiated.  THIS is what was paying for one hour patient visits with a surgeons and made Cardiologists materialize out of thin air.  The idea that any of these large health systems were low cost was a myth.

The New Yorker article was published June of 2009 and received widespread attention.  Barack Obama subsequently held a health care town hall Grand Junction, Colorado a few months later to highlight that members of the community were getting “better results and wasting less money” as part of the push to pass the Affordable Care Act.  The paper from Gaynor & colleagues arrived six years later in December 2015 to what would seem to be much less fanfare.  Gawande wrote a brief mea culpa,  acknowledging large health systems (like the one he had worked for his whole career) were not cheap, but they were still the path to providing high value care. It was just the cost side of the equation that had to be figured out.  The New York times did a wonderful bit of reporting (by Margot Sanger Katz) in their Upshot blog as well.  In today’s era were likes, clicks, and impressions declare victors, I would be willing to bet the non-narrative fitting paper and subsequent coverage lost by a mile.

I recall being profoundly disappointed on reading the Gawande mea culpa.  It was reflective of the approach of most of the policy folks in charge at the time.  Rationalization of one sort or the other were made – Coverage! Not Cost! – and the importance of staying on the path was reaffirmed.

This had been an exercise in a narrative in search of data.  It turns out that empiricism in health policy isn’t quite like the science of sending rockets to Mars.  Ideology rules – academics that worked for large health care systems produced data to support their world view.  Even the paper from Gaynor had been produced with the help of claims data from private insurers who formed the Health Care Cost Institute in 2011.  Everyone seemingly has a side of the story to tell.  The mistake may be to reimagine the healthcare system based on any one of these views.

Designing health care policy would seem to be a thankless task.  Some will no doubt conclude that we just need better data to design a more perfect system.  I am not so sure.  It may be that the lesson to learn from all this is to design as little as possible, and foster an environment that lets patients, not regulators, pick winners and losers in healthcare. The entrenched interests that control much of the flow of the $3 trillion dollars we spend annually are unlikely to let that happen.  They have lots of studies that say so.

Evidence based health care policy awaits, saved from the Great White known as Donald Trump.

I can’t wait.

Categories: Uncategorized

9 replies »

  1. There will never be a perfect health care system. Whatever system is adopted, there will be those who find the loopholes to exploit. As pointed out, Mayo can do what they do because of they exploited their near-monopoly (personal attestation- wife had outpatient cerebral angio there for tinnitus- $50,000 charge paid in full by insurer). If we go back to capitation, doctors will see as few patients as possible; the prescriptions of Z pack by phone will skyrocket. If pay for quality gains more traction, doctors will cherry pick patients. Pharma of course is already exploiting their ability to charge whatever they want for drugs with marginal or no benefit.

  2. I vote for a THCB invitation to the AAMC, JCAHO, Academy of Medicine, AMA and APHA leadership folks for a collective Exorcism to redress their biased confusion about the humanitarian and scientific realms of knowledge, applicable to universal health care.

    I further offer a quotation from Eric Hoffer: ” In a time of drastic change, it is the learned who inherit the future. The learned usually find themselves equipped to live in a world that no longer exists. ”

  3. 1) I said that the rates are likely not accurate, but we have not succeeded in making these never events. Yes we can agree blindly checking off a checklist doesn’t make patients safer.

    2) Utopia usually doesn’t exist

    3) I do focus a lot on -ve govt interventions- its a response to the incremental nature by which medical care delivery is impacted by regulators. For most of health care getting out of the business of insurance covering things is probably best. Then its just between patient and doctor…

    4) I’m not suggesting i’m an amazing doc never making errors. The question is how much variance one should tolerate. We should probably stick to standards that find egregious outliers. Its a conversation which physicians traditionally haven’t lead.

    Medicine would appear to be a self correcting discipline. As evidence comes out, doctors follow and change practices. This can be seen in the 50% reduction of stents placed for stable CAD in the last decade.. (I repeat this constantly, sorry if you’ve heard this before). Dissemination of information/education works. We seem to have a working vibrant physician community that is constantly questioning what its doing. I’d try to stay out of the way..

  4. Health care is stuck. We are not making any progress. We’ve tried many theories…few with evidence beforehand. Nothing is working with costs. For access, if we subsidize enough or force take-up we get more access. No surprises here.

    Many folks are making big money in the present system. Naturally, they can’t see any reason to change.

    In order to move ahead, we may have to get rid of ideology and rely on empiricism and become tough politically.

    Think way outside: Try pure socialism somewhere with healthcare a public good. We may have to try hospital care-only for costs. Go all the way and get rid of billers and insurers. Run as a state or county or district utility.

    Try pure laissez-faire capitalism somewhere with convertible vouchers that are used to allow altruism for poor and needy. Providers and plans must feel that patients are at least somewhat shopping and paying even if this is a virtual-faux exercise.

    Try monopsony purchasing of drugs and medical equipment.

    Try foreign drug purchasing.

    Try having docs and patient groups running hospitals.

    Try bringing Pharma into Board structures of hispitals and health plans.

    Try bringing patients into Board structures of hospitals and plans.

    Have required hospital cost “rounds” by Boards in JCAHO approval criteria.

    Try putting every provider on salary, including administrations.

    Reconfigure US patent laws and clinical trials in ways that might lower drug costs.

    you get the ideas…..et al ad infinitum.

    My point is that we may have to give up our shibboleths and ideology to move ahead. We are lucky to have so many laboratories–states, governments–in which to experiment.

    But, right now, we really are just rearranging the actors and subsidies and trying to keep everyone happy under same incentives.

  5. Truly, it is difficult to compare apples and oranges when the definitions defining the scope of a measure differ for the various resources available. There is also no credible evidence that the underlying definitions of the various global measurement tools have changed over the last 20 years. Given that, it is difficult to dismiss the relevant divergence between the OECD nation’s, as a group, and the USA for national ‘health spending’ as a portion of their GDP OR maternal mortality.

    I fully acknowledge that the international basis for evaluating the results of Population Studies should be seriously considered as possibly useless. On the other hand, the steadily progressive divergence of change rates over-time seem otherwise. If this is possibly true, we will be better off in 30 years by taking a serious look at the implications of over-spending and worsening maternal health. Our nation’s future autonomy within the world-wide community may depend on the actions we take to resolve the implications.

  6. Excellent piece exposing that profound changes to our health care system are too often based on theories that sound good, but then turn out to be half baked, ideological…or both.

  7. In the U.S., the National Health Expenditure data provides a very detailed quantification of the components of our healthcare spending. We get to see how much goes for hospital based care, physician and clinical services, prescription drugs, long term custodial care, dental and vision care, administrative costs, medical research, medical education, hospital construction, public health initiatives including spending by the CDC, etc. For other countries, by contrast, we get one raw number claiming to quantify what percentage of GDP each country spends on healthcare.

    Just as there are significant differences in how different countries define a live birth in the computation of infant mortality rates with the U.S. using the broadest possible definition of a live birth, how do we know that there also aren’t significant differences in how healthcare spending is defined and calculated? I, for one, have never seen a detailed breakdown of healthcare spending from any other country just as I haven’t seen any detailed studies comparing hospital costs, including the number of employees per licensed or occupied inpatient bed, between U.S. hospitals and their overseas counterparts. Where’s the data? I would sure like to see it.

    If we look at what U.S. healthcare payers actually pay for, we’re talking about hospital based care, physician and clinical services and prescription drugs. For Medicaid, nursing home care is also a big expense item. Throw in payer administrative costs and patient out of pocket costs and we’re only looking at 75% or so of healthcare costs laid out in the National Health Expenditure data. That 75% equates to 13.5% of GDP.

    Medicare’s reimbursement rates are not significantly different from payment rates in other developed countries and Medicaid’s rates are probably lower. Commercial insurance rates outside of Medicare Advantage are higher. I would also note that our system likely has an upward spending bias because of the inherent litigiousness of our society which results in defensive medicine being built into practice patterns that define the standard of care. Also, we tend to be much more aggressive in our approach to end of life care. We may well be seeing some improvement in the latter but the former is a much heavier long term lift.

  8. From 1960 thru 2016, the portion of our nation’s GDP devoted to ‘health spending’ has increased from 5.0% to 18.2%. The annual increase in ‘health spending’ represented 5.0% annually, compounded, as corrected by inflation and economic growth. This change has simultaneously occurred along with our nation’s decline in social mobility. Robert D. Putnam wrote most poignantly about this decline in his book “OUR KIDS The American Dream In Crisis” (2015). Associated with this decline, the Social Capital necessary to support each community’s COMMON GOOD has severely affected the actual health of many citizens. Our nation’s worsening level of maternal mortality for 25 years represents the most egregious attribute of declining community based Social Capital. It’s unlikely that the opioid crisis, homelessness or “the mindless menace of violence” will be mitigated substantially without a local investment in its Social Capital, community by community.

    In effect, the increase in ‘health spending’ has slowly reduced our nation’s investment in the Social Capital of education. There is a well-documented ROI of 3:1 for education, 7:1 for early childhood education. We have the most efficient and effective agriculture industry among the 35 OECD nations. Among these same nations, we have the least efficient and, arguably, the least effective healthcare industry.

    I propose that the Paradigm Paralysis of our nation’s health care has occurred from the business model, separately pervading the institutional payers and providers of Complex Healthcare for managing their own market share. It is aggravated by confounding issues of State’s Rights, governmental controls, chaotic financial support for medical education, aberrant PHARMA pricing strategies, absence of appropriately capitalized and equitably available Primary Healthcare and poorly reconciled social adversities occurring historically from our nation’s dominant citizen groups.

    Finally, all of this is NOW aggravated by our nation’s annual Federal deficit that includes a substantial component related to our nation’s excessive growth of ‘health spending.’ Most of the other 34 OECD nation’s have ‘health spending’ that cluster at 12% of their GDPs. Some of these nations are much less than 12% and only one was more than 13% (the Netherlands at 13.1%). In 2016, the portion of our nation’s ‘health spending’ between 13% and 18% represent $1 Trillion. Our Federal government directly pays 40% of our nation’s total ‘health spending.’ Thus, @80% of our nation’s Federal deficit in 2016 was the result or our nation’s level of excessive ‘health spending.’ If it continues, the new tax law will add $150 Billion a year and our nation’s healthcare will continue to add at least $400 Billion a year to our nation’s TOTAL Federal deficit.

    We are in very uncertain times. Our nation’s AUTONOMY within the market-place arena’s of the world’s KNOWLEDGE, RESOURCES and HUMAN DIGNITY is increasingly under attack. Meanwhile, our nation’s economy is headed to bankruptcy on the back of our nation’s excessive ‘health spending.’ There is no reason to believe that any attribute of our nation’s current healthcare reform strategy will change this out come.

  9. 1) “But I think it would be reasonable to conclude that checklists didn’t result in these rare errors becoming never events.”

    I was in the Air Force. No matter what you do, you will still have some crashes. The goal, as with never events, is to reduce them as much as possible. I guess you can have some fun poking at the “never” terminology, but the goal is really to minimize. Also, no one who works in an OR thinks that there were zero events in 1995. This is a reporting problem. Also, if you actually work in an OR you probably realize that some people just don’t pay attention to the checklist.

    2) I had wondered the same thing about Mayo, so was not surprised about their true costs. We have low costs for both Medicare and private patients and we don’t have that kind of service. Close, but not quite there.

    3) ” It may be that the lesson to learn from all this is to design as little as possible, and foster an environment that lets patients, not regulators, pick winners and losers in healthcare. ”

    So, several of you here like to use this line. Exactly how will this work? In general, most spending goes to chronic care, and to big events like surgeries and chemo. You usually have just one splenectomy, or one gallbladder operation or one heart surgery or one cancer. (Yes, unlucky people have more than one heart surgery or cancer.) This is not like buying a car or a TV where you can take it back if you get a lemon. Pts can already make these decisions if they want, but they don’t for the most part. People mostly just go where their PCP suggests, where their family has always gone, or where geography dictates. (Don’t do outpatient care so a lot of this probably doesn’t apply there.)

    Where i will kind of agree is on the regulator part. What we need is more active and aggressive physician leadership. We default way too much decision making to too many people who don’t practice medicine. You focus on government (and just the negative parts ignoring the successes). I would point out how much influence the insurance companies have. I would also fault JCAHO and state agencies here for demanding rules and policies that make no sense, are definitely not evidence based and are enforced with absolute tyranny. Of course, everyone is afraid of them, so no one will really go after JCAHo and company. (I sure hope this is really anonymous.)

    4) More broadly, we are all influenced by our environment. Every doctor who writes here is an amazingly good doctor who always makes the correct decisions and doesn’t need to practice evidence based medicine. They practice the art of medicine. However, as I participate in rescuing troubled hospitals, what I see are a lot of docs with bad outcomes who are practicing the art of medicine. They don’t worry about guidelines or protocols. No one tells them what to do because these hospitals are desperate to keep the staff that they have. (And, to remind some of the physicians here, there are other people in hospitals besides doctors. You get more consistent and better care from nurses and support staff if there is some consistency in how problems are approached. Not cookbook and everyone does exactly the same, but mostly the same, knowing when to deviate.)

    What I suspect, and of course this is speculation as there isn’t good data to support my speculation, is that there is a top tier of physicians (and hospitals) who practice excellent medicine and guidelines, protocols, etc probably don’t help them much at all. They might hinder them. (In practice, not sure I see this. I review every bad preoperative outcome for our network and just don’t see being forced to follow a guideline or protocol as a case for bad outcomes.) There is probably a middle tier where protocols, etc sometimes help, sometimes hinder, with the above caveat. Then there is the bottom tier where pt outcomes would be greatly improved if they actually followed some sort of plan. Almost any plan. What I don’t know is if what we gain with the bottom tier is lost by imposing stuff on the top tier.