By KIP SULLIVAN

In my first comment  in this series (an open letter to President Obama), I criticized Obama for stating in an article  in the Journal of the American Medical Association that the Affordable Care Act is deflationary. I promised him I would post more essays showing how badly he had been misled by three experts who influenced him: Elliott Fisher and his colleagues at the Dartmouth Institute, Atul Gawande, and Peter Orszag.

My second post presented evidence that the research by Fisher et al. on regional variation in Medicare spending has been enormously influential with US policymakers for the last three decades.

In this comment, I demonstrate the gross inaccuracy of the Dartmouth group’s research.

Let me state at the outset: Even if every paper Fisher et al. wrote about regional variation in Medicare spending were true, none of them constituted evidence for the “accountable care organization.” In other words, even if we accept the Dartmouth group’s claim that regional and hospital variation is due primarily to overuse, we would still have no reason to accept the group’s claim that ACOs are the solution to all that overuse.

The ACO is a barebones, evidence-free construct that Elliott Fisher made up. It bears more resemblance to the rules of a simple parlor game, like the Six Degrees of Kevin Bacon game, than it does to a plan or blueprint to get something done in the real world. Like the Kevin Bacon game, the ACO does not depend on previous research for its appeal. It merely requires a receptive audience.

The rise …

The Dartmouth group’s influence, which was already considerable by the early 1990s, increased with the publication of the Dartmouth Atlas  in 1996. (The Atlas is a database showing Medicare spending and utilization broken down by region and hospital.) The group’s influence increased again in 2003 with the publication of a two-part paper  in the Annals of Internal Medicine, entitled “The implications of regional variations in Medicare spending,” which purported to demonstrate there is no correlation between how much a region spends on Medicare recipients and quality of care. Donald Berwick called this article “possibly the most important paper of the decade.” [1]

The Dartmouth group’s influence reached its peak in 2009-2010. In 2009 congressional Democrats sought the group’s advice in the course of writing the bills that would eventually become the Affordable Care Act (ACA). In 2010, the ACA was enacted with provisions authorizing ACOs.

The most convincing and worrisome sign of how powerful the Dartmouth ideology had become by 2009 was its impact on members of Congress from the allegedly more efficient regions of the country. Lawmakers from states with low per capita Medicare spending introduced legislation to force CMS to use a “geographic value index” that would reduce the difference in payments between the allegedly efficient and inefficient states. [2]

Blind faith in the Dartmouth Atlas was so widespread among Democrats from the low-Medicare-spending states that Obama promised to request a study examining the factors behind geographic variation in Medicare spending. As the New York Times put it, “In March, [2009,] in response to the Congressional Democrats who would have otherwise withheld their support for the health legislation, the administration made a promise. It said it would ask the Institute of Medicine [IOM] … to consider ways of putting the Dartmouth findings into action by setting payment rates that would punish inefficient hospitals and reward efficient ones.” As it turned out, the IOM’s report, issued in 2013, contradicted the Dartmouth group’s most fundamental premise – that Medicare spending is a good proxy for total spending and can therefore be relied upon to determine which regions are efficient and which aren’t (see this Kaiser Health News article  for a readable summary of the IOM report).

And the decline

The years 2009-2010 were the high-water mark of the Dartmouth group’s influence. As their ideology grew more influential during the 2000s, resistance to it began to grow within the health policy community. By 2010 criticism from health policy experts had grown so frequent Fisher and Jonathan Skinner characterized their critics as “a cottage industry … trying to debunk our findings.” In 2010, the media began to pay attention to the criticism. Since 2010, the Dartmouth Atlas, and the research based on it, has taken a severe beating.

In the remainder of this comment I will offer a quick review of the criticism published since 2010. That criticism falls into two categories: (1) Criticism of Fisher et al.’s claim that they accurately adjusted their expenditure data to reflect differences in patient health; and (2) criticism of Fisher et al.’s claim that whatever is true about Medicare spending has to be true of total spending (spending by Medicare and other public programs plus all private-sector spending).

All the patients died, so they must have been equally sick

In his article for JAMA, Obama cited Fisher et al.’s 2003 two-part paper in Annals of Internal Medicine  In the first part Fisher et al. casually made an astonishing announcement: They could render risk-adjustment unnecessary if they limited their subjects to patients in the last six months of their lives. With no caveats and citing no research, they asserted that “regional differences in end-of-life spending are unrelated to underlying illness levels,” and, therefore, end-of-life spending “reflect[s] the component of regional variation in Medicare spending that is due to physician practice rather than regional differences in illness or price.” In later papers Fisher et al. would claim the last 24 months of life is the magic period in which we all become equally sick.

Just in case you can’t believe what you’re reading, let me repeat Fisher et al.’s assertion: Patients magically become equally sick at midnight of the day preceding the last two years or last six months of their lives; ergo, we do not need to risk adjust the expenditures on those patients: ergo we can say all differences in expenditures on those patients are caused by the “practice styles” of their doctors.

To my knowledge, no human being had ever before made that claim. But that’s what Fisher et al. said in 2003, and they’ve been saying it ever since. For example, in 2008 Fisher told the New York Times , “We are comparing patients [in different hospitals] with identical outcomes – all were dead in two years – so it’s unlikely that differences in the severity of illness account for the variations we saw.” [3]

As you can imagine, news of this exhilarating discovery at Dartmouth that risk-adjustment becomes unnecessary if you study dead people immediately drew criticism. Peter Bach et al., for example, published an article in JAMA late in 2004 with the biting title, “Resurrecting the treatment histories of dead patients: A study design that should be laid to rest.” The media paid little attention then. But when Bach published a similar paper  in the New England Journal of Medicine in February 2010, just a month before Obama signed the ACA, the media did pay attention. On June 2, 2010, the New York Times published an article very critical of the Dartmouth Atlas that appeared to have been stimulated by Bach’s paper.

I encourage readers to read the Bach and Times articles, and if you have the fortitude, the replies published by Fisher et al. But the dying-people-are-equally-sick claim is so patently foolish, and the rejoinders so obvious, that I believe that’s not necessary. It is sufficient to read my summary of Bach’s arguments below.

Bach , a physician and researcher at Memorial Sloan-Kettering, focused on Fisher et al.’s hospital research, but his criticism applies as well to their regional research. Bach discussed the two problems I noted in a previous post  about MACRA: The attribution problem (deciding which patients belong to which clinic or hospital) and the risk-adjustment problem (controlling for factors outside physician or hospital control).

Turning first to the attribution problem: Bach observed that Fisher et al. assumed that all decedents visited the same hospital during the final months or years of their lives, and that hospitals exercise control over all the doctors and other health care professionals patients visited. Your commonsense tells you this can’t be true, and Bach demonstrated it isn’t true. The “leakage” – the amount of money spent on Medicare patients outside the hospital to which Fisher et al. “attributed” them – is probably enormous, but since Fisher et al. don’t care about that problem, we don’t know how bad it is. Bach could only offer this example: “In one Atlas analysis, one third of the patients … had been admitted to the hospital in question only once.”

As for the risk-adjustment issue – “they all died, therefore risk adjustment isn’t necessary” – Bach’s rejoinder was the obvious one: Patients enter hospitals in varying states of health, and they also enter the last two years of life in varying states of health. Bach described calculations he did showing a two-fold difference in the risk of death among patients over 64 at the time of admission.

I have often wondered why Bach and other critics of the “they all died” rationale didn’t simply ask this question: If Fisher et al. think that human beings become equally sick when they cross over into the last six or 24 months of life, who’s to say they’re not all equally sick (or healthy) at five years before death? What the hell, how about the last 50 years of life? And if it’s true we’re all equally sick or healthy once we enter the last half-century or whatever of life, why risk adjust any cost or quality variable? We’re home free. No need anymore to do all that pesky and expensive (and grossly inaccurate) risk adjustment. We can just report raw data and base policy on that.

Medicare spending is not a proxy for total spending
Fisher et al.’s sloppy attribution and risk-adjustment methods should have been enough to discredit their claims about hospital and regional efficiency. But there was more criticism to come. After 2010, several excellent studies appeared that demonstrated there is no correlation between regional Medicare spending and regional non-Medicare spending. Should we be surprised? Medicare spending has accounted for only about a fifth of total spending for decades. If we ask what portion of spending Medicare recipients in their last two years of life account for, it’s even smaller. According to Bach, the decedents Fisher et al. used “account for less than $3 of every $100 spent on health care.”
I have already mentioned the IOM’s 2013 study. As of 2013 it was the most authoritative refutation of the Dartmouth group’s assumption that Medicare spending is a good proxy for total spending. Perhaps the best paper since 2013 on this subject was one by Zack Cooper et al. published last year. It found that the correlation between private-sector and Medicare spending at the regional level was a measly 0.14 percent. There is, in other words, no correlation. In some cases, Cooper et al. found more than mere non-correlation. They found that some of Fisher’s and Obama’s poster-child cities, including Grand Junction, Colorado and Minneapolis, looked expensive, not inexpensive, when measured with private-sector data. [4]

Fisher et al. remain unrepentant. They think their sloppy attribution and “they all died” risk-adjustment methods work just fine, and they have not retracted the claims they have made about regions and hospitals. [5] Here are the words that greet you when you open the Dartmouth Atlas  website today: “Understanding of [sic] the Efficiency and Effectiveness of the Health Care System.” Fisher et al. still want you to believe their Medicare data is trustworthy, and they still want you to think Medicare spending is a good proxy for all spending.

Next steps

The Dartmouth group did more than any other group of scholars to reinforce the myth that overuse of medical care is rampant and underuse is either non-existent or trivial. But they didn’t invent that myth. By the late 1970s and early 1980s, which is when the Dartmouth group began to make a name for itself, the managed care movement had created the folklore that overuse of medical care is the primary reason US health care costs are so high. It was this folklore, not any evidence published by the Dartmouth group or anyone else, that permitted Fisher et al. to say over and over that regional differences in Medicare reflected overuse in the expensive regions, not underuse in the less expensive regions. It was the overuse myth, coupled with the myth that Medicare spending is a good proxy for total spending, that allowed Fisher et al. to claim that “unnecessary care” accounts for 30 percent of total US health care spending.

In my next post, I will examine the overuse myth. I’ll present evidence indicating overuse is nowhere near as common as Fisher et al. make it out to be, and that underuse occurs more frequently than overuse. The evidence I will present will in turn raise the question, Can the fee-for-service system really be the root cause of health care inflation, as Fisher and their acolytes claim, if underuse occurs more often than overuse?

[1] The Berwick quote appears at p. 50 in Overtreated, Shannon Brownlee’s flattering history of the Dartmouth Institute.

[2] In Chapter 1 of its 2013 report http://www.ncbi.nlm.nih.gov/books/NBK201653/ on regional variation in Medicare spending, the Institute of Medicine described the “value index” legislation introduced in Congress, and then stated in a footnote that Fisher et al. “do not recommend the use of a geographically based value index.” This is true. But it’s like saying King Henry II’s comments about Thomas Becket were misconstrued.

[3] In a March 31, 2010 letter http://documents.nytimes.com/how-dartmouth-atlas-explains-its-methodology to the New York Times, Fisher and Skinner offered a different and very circular rationale for limiting their subjects to dead people. In that letter (see p. 3 of the attachment), they said they limited patients to those near the end of life because to study all admitted Medicare patients would mean hospitals that admit healthier patients would look better. Can you see the small circle Fisher et al. are trapped in? They justify one fact-free, arbitrary assumption with another. They argue that patients arbitrarily defined as near death are all equally sick, ergo those they arbitrarily defined as not near death are not equally sick. See how that works? Neither assumption works unless you’re inside the circle with them. So cling to the real world. Stay out of that circle.

For lack of space, I have not addressed a secondary defense Fisher et al. offer in response to those who criticize their “all the patients died” method of risk adjustment. I address it briefly here. In the same March 31, 2010 letter to the Times, they claim that “comparing what happens to heart attack or hip fracture patients across regions and hospitals represents one way to measure risk adjusted differences in spending” (p. 9 of the attachment). We could all agree with that statement if it referred only to “differences in spending on heart attack and hip fracture patients.” But that’s not what they say. Fisher et al. want us to pretend patients suffering heart attacks and hip fractures represent all patients. That is just one more evidence-free, arbitrary assumption. Just as Medicare is not a good proxy for all spending, so spending on heart attack and hip fracture patients is not a good proxy for all medical spending.

[4] In a December 15, 2015 article http://www.nytimes.com/interactive/2015/12/15/upshot/the-best-places-for-better-cheaper-health-care-arent-what-experts-thought.html about the Cooper study, the New York Times published a graphic comparing the Dartmouth Atlas map of Medicare spending with a map showing Cooper et al.’s private-sector-spending data. The maps bear almost no resemblance to each other. If I could show just one graphic to Obama and others who drank the Dartmouth Atlas Kool-Aid, it would be that one.

[5] The closest thing we have to a retraction from the Dartmouth group is this statement by Skinner about the Cooper study: “This idea that if the entire country turned into Grand Junction, that we’d suddenly save 20 percent on health spending, maybe that’s not totally true.” (See the New York Times article mentioned in footnote 4).