Why We Have so Little Useful Research on ACOs

flying cadeuciiOur country urgently needs research on the impact of “accountable care organizations” on cost and quality. The ACO has been the establishment’s great hope for health care reform since the concept was invented at the November 9, 2006 meeting of the Medicare Payment Advisory Commission. If ACOs are not going to work, we need to know sooner rather than later.

Although it’s been almost a decade since the ACO concept was invented and six years since Congress endorsed it, we know remarkably little about ACOs. What little reliable research we have was done on CMS’s ACO programs, but even that research is woefully incomplete. As for the ACOs set up by state Medicaid agencies and insurance companies, we know almost nothing.

Yes, I know, we have a few dozen papers telling us where ACOs are starting up, whether physicians or hospitals are “leading” them, and whether their managers tells pollsters they can “monitor care across the continuum” and “have programs in place to reduce hospital admissions,” etc. But we have no idea what ACOs do for patients that non-ACO providers do not do.

There are two reasons for this information vacuum. The first is the definition of the ACO. ACO proponents have never defined the ACO; they have told us only what they hope ACOs will do (they tell us they want ACOs to “hold providers accountable”). The second problem is the cavalier attitude toward evidence with which ACO proponents and analysts approach ACO research. Until the US health policy community addresses these problems, the dearth of useful research on ACOs will continue.

In this comment, I will describe these twin problems – the amorphous, aspiration-based definition of ACO, and the casual attitude toward evidence exhibited by ACO proponents and analysts. In Part II of this series I will illustrate these problems with a report on ACOs financed by the Robert Wood Johnson Foundation. The report, entitled “Accountable Care Organizations: Looking back and moving forward,”http://www.chcs.org/media/ACOs-Looking-Back-and-Moving-Forward.pdf was released last month by the Center for Health Care Strategies. In Part III I will argue that the vague definition of ACOs and the cavalier attitude toward evidence exhibited by ACO proponents is a result of a permissive culture that evolved first within the managed care movement and then spread throughout the American health policy community.

The unbearable vagueness of ACOs

The ACO has always been “defined” by the hopes of its proponents, and health policy analysts and editors have let them get away with it. Here is the typical definition of “ACO” taken from one of the earlier papers http://content.healthaffairs.org/content/29/5/982.abstract by ACO advocates, including Mark McClellan and Elliot Fisher: “ACOs consist of providers who are jointly held accountable for achieving measured quality improvements and reductions in the rate of spending growth. Our definition emphasizes that these cost and quality improvements must achieve overall, per capita improvements in quality and cost, and that ACOs should have at least limited accountability for achieving these improvements while caring for a defined population of patients.”

This definition contains the ingredients common to all ACO definitions, notably, language depicting a (poorly defined) group of providers being “held accountable” (by unidentified means by unidentified parties) for “measured improvements” (measured at an unknown cost to providers and the measurer) in the “cost and quality” of (unspecified) health care services delivered to a “population.” This definition tells us nothing about ACOs.

All we can divine from this “definition” are the authors’ hopes for the ACO. They want providers to be held “accountable,” and they want expenditures to fall and quality to go up. That’s like defining a drug according to the hopes of the drug manufacturer rather than by its ingredients and the expected action of those ingredients. The aspirational definition of the ACO tells us nothing about the services ACOs will provide to patients that non-ACO providers do not deliver, and it fails to identify a single mechanism that will cause these services to be delivered.[1]

The unbearable lightness of the ACO “definition” puts researchers in a quandary. How do they test the claims made for a concept that is defined only by what the concept’s promoters want the concept to achieve?

Here is how L&M Policy Research, one of CMS’s contractors, described this quandary in a report https://innovation.cms.gov/Files/reports/PioneerACOEvalRpt2.pdf evaluating the first two years of the Pioneer ACO program: “The ACO ‘treatment’ under investigation is not a prescribed set of activities or interventions. Rather, it is a financial arrangement in which provider organizations attempt to reduce expenditures below a set target while maintaining high quality metrics in exchange for bearing risk for reducing expenditures.” (p. 1) What? “Financial arrangement in which someone attempts….?” Can it get any vaguer than that?

How do investigators operationalize – reduce to testable components – this thing, this, um, “arrangement”? They can’t. Pity the poor contractor, such as L&M, who has to make up something to test. Here are the “features” of ACOs L&M dreamed up to “test” in its evaluation of the Pioneer ACO program:

  • “hospital relationships,”
  • “capacity to follow and monitor beneficiaries through the care continuum,”
  • “engaged leadership,”
  • “provider engagement,” and
  • “market pressures.”

Notice the promiscuous use of waffle words – “relationships,” “care continuum,” “engaged,” and “pressures.” How is “market pressure” a “feature” of an ACO?

Not surprisingly, L&M offered no useful definition of these “features” (with the possible exception of “hospital relationship”). Not surprisingly, L&M found no relationship between these five “features” and spending nor between them and patient satisfaction. And what if they had? What if “provider engagement” had been correlated? What would that have told us, for example, about what ACOs do to lower, say, hospital use for their “defined population,” or for particular types of patients, such as those with Alzheimer’s?

L&M’s useless research is not L&M’s fault. It’s just not possible to produce useful research about a concept defined by listing outcomes expected by the concept’s proponents. My only criticism of L&M’s report is that L&M failed to say what I’m saying – useful research on the ACO is not possible as long as the definition remains so amorphous. All we can do is measure cost and quality of the entities that call themselves ACOs, but since there is no “prescribed set of activities or interventions” for ACOs, as L&M put it, we can’t conclude that any particular activity or intervention caused whatever impact on costs or quality our “research” found.

The managed care culture and indifference to evidence-based policy

Why did the health policy community allow a concept as poorly defined as the ACO to acquire the status of conventional wisdom? Why didn’t the members of that community – scholars, members of Congress, employees of foundations such as the Commonwealth Fund and the Robert Wood Johnson Foundation, employers, and editors of professional journals – notice the problem and demand that it be addressed before the ACO skyrocketed from obscurity to national health policy?

That same health policy community would scream bloody murder if a drug company promoted a drug based on the aspirations of the drug company. The entire world, including health policy experts, would chortle if the drug company produced “research” analogous to L&M’s that found, for example, that the drug company’s “leadership” is “engaged.” We don’t care whether the drug manufacturer’s leadership is “engaged.” We want to know if the drug works, and if so by what mechanisms.

But the vast majority of health policy experts just can’t bring themselves to apply the same standard to a health policy proposal, in this case, to the ACO? Why is that?

To answer that question, we must put the culture of health policy experts under the microscope. What is it about the culture of the US health policy community – the incentives to which they are exposed, the attitudes of their more successful members, their articulated and unarticulated mores – that induce them to tolerate, even take seriously, a proposition as vaguely defined as the ACO?

In Part III of this series I will describe this culture and its origin in more detail. Here I will describe briefly the mores of health policy experts that enabled the rise of a concept as poorly defined and documented as the ACO, an event which in turn guaranteed useful research on ACOs would be extremely difficult if not impossible to conduct.

Since the birth of the managed care movement in the early 1970s, that movement has displayed a breezy indifference to the evidence for and against its claims (for example, the claim that fee-for-service payment causes “fragmentation” and overuse of medical services, and that HMOs are the solution to the alleged fragmentation and overuse). In a paper I co-authored with Ted Marmor that we delivered at Yale Law Schoolhttps://www.nasi.org/civicrm/event/info?reset=1&id=176 late in 2014, we identified three dysfunctional characteristics of the managed care movement: “[A] penchant for unnecessarily abstract concepts, labels designed to persuade rather than illuminate … , and assertions based on little or no evidence.”


The ACO fad illustrates all three characteristics:

  • The concept is very abstract;
  • its proponents gave it a label designed to manipulate rather than illuminate (it implies that the medical profession, one of the most regulated professions on the planet, is “unaccountable” and ACOs, at long last, are going to change that); and
  • there was very little evidence supporting the claims made for the ACO when it was sold to Congress and very little now.

These habits of thought were adopted by the first proponents of managed care in the 1970s, and spread rapidly throughout the health policy community, a community which emerged roughly simultaneously with the rise of the managed care movement.

The effect of these three habits is evident to some degree in nearly every paper published about ACOs. In Part II of this series I will illustrate these habits with an examination of a document that purports to be an up-to-date review of ACOs, “Accountable Care Organizations: Looking back and moving forward.” http://www.chcs.org/media/ACOs-Looking-Back-and-Moving-Forward.pdf This paper was described by its authors as a study designed to provide “lessons” about ACOs based on their “initial successes.” In fact, the paper provides no evidence of “successes” in controlling costs and only sparse evidence of “successes” in improving quality, and it provides no “lessons” – no useful feedback that would enable ACOs, analysts, or policy-makers to determine whether ACOs are working or failing and, if so, why,

Instead, the paper is an excellent illustration of why we have so little useful research on ACOs.

[1] The definition of the ACO in the Affordable Care Act is another example of a hope-based definition. Section 3022 states: “The ACO shall be willing to become accountable for the quality, cost, and overall care of the Medicare fee-for-service beneficiaries assigned to it.”

Categories: Uncategorized

Tagged as:

16 replies »

  1. ” Hadler argues for a clinical practice that draws on the best available scientific knowledge, transmits the wisdom of experienced clinicians, reforges an empathetic relationship between physician and patient, and treats each patient as an individual–all centered on restoring the mandate to care.”

    Wow, a voice of reason in the wilderness. Thank you Dr. Hadler.

  2. Interesting. I just bought your book (Kindle edition, a pricey tome at $26.60), and will be assessing and citing it on my KHIT blog (as I did with your prior book Citizen Patient).

    Margalit recently argued that “structured data” are actually INIMICAL to effective medical care.

  3. Yes, there are many unhappy members of the AAFP right now, and many are opting out of membership.

  4. And as usual, the comments contain more truth than the actual original post at the AAFP site. The frequently cited PCPCC is very deeply invested in PCMH, so failure is not an option. Evidence be damned.

  5. Excellent article and discussion.

    In the current issue of the Annals of Internal Medicine ( doi:10.7326/M14-2633) Kern et al have published the results of the 5-year Hudson Valley cohort study comparing the “Patient-centered medical home” approach to practice with practices that utilized just the EHR or retained their paper charts.The study was supported by the Commonwealth Fund and the NYS Department of Health. It recruited 438 primary care physicians in 226 practices, with 136 480 patients across 5 health plans.

    There was no clinically meaningful difference in health care quality or utilization between the 3 practice types.

    No one should be surprised. Everyone should be disappointed. Some might be upset that prescience drives policy more than science. I am of all these persuasions: http://uncpress.unc.edu/books/12931.html

  6. This is a terrific article. If indeed we seriously “put the culture of health policy experts under the microscope,” we might see that the experts have no clothes.

    I would add to Mr. Sullivan’s argument that what he describes probably predates even the managed care years, and can be traced back to the beginning of the so-called Progressive Era. The modern welfare state does not like that individual citizens be held accountable, preferring instead to put accountability on the shoulder of organizations.

    But organizations are not moral agents, people are. The “accountable organization” is a fiction, all the more so when large organization are invariably seen as “too big to fail.” That was true for AIG, GM, and the banks a few years ago, it will be true for large healthcare conglomerates in the years to come.

  7. Beware any agenda that bypasses science, bypasses the market, and proceeds directly to legislation.

  8. I believe studies on HMO’s have provided some of the research you desire concerning some of the dangerous incentives of the ACO. The Ware study “Differences in 4-Year Health Outcomes for Elderly and Poor, Chronically Ill Patients Treated in HMO and Fee-for-Service Systems” “Conclusions: During the study period, elderly and poor chronically ill patients had worse physical health outcomes in HMO’s than in FFS systems.”

    We can expect the same from ACO’s which have similar incentives to HMO’s. The bad incentive of FFS leads to over treatment such as an extra EKG where mostly no harm is done. The bad incentives of capitation and the ACO is denial of necessary treatment such as open heart surgery that can lead to an unnecessary death of a patient. In FFS there is a paper trail. In the other models there is no paper trail so that certain heinous methods to save money can not be proven.

  9. Not vague, really. ACO is like a shopping mall food court, a collection of profit centers (not sharing one common profit line).

    [via TweetBot]

  10. ACO useful in helping restrict overfragmentation of a health system but sharing learning should be essential

    [via TweetBot]

  11. Re GeriTechBlog’s statement:
    “Now, whether it was realistic to expect providers to design effective solutions”

    It isn’t the providers who ever design these models and systems that always move to centralization and more and more layers of control and oversight of the practicing docs….and increasingly attempting to monitor and control patients. ACO’s, EHR’s, Big Data analytics.

    What Kip Sullivan so effectively points out is that these grand efforts are faith and hope based initiatives….with wide implementation and acceptance as common wisdom without any evidence…..our legions of Ivy League educated system designers, all with such a primitive faith in their creations [actually, it is a variant of narcissism].

    There are promising countertrends….of course HSA’s….and Direct Primary Care….but they are vastly out numbered and out gunned by the big centralized system advocates.

  12. The author brings up a number of important points. But I’m not sure it’s so crazy for ACOs to be defined mainly based on the outcomes people are hoping for. Presumably this was meant to allow providers flexibility in how they were going to design a solution to achieve the desired outcomes.

    Now, whether it was realistic to expect providers to design effective solutions — short of converting themselves into tightly integrated inpatient/outpatient systems — is another story. I also wonder if it wasn’t a mistake to conflate population care with compassionate and cost-effective long-term care of individuals…seems to me they are not exactly the same thing.

  13. Is it any wonder physicians are frustrated? Many of the payment schemes now being worked up by CMS are going to concentrate on ACOs. So a vague organization with questionable accountability and no serious evidence behind it is in charge of dispersing payment (or withholding it) to doctors who will be judged on evidence-based practice and “quality measures”, etc, etc. Sounds like a great plan to me…