By KIP SULLIVAN

Our country urgently needs research on the impact of “accountable care organizations” on cost and quality. The ACO has been the establishment’s great hope for health care reform since the concept was invented at the November 9, 2006 meeting of the Medicare Payment Advisory Commission. If ACOs are not going to work, we need to know sooner rather than later.

Although it’s been almost a decade since the ACO concept was invented and six years since Congress endorsed it, we know remarkably little about ACOs. What little reliable research we have was done on CMS’s ACO programs, but even that research is woefully incomplete. As for the ACOs set up by state Medicaid agencies and insurance companies, we know almost nothing.

Yes, I know, we have a few dozen papers telling us where ACOs are starting up, whether physicians or hospitals are “leading” them, and whether their managers tells pollsters they can “monitor care across the continuum” and “have programs in place to reduce hospital admissions,” etc. But we have no idea what ACOs do for patients that non-ACO providers do not do.

There are two reasons for this information vacuum. The first is the definition of the ACO. ACO proponents have never defined the ACO; they have told us only what they hope ACOs will do (they tell us they want ACOs to “hold providers accountable”). The second problem is the cavalier attitude toward evidence with which ACO proponents and analysts approach ACO research. Until the US health policy community addresses these problems, the dearth of useful research on ACOs will continue.

In this comment, I will describe these twin problems – the amorphous, aspiration-based definition of ACO, and the casual attitude toward evidence exhibited by ACO proponents and analysts. In Part II of this series I will illustrate these problems with a report on ACOs financed by the Robert Wood Johnson Foundation. The report, entitled “Accountable Care Organizations: Looking back and moving forward,”http://www.chcs.org/media/ACOs-Looking-Back-and-Moving-Forward.pdf was released last month by the Center for Health Care Strategies. In Part III I will argue that the vague definition of ACOs and the cavalier attitude toward evidence exhibited by ACO proponents is a result of a permissive culture that evolved first within the managed care movement and then spread throughout the American health policy community.

The unbearable vagueness of ACOs

The ACO has always been “defined” by the hopes of its proponents, and health policy analysts and editors have let them get away with it. Here is the typical definition of “ACO” taken from one of the earlier papers http://content.healthaffairs.org/content/29/5/982.abstract by ACO advocates, including Mark McClellan and Elliot Fisher: “ACOs consist of providers who are jointly held accountable for achieving measured quality improvements and reductions in the rate of spending growth. Our definition emphasizes that these cost and quality improvements must achieve overall, per capita improvements in quality and cost, and that ACOs should have at least limited accountability for achieving these improvements while caring for a defined population of patients.”

This definition contains the ingredients common to all ACO definitions, notably, language depicting a (poorly defined) group of providers being “held accountable” (by unidentified means by unidentified parties) for “measured improvements” (measured at an unknown cost to providers and the measurer) in the “cost and quality” of (unspecified) health care services delivered to a “population.” This definition tells us nothing about ACOs.

All we can divine from this “definition” are the authors’ hopes for the ACO. They want providers to be held “accountable,” and they want expenditures to fall and quality to go up. That’s like defining a drug according to the hopes of the drug manufacturer rather than by its ingredients and the expected action of those ingredients. The aspirational definition of the ACO tells us nothing about the services ACOs will provide to patients that non-ACO providers do not deliver, and it fails to identify a single mechanism that will cause these services to be delivered.[1]

The unbearable lightness of the ACO “definition” puts researchers in a quandary. How do they test the claims made for a concept that is defined only by what the concept’s promoters want the concept to achieve?

Here is how L&M Policy Research, one of CMS’s contractors, described this quandary in a report https://innovation.cms.gov/Files/reports/PioneerACOEvalRpt2.pdf evaluating the first two years of the Pioneer ACO program: “The ACO ‘treatment’ under investigation is not a prescribed set of activities or interventions. Rather, it is a financial arrangement in which provider organizations attempt to reduce expenditures below a set target while maintaining high quality metrics in exchange for bearing risk for reducing expenditures.” (p. 1) What? “Financial arrangement in which someone attempts….?” Can it get any vaguer than that?

How do investigators operationalize – reduce to testable components – this thing, this, um, “arrangement”? They can’t. Pity the poor contractor, such as L&M, who has to make up something to test. Here are the “features” of ACOs L&M dreamed up to “test” in its evaluation of the Pioneer ACO program:

• “hospital relationships,”
• “capacity to follow and monitor beneficiaries through the care continuum,”
• “engaged leadership,”
• “provider engagement,” and
• “market pressures.”

Notice the promiscuous use of waffle words – “relationships,” “care continuum,” “engaged,” and “pressures.” How is “market pressure” a “feature” of an ACO?

Not surprisingly, L&M offered no useful definition of these “features” (with the possible exception of “hospital relationship”). Not surprisingly, L&M found no relationship between these five “features” and spending nor between them and patient satisfaction. And what if they had? What if “provider engagement” had been correlated? What would that have told us, for example, about what ACOs do to lower, say, hospital use for their “defined population,” or for particular types of patients, such as those with Alzheimer’s?

L&M’s useless research is not L&M’s fault. It’s just not possible to produce useful research about a concept defined by listing outcomes expected by the concept’s proponents. My only criticism of L&M’s report is that L&M failed to say what I’m saying – useful research on the ACO is not possible as long as the definition remains so amorphous. All we can do is measure cost and quality of the entities that call themselves ACOs, but since there is no “prescribed set of activities or interventions” for ACOs, as L&M put it, we can’t conclude that any particular activity or intervention caused whatever impact on costs or quality our “research” found.

The managed care culture and indifference to evidence-based policy

Why did the health policy community allow a concept as poorly defined as the ACO to acquire the status of conventional wisdom? Why didn’t the members of that community – scholars, members of Congress, employees of foundations such as the Commonwealth Fund and the Robert Wood Johnson Foundation, employers, and editors of professional journals – notice the problem and demand that it be addressed before the ACO skyrocketed from obscurity to national health policy?

That same health policy community would scream bloody murder if a drug company promoted a drug based on the aspirations of the drug company. The entire world, including health policy experts, would chortle if the drug company produced “research” analogous to L&M’s that found, for example, that the drug company’s “leadership” is “engaged.” We don’t care whether the drug manufacturer’s leadership is “engaged.” We want to know if the drug works, and if so by what mechanisms.

But the vast majority of health policy experts just can’t bring themselves to apply the same standard to a health policy proposal, in this case, to the ACO? Why is that?

To answer that question, we must put the culture of health policy experts under the microscope. What is it about the culture of the US health policy community – the incentives to which they are exposed, the attitudes of their more successful members, their articulated and unarticulated mores – that induce them to tolerate, even take seriously, a proposition as vaguely defined as the ACO?

In Part III of this series I will describe this culture and its origin in more detail. Here I will describe briefly the mores of health policy experts that enabled the rise of a concept as poorly defined and documented as the ACO, an event which in turn guaranteed useful research on ACOs would be extremely difficult if not impossible to conduct.

Since the birth of the managed care movement in the early 1970s, that movement has displayed a breezy indifference to the evidence for and against its claims (for example, the claim that fee-for-service payment causes “fragmentation” and overuse of medical services, and that HMOs are the solution to the alleged fragmentation and overuse). In a paper I co-authored with Ted Marmor that we delivered at Yale Law Schoolhttps://www.nasi.org/civicrm/event/info?reset=1&id=176 late in 2014, we identified three dysfunctional characteristics of the managed care movement: “[A] penchant for unnecessarily abstract concepts, labels designed to persuade rather than illuminate … , and assertions based on little or no evidence.”

http://digitalcommons.law.yale.edu/cgi/viewcontent.cgi?article=1231&context=yjhple

The ACO fad illustrates all three characteristics:

• The concept is very abstract;
• its proponents gave it a label designed to manipulate rather than illuminate (it implies that the medical profession, one of the most regulated professions on the planet, is “unaccountable” and ACOs, at long last, are going to change that); and
• there was very little evidence supporting the claims made for the ACO when it was sold to Congress and very little now.

These habits of thought were adopted by the first proponents of managed care in the 1970s, and spread rapidly throughout the health policy community, a community which emerged roughly simultaneously with the rise of the managed care movement.

The effect of these three habits is evident to some degree in nearly every paper published about ACOs. In Part II of this series I will illustrate these habits with an examination of a document that purports to be an up-to-date review of ACOs, “Accountable Care Organizations: Looking back and moving forward.” http://www.chcs.org/media/ACOs-Looking-Back-and-Moving-Forward.pdf This paper was described by its authors as a study designed to provide “lessons” about ACOs based on their “initial successes.” In fact, the paper provides no evidence of “successes” in controlling costs and only sparse evidence of “successes” in improving quality, and it provides no “lessons” – no useful feedback that would enable ACOs, analysts, or policy-makers to determine whether ACOs are working or failing and, if so, why,

Instead, the paper is an excellent illustration of why we have so little useful research on ACOs.

[1] The definition of the ACO in the Affordable Care Act is another example of a hope-based definition. Section 3022 states: “The ACO shall be willing to become accountable for the quality, cost, and overall care of the Medicare fee-for-service beneficiaries assigned to it.”