By KIP SULLIVAN

The latest report  on one of CMS’s “patient-centered medical home” (PCMH) demonstrations is more bad news for the “medical home” movement. According to the report, the second-year evaluation of Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration released by CMS on May 11, PCMHs are not cutting Medicare costs and are having almost no impact on quality.  Here is how the report summarized its findings on the eight states participating in the demonstration: “Our quantitative analysis [finds] very few consistent, favorable changes associated with the MAPCP Demonstration across the eight states.” (p. 11-6)

The MAPCP demo is one of three “medical home” demonstrations CMS has conducted. As of last May, CMS had released reports on two of them, the Comprehensive Primary Care Initiative and the FHQC Advanced Primary Care Practice Demonstration.

As I reported in an article posted here  on May 5, those reports indicated the PCMHs in those demos are having almost no impact on quality and may be raising Medicare’s costs.

The news that all three of CMS’s PCMH experiments are failing is also bad news for proponents of MACRA. The PCMH is one of the three “alternative payment models” that Congress and CMS are counting on to lower Medicare’s costs under MACRA. (ACOs and bundled payments are the other two.)

It is time to ask what we have learned from the “home” experiment, and either change it radically or abandon it. In this essay I will review the findings from the latest report on the MAPCP demo and comment on two features of the report that constitute clues to why PCMHs are failing: The high ratio of superficial commentary to useful data, and the absence of clearly defined independent variables. In two essays that follow this one I will ask what we can learn from the report.

Summary of the second-year evaluation of the MAPCP demo

On May 11, 2016, CMS released new information on the MAPCP demo in the form of two reports – the second-year evaluation of the demo  and the third-year evaluation.  Both were written by RTI International and two subcontractors. Because the second-year evaluation is the one that reports that PCMHs are failing to cut costs and improve quality, I will focus on that one. (The third-year evaluation presents data on the Medicare beneficiaries participating in the demo.)

The MAPCP demo is testing the PCMH notion in eight states. The demo has “multi-payer” in its title because multiple insurers – Medicare, Medicaid and insurance companies – agreed to help subsidize the PCMHs and to give them “feedback” on how they are doing. To participate in this demo, states had to have their own PCMH experiments underway, and they had to apply to CMS to add Medicare as one of the participating payers.

The second-year evaluation reports that the PCMHs have been unable to lower Medicare costs in seven of the eight states compared with non-PCMH clinics (see Table 2-9 pp. 2-38/2-39), and are having virtually no impact on the small handful of quality measures chosen for the demo (see Tables 2-6 and 2-7 pp. 2-28/29 and 2-30/31). Here are some relevant quotes:

“[O]ur analysis found hardly any evidence that the state initiatives were associated with reductions in utilization rates.” (p. 2-41)

“We observed mixed findings for process of care measures among the states.” (p. 2-26)

“For preventable hospitalizations, we found no significant differences in all eight states in terms of avoidable catastrophic events. For the prevention quality indicator composite measures, we found no differences in six out of the eight states and unfavorable findings in the remaining two states.” (p. 2-27)

Moreover, as is virtually always the case in analyses of “homes,” the cost measure RTI used to compare PCMHs to non-PCMHs did not take into account the fees CMS paid to PCMHs nor the expenses PCMHs incurred to set up and run “homes.” [1] If utilization rates are not falling, but CMS is subsidizing the “home” clinics and is incurring administrative costs to run this demo, that probably means the MAPCP demo is raising Medicare’s costs.

The report contains no information on whether the PCMHs cut costs or improved quality for Medicaid. It does mention dissatisfaction with PCMH performance among some insurance companies. The first-, second-, and third-year evaluations each promised a report on the PCMHs’ impact on Medicaid in a final report that will apparently be published in 2017.

There’s a pony in here somewhere

In all of three of the MAPCP evaluations, the ratio of uninformative commentary to useful data is astonishingly high. The authors wrap the few nuggets of useful information in great clouds of speculative observations, anecdotes, and subjective impressions conveyed with abstract and imprecise words. Take, for example, this remark at page 2-19 about “coaching”: “Many states offered practice coaching to some extent….” Say what? How many is “many”? What does it mean to coach “to some extent”? In the sentence that follows the one I just quoted, RTI informs us that the “coaches” in the unidentified states helped clinics apply for PCMH certification. But RTI does not tell us what wisdom the coaches shared with the doctors and nurses about how to make patients healthier. In short, this sentence about coaching is like junk food: You expend energy to consume it, but you get nothing in return.

I could fill hundreds of pages with other examples of similar fluff. Below are several other examples. Note the frequent use of abstract words like “activities,” “support,” “focus,” “team,” “enhanced,” and “transformation,” as well as words that convey gross imprecision such as “sometimes,” “some,” “many,” “several,” “others,” “most,” “varied,”  and “generally.” In the following quotes, I have italicized words that drain the quotes of intellectual nutrition:

“Care managers sometimes engaged in a variety of activities….” (p. 2-17)
“Practices … varied in the degree to which they effectively used utilization data provided by payers….” (p. 2-17)

“[T]here was great variation in the level of comfort practices had with their EHRs. In the second year, practices used their EHRs to support practice transformation with varying degrees of success.” (p. 2-18)

“Several practices described an increased focus on care plan development…. In most care plans, patients worked with providers to identify goals….” (p. 7-35) [2]

The second-year evaluation is 662 pages long; the third-year evaluation is 296 pages long. The useful information in these two reports could have been presented in 100 pages.

What are we testing?

At the beginning of all three evaluations, RTI makes the bold statement that they intend to find correlations between the “improved outcomes” RTI assumes the MAPCP demo will produce and independent variables RTI calls “features.” [3] They divide these variables into features of the states’ “initiatives” and features of the PCMH clinics. In the first- and third-year evaluations, RTI presents a long list of excellent questions they intend to ask about these features (this list appears in Appendix 1A in the first-year evaluation and Appendix A in the third-year evaluation). Examples of the research questions include, “What are the features of participating PCMHs?” (question 8) and, “What features of state initiatives are most responsible for the positive impacts seen?” (question 40).

But even though RTI says they will look for relationships between “features” and “improved outcomes,” nowhere does RTI explicitly identify the features they intend to treat as independent variables. The closest thing to a list of such variables is Figure 1-1 entitled “Conceptual framework for the MAPCP Demonstration evaluation” (p. 1-3). There we see boxes containing vague phrases with arrows pointing left-to-right to other boxes with vague phrases. The left-most box, entitled “MAPCP Demonstration Implementation,” contains six abstract phrases such as “practice-coaching/learning collaboratives” and “data systems/health IT/meaningful use,” while the box to the right of that one, entitled “Practice Transformation,” contains six vague and manipulative phrases including “whole-person orientation” and “commitment to quality and safety.” [4]

Let’s stop here and think for a moment about just two of the flabby phrases in RTI’s boxes – the “whole-person orientation” and “quality and safety” phrases. Do doctors who refuse to “transform” their clinics into PCMHs suffer from a “half-person orientation”? How is RTI supposed to distinguish between a whole-person orientation and a half-person orientation? Are non-PCMH doctors really not “committed to quality and safety”? How is RTI supposed to measure and document “commitment” or lack thereof? With a functional MRI? A lie-detector test?
If the twelve flabby concepts contained in those two boxes in Figure 1-1 constitute the independent variables RTI intends to examine, we should expect to see an explicit statement to that effect and some attempt by RTI to operationalize those flabby concepts, that is, to define them more precisely so that they can be measured. RTI does neither.

In fact, RTI informs us they can find virtually no uniformity in either the state “initiatives” or the PCMHs. RTI reports that the activities of the eight states and of the 785 participating PCMHs vary on every dimension imaginable. The state programs vary, for example, in how they define “homes,” the amount of money they pour into them, and the type of feedback they provide. PCMHs vary across and within states in the type of staff they hire, what those staff do, how they use technology, whether they attempt to track admissions of their patients, and whether they can make heads or tails of their “feedback” reports. [5]

Conclusion

RTI should never have promised CMS and its readers it would identify factors associated with “outcomes,” improved or otherwise. The definition of the PCMH offered by leading “home” advocates is flabby enough. Trying to test such a wobbly concept in eight states when each state is allowed to use a different definition is just a wild goose chase.

Is there anything at all we can learn from such a poorly designed experiment? I’ll attempt to answer that question in subsequent comments.

[1] A bizarre feature of this report is the use of two control groups. RTI, perhaps at CMS’s insistence, set up one control group consisting of non-PCMHs, as you might expect, and a second control group consisting of PCMHs not participating in the MAPCP. Setting up a second control group of PCMHs that only differed from the experimental group of PCMHs because Medicare was not one of the participating insurers makes no sense, especially given how poorly PCMHs are defined.

RTI then used the results of the comparison with the PCMH control group in a misleading manner. First, RTI conflated the results of the comparisons with the two control groups in the following summary of its findings on spending: “The state initiatives were associated with a slower rate of growth of total Medicare expenditures in only three of the eight MAPCP Demonstration states (Vermont, New York, and Michigan).” (p. 2-41) Upon inspecting the table that this remark refers to (Table 2-9), the reader learns that only in Vermont did PCMHs beat non-PCMHs. In the other two states, New York and Michigan, MAPCP PCMHs beat the other PCMH “control” group, but were unable to beat the non-“homes.” Is that not strange? Does it not require an explanation? RTI offered none.

RTI deepened the murk by using only the non-MAPCP PCMH comparison group to estimate whether CMS enjoyed any net savings from the MAPCP demo, that is, whether the MAPCP “homes” lowered or raised Medicare’s costs when the fees CMS paid the “home” clinics were taken into account. RTI determined that only Michigan’s MAPCP PCMHs achieved net savings for Medicare compared with other PCMHs in Michigan who were not in the demo. But who cares? We want to know whether the PCMHs in the demo achieved net savings vis a vis non-PCMHs. RTI made no attempt to explain why the reader should care whether one set of PCMHs outscored another. We are left to wonder whether PCMHs in any state achieved net savings, or caused a net increase in Medicare costs, compared with non-PCMHs.

[2] Here are other examples of uninformative chit-chat from the second-year evaluation of the MAPCP demo. I have italicized words that obscure rather than enlighten:

“In Year Two, participating practices continued to mature, demonstrating transformation progress.” (p. 5-11)

“The majority of the practices described an environment in which doctors … and staff members worked together as a team, supporting each other and providing enhanced patient care. Many practices held weekly meetings to talk about areas for improvement within the practice….” (p. 5-21)

“Interviewees generally believed that by continuing to focus on these chronic conditions, the state might be able to bend the cost curve and lower utilization.” (p. 5-35)

“Some practice staff indicated that the care coordinators had succeeded in reducing unnecessary ER visits….. Many clinics shared anecdotes about patients who visited the ER….” (p. 7-37)

[3] Here is how RTI articulated their aspirations at the outset of the second-year evaluation: “The goal of the evaluation is to identify features of the state initiatives or the participating PCMH practices that are positively associated with improved outcomes.” (p. 1-2. Emphasis supplied.)

[4] Here are the six vague phrases that appear in the “MAPCP Demonstration Implementation” box in Figure 1-1:

Patient-centered medical home certification;
Payments to practices from Medicare, Medicaid and private insurers;
Practice coaching/learning collaborative;
Data systems/health IT/meaningful use;
Feedback to practices;
Integration of community-based resources.

Here are the six vague phrases which appear in the “Practice Transformation” box:

Personal physician;
Physician-directed medical practice;
Whole-person orientation;
Coordinated or integrated care;
Commitment to quality and safety;
Enhanced access to care.

[5] For a more excruciating description of all the ways state PCMH programs and the PCMHs vary, see my comment on the first-year MAPCP evaluation here. http://pnhp.org/blog/2015/03/05/cms-state-sponsored-medical-home-experiment-is-a-mess