CMS’s Latest Report Is Bad News For Medical Homes

flying cadeuciiThe latest report  on one of CMS’s “patient-centered medical home” (PCMH) demonstrations is more bad news for the “medical home” movement. According to the report, the second-year evaluation of Multi-Payer Advanced Primary Care Practice (MAPCP) Demonstration released by CMS on May 11, PCMHs are not cutting Medicare costs and are having almost no impact on quality.  Here is how the report summarized its findings on the eight states participating in the demonstration: “Our quantitative analysis [finds] very few consistent, favorable changes associated with the MAPCP Demonstration across the eight states.” (p. 11-6)

The MAPCP demo is one of three “medical home” demonstrations CMS has conducted. As of last May, CMS had released reports on two of them, the Comprehensive Primary Care Initiative and the FHQC Advanced Primary Care Practice Demonstration.

As I reported in an article posted here  on May 5, those reports indicated the PCMHs in those demos are having almost no impact on quality and may be raising Medicare’s costs.

The news that all three of CMS’s PCMH experiments are failing is also bad news for proponents of MACRA. The PCMH is one of the three “alternative payment models” that Congress and CMS are counting on to lower Medicare’s costs under MACRA. (ACOs and bundled payments are the other two.)

It is time to ask what we have learned from the “home” experiment, and either change it radically or abandon it. In this essay I will review the findings from the latest report on the MAPCP demo and comment on two features of the report that constitute clues to why PCMHs are failing: The high ratio of superficial commentary to useful data, and the absence of clearly defined independent variables. In two essays that follow this one I will ask what we can learn from the report.

Summary of the second-year evaluation of the MAPCP demo

On May 11, 2016, CMS released new information on the MAPCP demo in the form of two reports – the second-year evaluation of the demo  and the third-year evaluation.  Both were written by RTI International and two subcontractors. Because the second-year evaluation is the one that reports that PCMHs are failing to cut costs and improve quality, I will focus on that one. (The third-year evaluation presents data on the Medicare beneficiaries participating in the demo.)

The MAPCP demo is testing the PCMH notion in eight states. The demo has “multi-payer” in its title because multiple insurers – Medicare, Medicaid and insurance companies – agreed to help subsidize the PCMHs and to give them “feedback” on how they are doing. To participate in this demo, states had to have their own PCMH experiments underway, and they had to apply to CMS to add Medicare as one of the participating payers.

The second-year evaluation reports that the PCMHs have been unable to lower Medicare costs in seven of the eight states compared with non-PCMH clinics (see Table 2-9 pp. 2-38/2-39), and are having virtually no impact on the small handful of quality measures chosen for the demo (see Tables 2-6 and 2-7 pp. 2-28/29 and 2-30/31). Here are some relevant quotes:

“[O]ur analysis found hardly any evidence that the state initiatives were associated with reductions in utilization rates.” (p. 2-41)

“We observed mixed findings for process of care measures among the states.” (p. 2-26)

“For preventable hospitalizations, we found no significant differences in all eight states in terms of avoidable catastrophic events. For the prevention quality indicator composite measures, we found no differences in six out of the eight states and unfavorable findings in the remaining two states.” (p. 2-27)

Moreover, as is virtually always the case in analyses of “homes,” the cost measure RTI used to compare PCMHs to non-PCMHs did not take into account the fees CMS paid to PCMHs nor the expenses PCMHs incurred to set up and run “homes.” [1] If utilization rates are not falling, but CMS is subsidizing the “home” clinics and is incurring administrative costs to run this demo, that probably means the MAPCP demo is raising Medicare’s costs.

The report contains no information on whether the PCMHs cut costs or improved quality for Medicaid. It does mention dissatisfaction with PCMH performance among some insurance companies. The first-, second-, and third-year evaluations each promised a report on the PCMHs’ impact on Medicaid in a final report that will apparently be published in 2017.

There’s a pony in here somewhere

In all of three of the MAPCP evaluations, the ratio of uninformative commentary to useful data is astonishingly high. The authors wrap the few nuggets of useful information in great clouds of speculative observations, anecdotes, and subjective impressions conveyed with abstract and imprecise words. Take, for example, this remark at page 2-19 about “coaching”: “Many states offered practice coaching to some extent….” Say what? How many is “many”? What does it mean to coach “to some extent”? In the sentence that follows the one I just quoted, RTI informs us that the “coaches” in the unidentified states helped clinics apply for PCMH certification. But RTI does not tell us what wisdom the coaches shared with the doctors and nurses about how to make patients healthier. In short, this sentence about coaching is like junk food: You expend energy to consume it, but you get nothing in return.

I could fill hundreds of pages with other examples of similar fluff. Below are several other examples. Note the frequent use of abstract words like “activities,” “support,” “focus,” “team,” “enhanced,” and “transformation,” as well as words that convey gross imprecision such as “sometimes,” “some,” “many,” “several,” “others,” “most,” “varied,”  and “generally.” In the following quotes, I have italicized words that drain the quotes of intellectual nutrition:

“Care managers sometimes engaged in a variety of activities….” (p. 2-17)
“Practices … varied in the degree to which they effectively used utilization data provided by payers….” (p. 2-17)

“[T]here was great variation in the level of comfort practices had with their EHRs. In the second year, practices used their EHRs to support practice transformation with varying degrees of success.” (p. 2-18)

“Several practices described an increased focus on care plan development…. In most care plans, patients worked with providers to identify goals….” (p. 7-35) [2]

The second-year evaluation is 662 pages long; the third-year evaluation is 296 pages long. The useful information in these two reports could have been presented in 100 pages.

What are we testing?

At the beginning of all three evaluations, RTI makes the bold statement that they intend to find correlations between the “improved outcomes” RTI assumes the MAPCP demo will produce and independent variables RTI calls “features.” [3] They divide these variables into features of the states’ “initiatives” and features of the PCMH clinics. In the first- and third-year evaluations, RTI presents a long list of excellent questions they intend to ask about these features (this list appears in Appendix 1A in the first-year evaluation and Appendix A in the third-year evaluation). Examples of the research questions include, “What are the features of participating PCMHs?” (question 8) and, “What features of state initiatives are most responsible for the positive impacts seen?” (question 40).

But even though RTI says they will look for relationships between “features” and “improved outcomes,” nowhere does RTI explicitly identify the features they intend to treat as independent variables. The closest thing to a list of such variables is Figure 1-1 entitled “Conceptual framework for the MAPCP Demonstration evaluation” (p. 1-3). There we see boxes containing vague phrases with arrows pointing left-to-right to other boxes with vague phrases. The left-most box, entitled “MAPCP Demonstration Implementation,” contains six abstract phrases such as “practice-coaching/learning collaboratives” and “data systems/health IT/meaningful use,” while the box to the right of that one, entitled “Practice Transformation,” contains six vague and manipulative phrases including “whole-person orientation” and “commitment to quality and safety.” [4]

Let’s stop here and think for a moment about just two of the flabby phrases in RTI’s boxes – the “whole-person orientation” and “quality and safety” phrases. Do doctors who refuse to “transform” their clinics into PCMHs suffer from a “half-person orientation”? How is RTI supposed to distinguish between a whole-person orientation and a half-person orientation? Are non-PCMH doctors really not “committed to quality and safety”? How is RTI supposed to measure and document “commitment” or lack thereof? With a functional MRI? A lie-detector test?
If the twelve flabby concepts contained in those two boxes in Figure 1-1 constitute the independent variables RTI intends to examine, we should expect to see an explicit statement to that effect and some attempt by RTI to operationalize those flabby concepts, that is, to define them more precisely so that they can be measured. RTI does neither.

In fact, RTI informs us they can find virtually no uniformity in either the state “initiatives” or the PCMHs. RTI reports that the activities of the eight states and of the 785 participating PCMHs vary on every dimension imaginable. The state programs vary, for example, in how they define “homes,” the amount of money they pour into them, and the type of feedback they provide. PCMHs vary across and within states in the type of staff they hire, what those staff do, how they use technology, whether they attempt to track admissions of their patients, and whether they can make heads or tails of their “feedback” reports. [5]


RTI should never have promised CMS and its readers it would identify factors associated with “outcomes,” improved or otherwise. The definition of the PCMH offered by leading “home” advocates is flabby enough. Trying to test such a wobbly concept in eight states when each state is allowed to use a different definition is just a wild goose chase.

Is there anything at all we can learn from such a poorly designed experiment? I’ll attempt to answer that question in subsequent comments.

[1] A bizarre feature of this report is the use of two control groups. RTI, perhaps at CMS’s insistence, set up one control group consisting of non-PCMHs, as you might expect, and a second control group consisting of PCMHs not participating in the MAPCP. Setting up a second control group of PCMHs that only differed from the experimental group of PCMHs because Medicare was not one of the participating insurers makes no sense, especially given how poorly PCMHs are defined.

RTI then used the results of the comparison with the PCMH control group in a misleading manner. First, RTI conflated the results of the comparisons with the two control groups in the following summary of its findings on spending: “The state initiatives were associated with a slower rate of growth of total Medicare expenditures in only three of the eight MAPCP Demonstration states (Vermont, New York, and Michigan).” (p. 2-41) Upon inspecting the table that this remark refers to (Table 2-9), the reader learns that only in Vermont did PCMHs beat non-PCMHs. In the other two states, New York and Michigan, MAPCP PCMHs beat the other PCMH “control” group, but were unable to beat the non-“homes.” Is that not strange? Does it not require an explanation? RTI offered none.

RTI deepened the murk by using only the non-MAPCP PCMH comparison group to estimate whether CMS enjoyed any net savings from the MAPCP demo, that is, whether the MAPCP “homes” lowered or raised Medicare’s costs when the fees CMS paid the “home” clinics were taken into account. RTI determined that only Michigan’s MAPCP PCMHs achieved net savings for Medicare compared with other PCMHs in Michigan who were not in the demo. But who cares? We want to know whether the PCMHs in the demo achieved net savings vis a vis non-PCMHs. RTI made no attempt to explain why the reader should care whether one set of PCMHs outscored another. We are left to wonder whether PCMHs in any state achieved net savings, or caused a net increase in Medicare costs, compared with non-PCMHs.

[2] Here are other examples of uninformative chit-chat from the second-year evaluation of the MAPCP demo. I have italicized words that obscure rather than enlighten:

“In Year Two, participating practices continued to mature, demonstrating transformation progress.” (p. 5-11)

“The majority of the practices described an environment in which doctors … and staff members worked together as a team, supporting each other and providing enhanced patient care. Many practices held weekly meetings to talk about areas for improvement within the practice….” (p. 5-21)

“Interviewees generally believed that by continuing to focus on these chronic conditions, the state might be able to bend the cost curve and lower utilization.” (p. 5-35)

“Some practice staff indicated that the care coordinators had succeeded in reducing unnecessary ER visits….. Many clinics shared anecdotes about patients who visited the ER….” (p. 7-37)

[3] Here is how RTI articulated their aspirations at the outset of the second-year evaluation: “The goal of the evaluation is to identify features of the state initiatives or the participating PCMH practices that are positively associated with improved outcomes.” (p. 1-2. Emphasis supplied.)

[4] Here are the six vague phrases that appear in the “MAPCP Demonstration Implementation” box in Figure 1-1:

Patient-centered medical home certification;
Payments to practices from Medicare, Medicaid and private insurers;
Practice coaching/learning collaborative;
Data systems/health IT/meaningful use;
Feedback to practices;
Integration of community-based resources.

Here are the six vague phrases which appear in the “Practice Transformation” box:

Personal physician;
Physician-directed medical practice;
Whole-person orientation;
Coordinated or integrated care;
Commitment to quality and safety;
Enhanced access to care.

[5] For a more excruciating description of all the ways state PCMH programs and the PCMHs vary, see my comment on the first-year MAPCP evaluation here. http://pnhp.org/blog/2015/03/05/cms-state-sponsored-medical-home-experiment-is-a-mess

Categories: Uncategorized

Tagged as:

4 replies »

  1. PCMH certification/recognition is another example of how disconnected form and function become when non-clinicians far removed from a patient population try to micromanage how trained professionals treat those patients. The rigidity of PCMH standards defies the very meaning of patient centeredness.

    Just one example: Before PCMH, practices that accommodated many requests for same day appointments by “double booking” earned patient loyalty. Now, all we have to do to qualify as a PCMH is keep a single open slot in our schedule!

    One fundamental problem with the current incarnation of PCMH is that it isn’t physician led. Physicians are still seeing a patient every fifteen minutes with little or no time to participate in, let alone lead, any population management efforts.

    PCMH certification is a lot like designing a car without really knowing what it’s like to drive one:


  2. Unfortunately it will take many more nails in the coffin before the PCMH crowd will relent. The only way to actually understand it to the degree you have taken it, is to suffer under NCQA PCMH implementation or certification. The promoters repeatedly resort to “cloud talk” as you have pointed out. This is to make up for mountains of accumulating evidence of no benefit, and routine ignorance of costs. The processes are overwhelming, intrusive, and frequently unnecessary. So there is no surprise that costs are up and utilization has not budged. Also no wonder that payers are unimpressed and unwilling to continue to fund this monstrosity anywhere near the level to maintain it. CMS has demonstrated some wisdom in deciding this is nowhere close to where they were going with the APM model. In PCMH world, care coordination is way overplayed. Expecting cost savings to arise when you have to hire people to undo systemic inefficiencies created by other people that are systemically entrenched to create them, is absurd. In the meantime, the NCQA is busy at work entertaining more process, more special interests, more guidelines. This is gumming up the machinery in Primary Care clinics. People have to go elsewhere because they can’t go to the doctor for a simple problem anymore without wading through “process centered” care. There may as well not even be a “chief complaint” anymore, these visits are not about patients, they are about maintaining certification. So now, PCMH proponents are going to torture years of existing data demonstrating “this isn’t working” until they find some morsel they can continue to sell. An atrocious amount of time and effort has been wasted on this effort that will never be accounted for or recovered. Certainly, it won’t be figured into “costs”. How many administrators does it take to save 1 dollar? Health care costs will never come down if we continue to fund the rise of administration.

  3. Thanks, Kip, for diving into this muck and explaining it so well to the group. You must feel as if you have to take a shower afterward.

    Anyway, the science is moving along and I think we are going to be treated with marvelous news about aging within a few years. This will distract us from this endless focus on value… because everyone will be trying to access this new technology and trying to figure out how to pay for it. We will refocus just as we would if an asteroid was found to be targeting earth.

  4. In 1969, the cost of our nation’s healthcare was @ 6% of the national economy. Currently, it is about 18% of our nation’s economy. All of the other developed nation’s of the world (except The Netherlands) spend 12% or less of their economy on healthcare, with better effectiveness. Our nation’s maternal mortality ratio in 2013 was 17 deaths per 100,000 live births. We would need to reduce this death rate by 75% to rank among the best 10 nations of the world. With reasonable assumptions, nearly 500 women died with a pregnancy in 2013 that would be still be alive now, if they had lived in Denmark, Ireland, Sweden, Spain….. Remember, the life-time economic value of a mother to her family has been estimated a $1 Million.
    The increasing cost of healthcare for more than 50 years follows Parkinson’s Law perfectly. And, there is no reason to believe that the current strategy for healthcare reform will fix its high cost and highly variable effectiveness. Recently, I encountered the Power Law Distribution curve and noted that it offered a possible view for an alternative analysis. You may recall that the Power Law Distribution curve indicates that 20% of the users consume 80% of the resources and 80% of the users consume 20% of the energy. For healthcare, @5% of citizens use 50% of healthcare, the next 25% of citizens use 20% of healthcare and lowest 50% of citizens use 5% of healthcare. In essence, we have a highly focused commitment to manage the cost of the top 25% of healthcare users with almost no overall strategy for 50% of our nation’s citizens.
    Remember, the effort to stabilize health is different. I propose that without a community by community effort (@400,000 citizens each) to focus on the overall HEALTH of it own community, cost savings will be futile as any citizen ages from an episodic cough (minor illnesses) to chronic lung disease (chronic illnesses) later on. The Design Principles for managing a common-pool resource would apply, as in the number of hospital days used monthly by the community. A community’s HEALTH needs should also include an assessment process for promoting the local provision of available and accessible Primary Healthcare for each citizen. A model for this already exists for the agriculture industry, viz., Cooperative Extension Service as established by the Smith-Lever Act passed by Congress in 1914. Please note that our nation’s agriculture industry produces more food with resources applied than any other nation, by a wide margin. Simply stated, the strategy with the most well-proven knowledge base for healthcare reform can be found in the research defined by Professor Elinor Ostrom (2009 Nobel Prize winner) along with many colleagues.
    See http://www.nationalhealthusa.net for one alternative strategy.