Apples to Grapefruits

Last week, I commented on a New York Times story that appeared Wednesday, June 2, attacking the Dartmouth Research.  The work that Dartmouth has done over the past two decades suggests that hospitals in some parts of the country are over-treating patients. Over-treatment means that patients who didn’t need to be in the hospital in the first place are exposed to the side effects of treatment as well as gruesome hospital- acquired infections, medication mix-ups and a host of other medical errors. Thus unnecessary care puts patients at risk while helping to drive health care bills heavenward— and suggests that we could rein in Medicare spending by squeezing some of that hazardous waste out of the system.  But according to the Times: “Data [from Dartmouth] Used to Justify Health Savings Effort is Sometimes Shaky.”

In Part 1 of this post I discussed what two of the Times’ sources told me about how the Times’ reporters misrepresented what they said. Both Harvard economist David Cutler and Yale’s Dr. Harlan M. Krumholz complained that the story made it seem that they are critics of the research, when in fact they agree with Dartmouth on the basic message of the data, and see the work as, in Krumholz’ words “pivotal to moving us forward  . . . we all agree that there is lots of waste and it is unevenly distributed across the country.” A third source in Washington D.C. who talked to the Times reporters confided that they seemed to have a clear agenda: “to take down Dartmouth.”

Today, I received evidence from yet another unhappy source—the Wisconsin Collaborative for HealthCare Quality, a voluntary consortium of organizations working to improve the quality and cost-effectiveness of healthcare in Wisconsin. Chris Queram, the Collaborative’s president, and Jack Bowhan, who guides the development of value metrics for the group, report that they tried to caution New York Times reporter Gardiner Harris that he was misusing their data,   “comparing apples to grapefruits,” and “jumping to a conclusions that  you just can’t make.”  Harris ignored their warnings.

As proof, they produced a series of e-mails that they sent to Harris, and with their permission, I’m quoting from those messages. But first, an excerpt from the Times’ story talking about the Collaborative’s data.

“Last June, as Mr. Obama campaigned for his health care overhaul, he visited Green Bay, Wis., praising the city for getting “more quality out of fewer health care dollars than many other communities. Last June, as Mr. Obama campaigned for his health care overhaul, he visited Green Bay, Wis., praising the city for getting “more quality out of fewer health care dollars than many.

“Two of Green Bay’s hospitals, Bellin and St. Mary’s Hospital Medical Center, rank fourth and 11th within Wisconsin on the Dartmouth list.

“But again, Dartmouth ranks hospitals only by costs and number of treatments and procedures. A different picture emerges from work done by the Wisconsin Collaborative for Healthcare Quality, a voluntary group of health care organizations that uses both price and quality of care measures. In an analysis of heart attack care, for example, it ranks Bellin second, and St. Mary’s 15th, among the 22 hospitals in the state.

“And a Medicare ranking based on its own data that shows how many people die after treatment for certain conditions — statistics that exclude costs entirely — puts Bellin fifth, but drops St. Mary’s to second-to-last: 67th of the 68 hospitals statewide that were measured by both Dartmouth and Medicare.

“Do the Green Bay hospitals favored by Dartmouth really offer better care? Maybe not.”

The e-mail Trail

Here is the statement from the Collaborative that I received today:

The Wisconsin Collaborative for Healthcare Quality was contacted by Gardiner Harris (GH) on March 30th seeking information about the comparison of Dartmouth Atlas rankings versus WCHQ quality rankings. That is, “- do the same systems that show positive performance on Dartmouth data show the same performance on your data? Put another way; are these systems as good as Dartmouth says they are using other measures?”

There are two series of email exchanges with GH [Gardiner Harris] ; one set on March 30-31st and again on April 19-20th. Throughout the emails, GH was cautioned not to use WCHQ’s data and methodology for comparison to ranking results generated by the Dartmouth Atlas. Examples of those cautions from the Collaborative’s Chris Queram (CQ)  and Jack Bowhan (JB) include:

CQ, 3/31/10, 10:47AM – “our data relate to physician groups, Dartmouth’s relates to hospitals. And, the conditions being measured are different in many cases. So, the comparisons are very limited and should not be used to cast aspersions on the Dartmouth data.”

JB, 3/31/10, 1:10 PM –”There really is no way to reasonably compare the WCHQ metrics against Dartmouth and its process.“

CQ, 3/31/10, 1:53 PM – “I think you are raising an important issue, but want to be sure not to cast the [Dartmouth] Atlas in an unfair light.”

JB, 3/31/10, 5:33 PM – “I think you are jumping to a conclusion you cannot make. Here is why – I don’t think you can say Dartmouth is, or is not, a good proxy when you are trying to compare apples to grapefruits. We need an apples-to-apples comparison…”

CQ, 3/31/10, 6:59 PM – “It does make me nervous that decades worth of research by the team at Dartmouth might be impugned by the differences show in our hospital quadrants. Our methods while sufficient to enable our members to feel comfortable reporting this data and using it to guide internal improvement efforts — has not been the subject of or withstood rigorous scientific evaluation to confirm the association / correlation between the data reported on the two axis.”

Maggie Mahar is an award winning journalist and author. A frequent contributor to THCB, her work has appeared in the New York Times, Barron’s and Institutional Investor. She is the author of Money-Driven Medicine: The Real Reason Why Healthcare Costs So Much,” an examination of the economic forces driving the health care system. A fellow at the Century Foundation, Maggie is also the author the increasingly influential HealthBeat blog, one of our favorite health care reads, where this piece first appeared.

9 replies »

  1. Non-surgical treatment includes orthotics, ultrasound therapy (3 MHz
    each day for 10-15 minutes), use of night splints that protect the foot all night by
    wrapping it in a special cushion and lastly, use of Nonsteroidal anti-inflammatory drugs (NSAIDs).
    Once you can, move on to doing this exercise
    as you are standing up. Plantar fasciitis occurs once the weight that is applied to the foot is so excellent that the tension within the plantar fascia increase, causing damage
    as it begins to take away from the heel bone tissue.

  2. Maggie,
    What is your explanation for the detailed response from Ms. Abelson and Mr. Harris, which still fails to note the most damning aspect of the Dartmouth analysis?

  3. Sorry, I know Ms Mahar probably hates to see my name in the comment section in her posts, but, once again, I just ask the readers to step back and think what is the agenda this blog author has in her repeated posts here!
    I really don’t think this woman has any other agenda but just selling the false message of the failed health care reform legislation that Washington is trying to force on American citizens. And I know, in my heart, she will disappear once the truth is fully exposed!
    Hey people, there is truth, and there is agenda. If they really overlap, the dissenters are exposed and responsibly refuted. A lot of my comments and links seem to be ignored here. Is that because taking the risk of denying the truth gets outed? Well, you folks who read here and are looking for what is right and responsible, pay attention to the deeds and not the words. I practice health care interventions every day.
    Ask Ms Mahar what she does for a living. You may find it interesting if she tells you the truth!

  4. Maggie,
    Having actually supervised and reviewed the data, including the all-payer Texas file, I can assure you that I am not wrong. If you read the post, you can see it for yourself.
    I am interested that you are interpreting Skinner’s comments, which were actually around whether price was relevant in a value discussion.
    I would suggest that geography and demography are destiny, which will do as good a job in explaining IHC and Geisinger and Green Bay as the Dartmouth research.

  5. Hal–
    You are mistaken on all counts.
    First, you seem to think that there is only one study using 2005 data. Dartmouth has been doing this reserach since the 1970s. The Atlas data goes back many years. The reserach–and refinement continues today.
    The places where patients were getting more aggressive, more intensive treatment in 2005 are the places where they were getting more intensive expensive treatment in 2000 and in 1995 . . .and today. Study after study slices into the data in different ways, and comes to the same conclusion.
    Different Dartmouth studies adjust for different things. In those studies where they are comparing the cost of care they DO adjust for differences in local prices.
    Harvard’s David Cutler told the NYT reporters that the DArtmouth folks adjust for prices–they just didn’t want to hear it.
    In other studies, Dartmouth doesn’t compare how much money Medicare spends on very similar patietns in diferent hospitals , but rather the volume of medical resrouces that are spent on patients: How many specialists see them during the final two years of life? How many days do they spend in the hospital? How many tests do they undergo? How many procedures? What percent die in an ICU? Obviously, in these cases you don’t have to adjust for differences in local prices because you are not comparing how much was spent. (This is what Skinner meant when he told you prices are irrelvant.)
    Many Dartmouth studies adjust for age, sex, race,income, underlying health of the population, and severity of illness.
    The Dartmouth data is not tagged to zip codes where people live, it is sorted by hospital referral regions–the rather large region surrounding a hospital that the hospital services. It includes many zip codes.
    The Goverment Acccounting Office (GAO) follows the Dartmouth model in using hospital referral regions to track patient care, and GAO explains
    . Each Hospital Service Area is examined to determine where most of its residents went for these services. The result was the aggregation of the over 3,000 Hospital Service Areas into 306 Hospital Referral Regions.
    Most Americans get their care at hospitals close to home.

  6. It seems all the people opposed to the NY article oppose the results not the opinion.
    “It does make me nervous that decades worth of research by the team at Dartmouth might be impugned by the differences show in our hospital quadrants.”
    ” Both Harvard economist David Cutler and Yale’s Dr. Harlan M. Krumholz complained that the story made it seem that they are critics of the research, when in fact they agree with Dartmouth on the basic message of the data”
    We don’t care if they are critics or not, if data or studies they did cast a bad light on the accuracy and value of dartmouth studies their personal opinions or goals have no relavance on that.
    This is like the global warming fanatics crying when their own data is used against them.
    Curious how do you get treatment at a hospital without seeing any doctors?
    “our data relate to physician groups, Dartmouth’s relates to hospitals.”
    Can I chose my hospital and physician group independently? Its these types of arguments that make these academics so easy to dismiss.

  7. As John Morrow and I demonstrated a year ago in our piece “Lost in D.C. with the Dartmouth Atlas” in response to the Gawande article, the Dartmouth data is old (2005), not risk-adjusted, and attributes spending to ZIP codes where people LIVE, not where people are TREATED (i.e., more than 100% of the “excess” spending attributed to McAllen is delivered outside of the McAllen CBSA). Add to that J. Skinner’s assertion to me that price (i.e. charges) are irrelevant, and you have plenty of things for the NYT to question. The real issue is why, like the “scoop” of Partners pricing power in MA, it took so long to get here.

  8. Mark–
    Thanks for the comment. Yes, the article was hard to follow. When you’re trying to “write around” the facts, it can be difficulty. It’s like trying to write about something without sufficient information.

  9. Thanks for this Maggie,
    When I read the article originally, it seemed confused and disjointed as if it was torturing the data. This seems to confirm that, indeed, the authors had a point to make (cast doubt on health savings) and weren’t about to let inconvenient facts get in the way.