Commentology: Times Reporters Respond

New York Times health policy reporter Gardiner Harris responded to THCB founder and publisher Matthew Holt’s comments on the recent series of reports he has authored with business writer Reed Abelson questioning the science behind the Dartmouth Atlas.  Gardiner had this to say in defense of his newspaper’s investigation:

The main point of Reed’s and my pieces about the Dartmouth work is that the data are simply not good enough to guide spending decisions in the government’s $484 billion Medicare program. If the Dartmouth researchers had acknowledged this point, our story would not have been all that interesting. But they cannot bring themselves to do this, and in fact they have repeatedly exaggerated and mischaracterized their own work in public settings to suggest it can be prescriptive.

An ancillary point was to warn those on capitol hill, the administration and journalists to be wary of those highly popular maps from the Atlas. You have scoffed that it’s a small thing that the Dartmouth researchers fail to adjust their online data for price and illness. But misunderstandings about this are widespread. That landmark piece by Dr. Gawande that you cited used the Atlas’s unadjusted data. Dozens of stories in newspapers and magazines around the country have used the unadjusted data to criticize health institutions. Even David Cutler, among the top health economists in the country, was unaware that the atlas offered largely unadjusted data.

Accuracy may seem a small point to you. It is not to us.

Our Friday piece also pointed out that Dr. Elliott Fisher and Mr. Jon Skinner claimed that their 2003 Annals pieces had found a negative correlation between spending and outcomes. In fact, the pieces found no correlation between spending and outcomes. This is not a small distinction. If there’s a negative correlation, cuts in spending will actually improve health. If no correlation has been found, then cuts become far harder and perhaps more painful. We cannot go into reforms of our healthcare system believing that the work will be easy. But that is what the Dartmouth researchers have suggested, and this siren song has had an enormous impact on Capitol Hill.

In an aside, when was the last time you saw researchers so profoundly mischaracterize their own work? How is it possible that they could claim their annals pieces concluded something when they didn’t? I can’t remember ever seeing that happen.

–Gardiner Harris

19 replies »

  1. ” But it is convenient to tie the causality to the association reflected in the Atlas to spending variation – to support an economic policy argument for resource redistribution and reduction in healthcare.”
    problem is politicians and journalist are beating us over the head with this proven facts, like AGW science is being misused for political gain. People concerned with the truth have no choice but to attack the study becuase the study is being used to attack the truth. If the authors now claim their study can’t support the claims being made with it they should have stood up and said that 3 years ago.

  2. Of course Dr. Steve and Margalit have it correctly. The Atlas points out some interesting associations with spending variations that have unanticipated outcomes. The possible causal links here (SES variations, population illness severity differences, practice quality differences, healthy/healing environmental differences, etc.)are not explored by the Atlas. But it is convenient to tie the causality to the association reflected in the Atlas to spending variation – to support an economic policy argument for resource redistribution and reduction in healthcare. It is politically easier to argue this than to build policy responses to some of the realistically more likely causal realities, and economic resource redistribution in healthcare is probably a meritorious partial causal issue of some of our health system issues anyhow. It is hard to argue that resource redistribution to reinvigorate primary care wouldn’t be helpful, as a case in point.
    Neither side has to be all right or all wrong here for both sides to have legitimate arguments. It might be better to get our energies redirected to what needs to be done to correctly redirect the course of our healthcare system; as opposed to the in your face “he-said she-said” exercise.

  3. Way to go Margalit. Of course, it’s either fish or fowl. Either the Atlas researchers are saying that the two move independently and have nothing to do with each other (which is what their research seems to show) or that increased spending is linked to poorer outcomes (which is what they claim in public).
    Once they go for the latter, they then have to argue that it’s causal or who cares about any of their research.
    Of course most experts think that in fact spending and outcomes are both driven by a third ‘confounder’. Most health researchers fret about one or two – illness differences and SES differences – each which can affect both spending and outcome.
    So, the confusion here is not the difference between causation and association, it’s the reality that for the Atlas to conclude that higher spending makes worse outcomes THEY have to either assume causation, or they have to deny causation in which case they have to acknowledge the existence of a confounder that they have not adequately controlled for (which also undoes their research). What a dilemma.

  4. So after reading the new response, I keep going back to this one sentence:
    “They [NYT reporters] have confused the idea of a correlation (high spending hospitals tend to do poorly on most measures of quality and outcomes) with causation (if a hospital spends more money, outcomes for those patients will get worse).”
    The correlation example cited is number of storks and birth rates.
    So if the Dartmouth folks are saying that spending is only correlated with outcomes, but there is no cause and effect, then spending and outcomes have nothing to do with each other. There may be a third phenomena driving both, or they could be totally unrelated (like storks and babies). Is that what we are now saying? Or am I one of the confused?

  5. I have been wading through this tiff and have concluded that those that side with dartmouth are clinging to two very problematic syllogisms.
    1) The tone of the NYT article was mean, so Dartmouth must be beyond reproach.
    2) The health care system is in shambles, and Dartmouth says it is so, their researchers must be beyond reproach (and by extension, the NYT must be really saying the system is perfect and that can’t be right).
    I have put a little more thought into this and decided that it is safe to ignore the tone and the larger context of our disastrous healthcare system, and then you have to side with the reporters. Of course they are senior highly experienced science/health reporters at the largest newspaper in the world. Of course they have editors and fact checkers and a front page story gets special scrutiny for tone and content. Of course they got on this story because they saw inconsistencies and errors and then wrote it – not the other way around. And of course if they are right that the Atlas researchers have misrepresented their own work, there is every expectation that the Atlas researchers will continue to do so.
    But none of that is proof to me. That’s just context. What is proof that the reporters have shown in their followup that they are much better at backing up their points than the Atlas researchers are theirs. To back up their claim that they measure quality of hospitals, Atlas researchers posted in their rebuttal a map of diabetes quality measures. What? An error? Don’t think so. It was included as a link in their june 3rd letter, and was still there in a june 7th revision of this letter the researchers posted on their website too. I interpret this as a sign that these are people who will never admit they have made a mistake or misrepresented anything, and hope that no one will actually check to see what is behind their assertions of facts (bummer for them that the NYT did!).
    Compare that to the NYT data. To “prove” that atlas researchers misrepresented their findings as showing that more spending worsened outcomes, even thought their studies DO NOT show that, the nyt reporters linked to the atlas researchers making these statements to the US congress and in a letter to the NYT reporters that was public, and also to their internal recorded interviews with the researchers.
    All the concern about reporters writing biased stories is a distraction. I think the reporters were probably outraged that Ivy League professors were actively misrepresenting their own published work, and equally outraged that these same researchers then posted a ‘rebuttal’ that was again a series of misrepresentations. Reporters inevitably develop a view, and their uncovering the reasons for that view is what makes the story news, particularly when it is against the current as this one was.
    I have to say I have been really disappointed by thcb on this one. I read it to see up to date responses to the latest in health care. But on thcb all I see is maggie mahar trying to debunk rather than act as a thoughtful critic. She spent her time calling the quoted sources in the NYT story, rather than checking the facts reported herself (which the NYT reporters have now shown could be done using public documents). When she called the sources, she discovered that none denied that they said what was reported. Can we get some unbiased commentary on THCB please, lest thcb gets painted with the same brush as that applied to the nyt staff?

  6. Nate doesn’t seem to understand that bias is often in the eye of the beholder.
    People won’t complain when they read something that agrees with their own views, but when it’s the opposite situation, they’re quick to scream “Bias!” I guess it means their own biases are showing.
    Everyone has a bias. Simply by virtue of your gender, where you grew up, what your parents were like, your socioeconomic class and how much education you received, you have biases. It’s neither good nor bad; it’s just how it is.
    The key is in recognizing where your biases lie and trying to prevent your perceptions from being overly colored by your own biases.

  7. Like any ecological study the Dartmouth Atlas, and derivatives, are great as a generators of hypotheses, but a poor substitute for the hard work of healthcare efficacy and quality assessment (including access, effectiveness, etc.). In this the Times reporters are spot-on. It is somewhat human (or academic) nature to maximize (hype) the value of your research, and the Dartmouth group probably aren’t more guilty here than most academics. But this country is in extremely bad shape if (because) its academics are driving health policy decisions!

  8. ” Glaring errors like that kinda throw sand in the gears of your credibility.”
    Really Rick you gauge creditbility on grammer and spelling? Anything written in English by a non english speaker should be ignored then? Pretty ignorant perspective in my opinion.

  9. Nate is clearly no journalist. The word you’re looking for, Nate, is “biased” not “bias.” Glaring errors like that kinda throw sand in the gears of your credibility.
    And to Bill Jones, MD: fine, let’s put your money where your mouth is. All the Dartmouth folk are doing is taking an available data set and applying scientific method to it. If you think their method is wrong, then you must have a pretty good idea of what a good method would look like.
    Using the same data set, describe for us a broad outline of a good way to test whether the data shows practice pattern variation across differing geographies.
    Or were you merely questioning the politics and not the science? Because Abelson and Harris are questioning the science.

  10. Agree with Gardiner and Reed. They need to hold the officials accountable.

  11. “should not have a preconceived agenda.”
    maybe I just have bad luck but when I am in Cleveland the Plain Dealer is overtly bias. When I travel through the airports the LAT and NYT are terribly bais.
    AP and Reuters are caught in bias all the time, they don’t even pretend to be journalist any more.
    The only non bias papers I ever recall reading are the army times and some rural small town papers.
    “research/science should not have a preconceived agenda.”
    I agree it should not but most of it does.
    This article was far from as bad as what passes for normal journlism day in and day out. I could point to countless articles in the Plain dealer this week far more bias then this.
    “But it does happen, especially in research sponsered by institutions that have an interest in a specific outcome – one important example being research sponsered by the pharmaceutical industry”
    I would agree and use Lewin group and their healthcare “studies” as an example. Anything published in the past 2 years in regards to healthcare has been bias and politically motoivated. MSMs coverage of healthcare and Obama will probably be looked back on as the official time of death for journalism. They went full blown sale job with no pretence for journalism ethics.

  12. Nate,
    in risk of stating the obvious:
    -no, newspaper articles (unless you refer to opinion journalism) should not have a preconceived agenda. They may often have one, but at the very least the auhor(s) tries to hide it.
    -no, research/science should not have a preconceived agenda. You may be referring to generation of a scientific hypothesis, which is subsequently supported or refuted by the data (for instance, results of an experiment, or data exploration). But it does happen, especially in research sponsered by institutions that have an interest in a specific outcome – one important example being research sponsered by the pharmaceutical industry
    I believe that it is important to consider possible motives by the authors. In that regard, it is possible but rather unlikely that the Dartmouth group had motives to predetermine their findings. They may be under pressure to have ANY real result (i.e. find a coherent explanation for their results, because without, their research is pointless), or they may be inclined to overstate their results (one of the NYT criticisms, and maybe a justified one) … but that is sthg different.

  13. Harris and Abelson make valid points on the significance of the Atlas.
    The US Government has selective vision and hearing,thanks to bigmouth lobbyists and others with their hands in the back pockets of medicine; and has put medical care on the wrong track for the past three decades. They have rarely if ever heard the reports from the front lines or even the consultancies.
    The reform ideas are flawed and Harris and Abelson attempt to debunk that. Nothing more, nothing less.
    The Atlas kool aid drinkers do not want to taste the bitter truth of reality.
    Next up: the US Government as deceived by the myths of the HIT promoters, mouthing the talking points of the highly profitable non-profit HIT industry trade association and lobbyist HIMSS; and its children CCHIT, e-Health Policy Institute, and other propaganda producing fact manipulating exhibit hall booths replete with booth bunnies.
    There is a huge story here for you, Gardiner and Reed.

  14. rbar by your logic wouldn’t any article on the banking crisis have a preconceived agenda and thus be ignored? In fact how can you ever question anything withotu being accused of having a preconceived agenda?
    Didn’t Dartmouth have a preconceived agenda whenb they did their reseach? Every store written using their data must have had a preconceived agenda. We know for certain Orszag had preconceived agendas. How do you talk about his preconceived agenda without haveing a preconceived agenda?

  15. since when has journalism not had a preconceived agenda? Pick up any daily paper and outside the obits and sports score you won’t find anything without a preconceived agenda.
    As long as that agenda agrees with people’s perspective they tend to ignore it but it is always there.

  16. These guys lost all their credibility based on their tone of the original article alone. It is hard to read their first piece and not to think that this journalism had a preconceived agenda.