The Effect Of Physicians’ Electronic Access To Tests: A Response To Farzad Mostashari

Our recent Health Affairs article linking increased test ordering to electronic access to results has elicited heated responses, including a blog post by Farzad Mostashari, National Coordinator for Health IT.  Some of the assertions in his blog post are mistaken.  Some take us to task for claims we never made, or for studying only some of the myriad issues relevant to medical computing.  And many reflect wishful thinking regarding health IT; an acceptance of deeply flawed evidence of its benefit, and skepticism about solid data that leads to unwelcome conclusions.

Dr. Mostashari’s critique of our paper, will, we hope, open a fruitful dialogue.  We trust that in the interest of fairness he will direct readers to our response on his agency’s site.

Our study analyzed government survey data on a nationally representative sample of 28,741 patient visits to 1187 office-based physicians.  We found that electronic access to computerized imaging results (either the report or the actual image) was associated with a 40% -70% increase in imaging tests, including sharp increases in expensive tests like MRIs and CT scans; the findings for blood tests were similar.  Although the survey did not collect data on payments for the tests, it’s hard to imagine how a 40% to 70% increase in testing could fail to increase imaging costs.

Dr. Mostashari’s statement that “reducing test orders is not the way that health IT is meant to reduce costs” is surprising, and contradicts statements by his predecessor as National Coordinator that electronic access to a previous CT scan helped him to avoid ordering a duplicate and “saved a whole bunch of money.” A Rand study, widely cited by health IT advocates including President Obama, estimated that health IT would save $6.6 billion annually on outpatient imaging and lab testing.  Another frequently quoted estimate of HIT-based savings projected annual cost reductions of $8.3 billion on imaging and $8.1 billion on lab testing.

We focused on electronic access to results because the common understanding of how health IT might decrease test ordering is that it would facilitate retrieval of previous results, avoiding duplicate tests.  Indeed, it’s clear from the extensive press coverage that our study was seen as contravening this “conventional wisdom”.

Nonetheless, Dr. Mostashari criticizes us for analyzing the impact of physicians’ electronic access to imaging and test results, but not other aspects of electronic health record (EHR) use.    We did, however, analyze the relationship of EHRs to test ordering in a subsidiary analysis.  While physicians use of a full EHR was associated with a 19% increase in image ordering, as we noted in the paper this finding was not statistically significant.  While we cautiously (and properly) interpreted this as a “null” finding, these data are inconsistent with Mostashari’s optimistic view that use of a full EHR reduces costs.

He asserts that our 2008 data are passe, and that health IT meeting today’s “meaningful use” criteria definitely saves money. The data we analyzed were the latest available data when we initiated the study.  While the proportion of outpatient physicians utilizing health IT has grown since 2008, we are unaware of any “game changing” health IT developments in the past four years that are would produce substantially different results if the study were repeated today.  The EHR vendors that dominated the market in 2008 remain, by and large, today’s market leaders, and their products have undergone mostly modest tweaks.  Mostashari’s contention that 2012  EHRs  – incorporating decision support and electronic information exchange – save money in ways not possible in 2008 should be tested through additional research but remains merely a hypothesis. We hope that some day his predicted savings can be achieved.

Dr. Mostashari offers his own explanation for our findings, suggesting that doctors who are inclined to order more tests are also inclined to purchase health IT for viewing test results electronically rather than on paper.  He offers no evidence for this assertion and ignores the fact that we explored (and rejected) this explanation by analyzing subgroups of doctors who are unlikely to be the decision maker for IT purchases – e.g. employed physicians, those working in an HMO setting etc.  In other words, electronic access to results predicted more test ordering whether or not the ordering physician was responsible for health IT purchases.

He incorrectly states that our analysis did not take into account patients’ severity of illness, physicians’ level of training, and the nature of physicians’ financial arrangements.  In fact, we reported subsidiary multivariate analyses that included several serious diagnoses; all of our models included physician specialty (which we specified in several different ways); and all models included adjustment for an extensive list of indicators of financial arrangements (e.g. whether the physician owned the practice or was an employee; the type of office; whether the practice was owned by a hospital; whether the physician was a solo practitioner; whether the physician’s compensation was based, in part or whole on “profiling”; and whether the practice was predominantly prepaid).  We also performed a series of subsidiary analyses that explored whether physicians with a proclivity to “self refer” patients for imaging tests accounted for our finding; they didn’t.

Dr. Mostashari criticizes us for failing to assess whether health IT improved the quality or appropriateness of care.  Of course, these were not the topic of our research.  Those are different studies for a different time.  However, we would note that other large-scale studies have found no, or trivial quality improvements associated with HIT outside of a few flagship institutions4-6.

Dr. Mostashari’s strongest claim is that observational studies like ours (and most other health policy studies, including some by Dr. Mostashari himself) cannot prove causation. This is surely true.  As long time teachers of evidence based medicine we took care to couch our conclusions in cautious terms, stating only that “Computerization, whatever its other benefits, remains unproven as a cost control strategy.”

But Dr. Motashari is less cautious, asserting that the case for HIT is closed.  The paper he cites to buttress this claim (authored by members of his own agency) culled studies reporting any impact of HIT on virtually any aspect of care, and accepted authors’ claims of benefit without regard to study quality or statistical niceties.  Thus, a focus group’s impressions of benefit are accorded the same weight as nationwide studies of Medicare data showing virtually no impact of computerization on quality measures.  Reports of a reduction from 70% to 38 % in “missed billing opportunities” or a $7,000 reduction in office supply costs are among the 92% of studies judged “positive”.  While the literature review he cites is interesting, nothing in it contradicts our findings.

Dr. Mostashari is also correct in reiterating that randomized trials are the best way to assess health IT.  In fact, no randomized trial has ever been published that examines patients’ outcomes or costs associated with off-the-shelf health IT systems that dominate the U.S. market.  No drug or new medical device could pass FDA review based on such thin evidence as we have on health IT.  Yet his agency is disbursing $19 billion in federal funds to stimulate the adoption of this inadequately evaluated technology. Dr. Mostashari is perhaps the only person in our nation who commands the resources needed to mount a well done randomized controlled trial to fairly assess the impact of health IT, and the comparative efficacy of the various EHR options.

Finally, Dr. Mostashari’s unbridled faith in technology is mirrored by his belief that ACOs are the next panacea for health costs and quality.  That health policy flavor-of-the-month also remains wholly unproven.

This post originally appeared on the Health Affairs Blog.

17 replies »

  1. I do think this might be too much info too, but I don’t think Rick is trying to politicize his loss. I think he’s trying to get some things across to people, and I think if he handles himself properly, i.e., answers questions honestly and soberly he’ll avoid raising the ire and antipathy of pallet independents. n nThis election will be won by a narrow margin. Conservatives can withstand the left independents staying in a place they have absolutely no intention of leaving. What they can’t withstand is encouraging right leaning independents to conclude President Obama, at the very least, represents stability and not further decline or confusion. n nWhat RS needs to do is tie the impossibility of limited government to the acceptance of pathology or the embrace of standard-lowering lifestyles. Social cohesion is a good building block. And while on social issues the Country is divided, it’s not divided on limiting government and growing the economy. n nIt can be done, but it has to be done without self-righteousness. And I’m afraid that can’t be rehearsed. The lack of self-righteousness is indicative of a security that admits the consent of the governed is requisite. n nMaybe he’s up to the task. We’ll soon find out.

  2. Again, I can’t disagree with your points. If BobbyG Ruled the ONC world, we wouldn’t have ,1,800+ “Certified” systems, a number of which I have seen that don’t even perform out of the box as advertised for the relatively puny MU requirements, much less facilitate improved workflow — including reduced error rates.

    I’ve had my own bit of sport with the whole idea here:


  3. I think many of the claimed benefits for EMRs cannot be realized because of deep flaws in our healthcare system; similarly, many of the problems with them are caused by the same flaws.

    We’ve been told that this bandaid could cover a gaping chest wound, and it shouldn’t be surprising that it’s proving to be completely inadequate to the task.

    The failures of EMRs are just peripheral by-products of the problems with health care in general in this country. As long as we treat the symptoms and ignore the root cause, EMRs won’t be good for much.

  4. We don’t seem to have a problem with making each and every new Statin drug prove it’s worth – each one must go through the same clinical trials, each one has to provide data to back up every claim (indication).
    While I agree that the HIT industry cannot be compared directly to Pharma, nor to a study like WHI, it would seem possible that each claimed benefit could be the subject of an industry wide study. There have been some small studies but so far nothing designed very well at all. My own experience, and that of many others (anecdotal) suggests that EHRs raise costs, lower productivity, and introduce new types of errors even while reducing others. I think that if an EHR vendor is going to claim an increase in productivity they need to back up that claim with actual data. I believe that the types of errors introduced by EHRs could be studied and we would benefit from such studies even if they are not comparative to paper charts. I think some sort of product labeling is in order – Even life-saving drugs have to include all the information on how they can kill you.
    While it clearly is too late to turn back the clock on EHRs and much of the other HIT, and I don’t think that really is the goal of any group, I find the reluctance of HIT advocates to submit their claims to serious study to be very backwards. It would appear that they have very little understanding of the idea of evidence based medicine and yet they claim to be the ones at the forefront of medical inovation.

  5. Follow up:

    What would be the “controls” in HIT RCTs (i.e., stratified representative samples of equivalent paper practices and institutions)?

  6. Fair enough.

    “…no randomized trial has ever been published that examines patients’ outcomes or costs associated with off-the-shelf health IT systems that dominate the U.S. market.”

    Precisely how would we design and execute such a study, particularly at a time when HIT markets are ramping up at warp speed? Which “dominant” off-the-shelf systems would get included (based on what cut-off %’s of current ambulatory and inpatient market shares)?

    There are now a total of 1,817 “Certified” EHR ambulatory and inpatient systems and modules (including 913 “complete” systems). Who gets “studied,” who gets left out? Would these be “Dummy Clinic” RCT mock studies, or operational CER?

    I’m a bit dubious of comparing HRT studies to what would be required to do RCTs on EHRs. Not that I buy uncritically that they comprise a panacea.

  7. “I agree. We should use stone tablet and chisels.”

    Wow. Totally awesome brilliant destruction of a straw man.

    To interpret the authors’ findings as a recommendation that we totally abandon EMRs demonstrates a serious lack of reading ability.

  8. Physicians collectively took some heat when it became apparent in a well designed study (Womens Health Initiative – WHI) that the use of HRT in post menapausal women did not have the cardiovascular benefits it was claimed to have and that there was significant associated morbidity. The result was counter intuitive.

    It is obvious that the intuitive take on EHRs and HIT is that they will lower errors, reduce costs, and improve quality of care. This may in fact be the case, but what if it isn’t? Stating “We should use stone tablets and chisels….they worked fine for centuries” is hardly a convincing sarcasm when it is apparent that what is being said is that the benefits of HIT are self evident and do not require that we take the time and effort to prove that the benefits actually exist. I suppose these same HIT proponents must feel that the WHI was a waste of time as well…

  9. The authors of this study are not opposing HIT or EHR. They found no correlation between existence of an EHR and increased imaging.
    They used data collected by NAMCS, not data collected by a carefully designed before and after study. They did control for self referral as best they could by eliminating specialties that are most likely to own equipment and found no effect.
    There is no financial gain for PCPs and internists to order CT scans and they are not likely to own any of these very expensive technologies. Note that the effect was largest for very expensive studies.
    Also noteworthy is that hospital owned practitioners ordered a lot more expensive studies than others, and solo practices were markedly more frugal than everybody else. Perhaps these observations triggered the angst over ACO and consolidation, but I am just guessing.

    We are all guessing, so instead of shooting the messenger, how about funding some quality longitudinal studies, as the authors are actually requesting ONC to do? There seems to be plenty of money for PR, why not allocate some of it to quality research?

  10. “Finally, Dr. Mostashari’s unbridled faith in technology is mirrored by his belief that ACOs are the next panacea for health costs and quality. That health policy flavor-of-the-month also remains wholly unproven.”

    Unhelpfully partisan, with an unnecessary Straw Man zinger to boot. Yeah, let’s just not try ANYTHING beyond twenty more years of Ivory Tower Analysis Paralysis. Have to agree with Matthew Holt above.

    BTW, check out what I just snipped out of a recent Cancer Journal article:


    “To reduce future projected cancers from diagnostic procedures, we advocate the widespread use of evidence- based appropriateness criteria for decisions about imaging procedures; oversight of equipment to deliver reliably the minimum radiation required to attain clinical objectives; development of electronic lifetime records of imaging procedures for patients and their physicians; and commitment by medical training programs, professional societies, and radiation protection organizations to educate all stakeholders in reducing radiation from diagnostic procedures.”

    CA Cancer J Clin 2012;62:75-100. Published 2012 American Cancer Society

  11. Followup Thought –

    Outside of integrated systems like Kaiser and Partners it is very clear in many cases that EHR sales reps present them as a way to upbill so I don’t mean to imply that this isn’t possible simply that this study doesn’t appear to capture whether or not it is the implementation itself that is the cause of the difference in ordering behavior or not but I will go back and review the data.

    FYI I have tremendous respect for Himmelstein’s thoughtfulness as well as his team’s approach and we all know that he has little control over the lay press headlines or articles conclusions.

  12. Sherry Reynolds is pointing in the direction of one key response. The researchers say they controlled for physicians who self-refer for imaging. I would be interested to see what that control consisted of. Are they saying that physicians who own their own equipment and self-refer are 40-70% more likely to order imaging when they have electronic access to images than when they don’t? And was there an attempt to control for the amount of self-referral before and after implementing an electronic image retrieval system for physicians who owned their own equipment?

    What percent of those with access to electronic images owned their own equipment, and how does that compare with the general population?

    This study is both very important and very flawed. It’s important in reminding us that if all you do is make it easier for a physician to quickly access results that he or his practice ordered, you are reducing a hassle with imaging, and increasing utilization given a constant demand function. I don’t think we can say yet that the effect is a 40-70% increase, but it is probably a bigger increase than I or other HIT advocates expected.

    Bu the study is seriously flawed in that, as far as I’ve seen, it makes no effort to distinguish between getting access to your own results and getting easy access to any result from any imaging test. This is where Dr. Mostashari is on firm ground.

    The case for HIE was explicitly all about how it would enable efficiencies from the use of EHRs (and eRx, PACS, etc) that EHRs by themselves would not be able to produce. How could HIT reduce duplicate testing unless physicians had easy access to results that they wouldn’t normally get, like those of an unrelated physician or hospital across town or across the country? Most of us HIT advocates were not confused about that. I remember multiple discussions of exactly that point on this blog and elsewhere over the last 5 years. The authors seem to be claiming ignorance of this, which is (I’ll hold my fire here) not cool.

  13. “We found that electronic access to computerized imaging results (either the report or the actual image) was associated with a 40% -70% increase in imaging tests”

    There is a fundamental flaw in this statement – if you in fact studied the same provider before and after implementing a full EHR your conclusion might have more validity but it appears that you simply compared practies that had the ability to view imaging online vs those that didn’t.

    One possible explanation is that those practices that already were ordering many more imaging tests were also those making enough profits to afford to order imagining tests independent of whether or not they had online imaging.

    Essentially what you show is that practices that maximize imaging are the ones who purchased online imaging first. (not the same as an EHR but I assume this holds true) The key test would be to track the same provider before and after an implementation.

  14. I agree. We should use stone tablets and chisels….they worked fine for centuries

    On the other hand, at least with EMRs we can actually FIND OUT what is being ordered, figure out whether it’s appropriate and just possibly, do something about it.

    Which, as the PNHP guys who wrote this know, is what eventually just might happen. Whether they or the evil insurance companies are in charge….

  15. Wow!!

    Finally someone standing for reason.

    The difference between off the shelves EMRs and what we really need is similar to the difference between a “Book Depository” and a “Library”.

  16. It’s tough work being out there proclaiming that the Emperor is buck-naked. Thank you.