OP-ED

More Work Is Needed on the Safety and Efficacy of Healthcare Information Technology

If one were writing about the improvement of gastronomy in America, one would probably not celebrate “over 300 billion hamburgers served.”  But that’s very much the type of success Dr. Ashish Jha is celebrating in last week’s piece on recent US healthcare IT sales. Unfortunately, the proliferation of Big Macs does not reflect superior cuisine, and healthcare IT (HIT) sales do not equate with better healthcare or with better health. Quantity does not equal quality of care.

To be sure, Dr. Jha acknowledges the challenges of rolling out HIT throughout US hospitals. And he should be strongly commended for his admission that HIT doesn’t capture care by many specialists and doesn’t save money. In addition, Dr. Jha points to the general inability of hospitals, outpatient physicians and laboratories to transfer data among themselves as a reason for HIT’s meager results.

But this is a circular argument and not an excuse. It is the vendors’ insistence on isolated proprietary systems (and the government’s acquiescence to the vendors) that created this lack of communication (non-interoperability) which so limits one of HIT’s most valuable benefits.

In our opinion, the major concern is that the blog post fails to answer the question we ask our PhD students:

So what? What is the outcome?

This entire effort is fueled by $29 billion in government subsidies and incentives, and by trillions of dollars spent and to be spent by hospitals, doctors and others [1].

So where is the evidence to back up the government’s and industry’s promises of lower mortality, improved health and lower health care costs?

Single studies tell us little. Sadly, as many as 90% of health IT studies fail the minimal criteria of the respected international literature syntheses conducted by the Cochrane Collaboration.

In other words, studies with weak methodology or sweetheart evaluation arrangements just don’t count as evidence.

Nevertheless, we know that healthcare IT can improve some processes of care, like reducing duplications of tests, prescribing better drugs for the elderly, and decreasing dosage errors, especially in integrated settings, like certain Kaiser systems.  But our and others’ research shows little or no evidence that such changes result in better health.

For example, all well-controlled studies in a recent Cochrane review show no effect of health IT on high blood pressure. The best data show no effects on healthcare costs although the administration and GAO promised us up to $100 billion. It turns out that the study promising those savings has now been debunked by the same group that made them, the Rand Corporation [2]. It also emerged that the original research was supported by two of the largest vendors of HIT, a fact not previously acknowledged.

Another difficulty with this research literature is the proliferation of undisclosed financial ties to the industry or with HIT operations. Given the vast sums involved, US policy should be based on rigorous syntheses of the entire literature by unbiased researchers.  We know financial and emotional ties are usually associated with finding more frequent and impressive HIT “successes [34].”

It is essential that all researchers disclose their conflicts of interest to the readers of this and other blogs, editorials, original research articles and other publications. Unfortunately, we have witnessed major financial ties, such as patents, board memberships, consultancies and research funding go undisclosed in influential studies. Why should health IT researchers and companies be treated any differently from drug companies?

We also have indisputable evidence that health IT can harm health. Dr. Jha, in fact, was a member of the federal committee that produced that report on HIT safety (not noted in his piece) [5] . The industry and the government agencies charged with promoting HIT sales aggressively propagandize any study with positive findings and condemn studies documenting no or adverse effects of HIT on health [6].

The tension between the industry’s sales  and patient safety concerns is reflected in the current efforts to prevent the FDA from exercising its regulatory authority over HIT. Both industry and its government promoters actively seek to avoid FDA oversight, citing the possible anti-innovative effects of regulation.

They’ve got a point, but so do those who worry about the innumerable errors enhanced by HIT’s clumsy, non-interoperable, and irrational user interfaces. We need a balance between regulation and freedom to innovate; there are valid concerns on both sides. Meanwhile, clinicians, scholars and the FDA itself have catalogued thousands of medication errors generated by these technologies  [7].

We agree with Dr. Jha that HIT offers many and significant advantages over paper. But if it had to meet the criteria required of medications, it would never be approved for human use. Even the best studies fail to support the advertised claims of improved health and lower costs. More important, there is overwhelming evidence of patient safety dangers and failures.

We also agree with Dr. Jha that the software will undoubtedly improve over the next decades. HIT is only about 40 years old, and has rocketed to widespread use in the last few years via the government’s incentive and subsidy programs. This has been a mixed blessing. We’ve learned a lot about its abilities and extraordinary potential, but we’ve also observed many of its challenges and errors.

We sincerely hope that HIT will achieve its currently unfulfilled promises of improved health outcomes and efficiency. But this will require policies focused on evidence-based analyses and data standards that allow interoperability, rather than a rush to obtain subsides for questionably effective functionalities designed to encourage sales in a compressed timetable.

Software sales must be secondary to rigorous and unbiased research examining whether such software represents good value for the trillions being spent by US hospitals and clinicians.

Stephen Soumerai, Sc. D., is Professor of Population Medicine at Harvard Medical School and Harvard Pilgrim Health Care Institute and Co-Chair of the Evaluative Sciences and Statistics concentration of the Harvard University-wide PhD Program in Health Policy.

Ross Koppel, PhD, FACMI, has taught sociology at the University of Pennsylvania for 21 years and is the PI on several studies at Penn’s Medical School.

Livongo’s Post Ad Banner 728*90

41
Leave a Reply

19 Comment threads
22 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
20 Comment authors
sleuthlegacyflyerDoctor MawrdoughBobbyGimproveoutcomes Recent comment authors
newest oldest most voted
S Silverstein MD
Guest
S Silverstein MD

New report on worsening of care quality due to today’s poorly conceived EHR systems, from the Cleveland Clinical Journal of Medicine.

Electronic siloing: An unintended consequence of the electronic health record
http://www.ccjm.org/content/80/7/406.full

Most of the observations in this new piece have been obvious to experienced clinicians for years, I add.

sleuth
Guest
sleuth

OMG, I just listened to the Senate Finance Committee hearings on HITECH and the testimony of @farzad_onc. Losts of words but said nothing, just like most EHRs. The senators were impressed with the narcissistic approach.

legacyflyer
Guest
legacyflyer

BobbyG, You have challenged us “guys” – and by that you presumably mean those that are skeptical about the benefits of an EHR – to describe how a demonstration of safety and efficacy would work. ” What would comprise such a “demonstration”? It’ll never suffice for you guys.” I am a practicing physician and not an expert on the design of research studies. But I will take your challenge. 1) Take 100 hospitals and divide them into two groups of 50 2) Require one group to adopt an EMR. No requirement for the other group. (Also no penalty for non-adoption)… Read more »

BobbyG
Guest
BobbyG

Now we’re getting somewhere. A reasonable proffer. OK, but, as of today there are 318 ONC Certified inpatient “complete” systems. – Would we have to multiply by some random sample fraction of those (say, maybe pick 10 systems randomly), and stratify? (e.g., those using EPIC ended up sucking, but those using McKesson or _________, _________, _________ did not). Or are we going to merely pit ONE system against paper? If so, which one would suffice as the generic proxy for all systems? – would these be randomly chosen facilities (from among for-profit, community non-profit, and academic medical centers)? – Given… Read more »

legacyflyer
Guest
legacyflyer

Again, I am NOT an expert on experimental design but … If the question we are seeking to answer is: “Is an EMR better than paper?” – then I would suggest we let the decision makers in each of the hospitals choose which EMR they want – as is the case today. “Is EPIC better than paper?” – then we would need to have all 50 hospitals use EPIC. If EMR’s are really (take your pick); The Savior of Healthcare, The Best Invention Since Sliced Bread, The Bane of Our Existence, then the relatively small sample size should be enough.… Read more »

BobbyG
Guest
BobbyG

Good stuff. My speculation, though, is that the time window is too narrow. And, I am skeptical that costs will be materially reduced irrespective of HIT. Especially in across a six month period). Too many other confounding factors. Consider the recent “Oregon Study” — 2 years, mildly “increased UTIL”, no “improvement in selected outcomes.” OK, first, this was outpatient Medicaid, so there’s no direct tightly coupled analogy with inpatient. Even so, What In The Hell Were They Thinking? Tangentially apropos, I’m 67, now a Medicare bene with the usual litany of Chronics. I’m a 99123 – 99214. I see my… Read more »

Doctor Mawrdough
Guest
Doctor Mawrdough

Jha has been an unabashed cheerleader for HIT, when, he does not use the systems. I asked an HIT proponent kool aid server how to access a particular transition of care document and the response was, “my residents do it, I never did and do not know how”.

Jha should use the systems before pontificating how wonderful they are.

Koppel and Soumerai render a sobering and candid report. In order for any benefits to accrue, it behooves the lawmakers to know the adversity.

Caregiver, MD PhD
Guest
Caregiver, MD PhD

@koppel: “Please remember that both Steve and I teach research methods for a living.”

Wow, the HIT manufacturers should take lessons from you…but then again, they have HIMSS to spread the gospel and pay RAND to write favorable white papers.

ytisebo
Guest
ytisebo

Having observed from afar, the above commentary is long overdue. Kudos to the authors. Taxpayers are, indeed, being scammed and federal laws are being broken to enable the distribution and use of HIT sans application of the F D and C Act.

Whatsen Williams
Guest
Whatsen Williams

Did anyone listen to the obtuse comments made by @farzad_onc at the Senate Finance Committee hearings on HIT? The also brought in a pediatrician who is high in command of CMS that insures old people. Hello?
The senators seemed to only be interested in numbers, a bit about privacy and hacking, some about care in the boon docks, and they all simply assumed that it was a safe building block for care. The spelling of patient has now become paytient. The computer is now the patient.

BobbyG
Guest
BobbyG

“The spelling of patient has now become paytient”
__

I like that. LOL.

But, that has long been the case.

Curly Harrison, MD
Guest
Curly Harrison, MD

Pardon my naivete but what organization is providing assessment of the safety, efficacy, and usability of CPOE before it is sold and once it is being used?

From what little I know, CPOE and its associated decision support modules are serving to treat and mitigate disease, and thus, are medical devices.

BobbyG
Guest
BobbyG

“From what little I know”
__

Operative clause

ross koppel
Guest
ross koppel

Thanks for all of these comments. Most are stunningly supportive….and appreciated. I certainly agree that a randomized control double blind study of HIT is an absurd requirement. Can’t be done. But there is a lot of very obviously very bad research on HIT that is touted to justify spending on HIT. The perfect should not be the enemy of the good, but obviously inadequate research should not be used to justify locking us into a decade of mediocre-to-worse software. Several points were raised: I’ll address a few: with the forthcoming consolidation of the industry, the congress and/or the ONC or… Read more »

Steve Soumerai
Guest
Steve Soumerai

Hi Ross: I should have read from the bottom up before commenting! At least we didn’t contradict each other! Nice note! Steve

S Silverstein MD
Guest
S Silverstein MD

Re: ” This piece reads like a demand for … running 5+ year randomized clinical trials) ather giving us useful guide posts getting from where we are to where we want to be.”

I fail to understand, scientifically and ethically, why the research protocols of pharma and the medical device sector to avoid the debacles of the past (e.g., thalidomide and many others) should not apply to health IT, which is increasingly central to, and mediates, all transactions of care.

Bobby Gladd
Guest
Bobby Gladd

Let’s hear your proposed HIT randomized trial study design, Scott. It would have to be a MANOVA on steroids, one stratified by all 70-odd medical specialties and every supported hardware system and firmware and OS release. I’m all for certification rigor. If you’ve published a design test proposal of such realism and stringency, let’s hear it. You certainly have the Sheet for it.

Calling for criminal prosecution of HIT vendors isn’t going to advance the cause one iota. First of all everyone has to sign Hold Harmless clauses. Second, the lawyers are wealthy enough.

Steve Soumerai
Guest
Steve Soumerai

Bobby: There are already a fair number of well designed, SIMPLE and large randomized clinical trials. If they are not dramatic they have less chance of publication in the major journals than a untrustworthy correlational study that “shows ” improved outcomes (there are data). Just one terrific example: the study led by Jerry Gurwitz, published in a geriatrics journal around 2007-8 (senior authored by David Bates). They studied an advanced CPOE system in a large number of horsing homes in Connecticut and found reduced drug interactions but zero effect on drug injuries–their main objective. Indisputably ineffective- so much so that… Read more »

BobbyG
Guest
BobbyG

ONE study, in CT “horsing homes,” published 5-6 years ago? Meaning the data collection and analysis was well before that?

Well, uh, OK… how timely.

Got a link?

Steve Soumerai
Guest
Steve Soumerai

Hi Bobby: My MAC spell checker is the cause of too high HITECH and nursing homes for horses!! Another cool software error. I gotta lose it.

Here is the link. Regardless of the 2008 timing, they changed important interactions but no adverse events in a big trial.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2359507/

If doesn’t work you have PMC number. S

BobbyG
Guest
BobbyG

Thanks, Steve.

Steve Soumerai
Guest
Steve Soumerai

Shoot. That’s the review before their own RCT. Emaill me at ssoumerai@hms.harvard.edu on Tues when back and I’ll send you the article. My project manager has it. Best, S

S Silverstein MD
Guest
S Silverstein MD

Actually, I don’t just “call” for prosecution, I actively educate and assist trial lawyers in prosecuting cases of EHR-related harms and evidence spoliation (tampering). As opposed to your apparent belief in laissez-faire, unprecedented in healthcare since at least the FD&C Act, I call for robust post marketing empirical studies of health IT risks, done with transparency (as called for by IOM and others), as well as pre-marketing software validation such as spelled out in depth in documents such as “General Principles of Software Validation; Final Guidance for Industry and FDA Staff” and “Certification Processes for Safety-Critical and Mission Critical Aerospace… Read more »

Matthew Holt
Guest

The question is when will user centered modern interfaces attached to cloud based systems (what I call health 2.0) start sucking the data out of the ERP-type monstrosities that Steve, Ross & Scott criticize? When that happens, these systems will be easier to use, easier to change and also provide better access to data. My sense is that the route to heaven is for Steve, Ross, Scott and the rest to start demanding the easy access to data, and the legislative demands for real inter-operability. which IS IN the meaningful use regs (however they actually get enforced). This piece reads… Read more »

40yearold doc
Guest
40yearold doc

“When that happens, these systems will be easier to use, easier to change and also provide better access to data”

As long as any system is designed for the benefit of CMS and large insurers (ICD, CPT, MU, P4P, and on and on), and not for the benefit of patients and physicians, nothing will get any better.

improveoutcomes
Guest
improveoutcomes

“As long as any system is designed for the benefit of CMS and large insurers (ICD, CPT, MU, P4P, and on and on), and not for the benefit of patients and physicians, nothing will get any better.”

Couldn’t agree more. This is it. It speaks directly to why the marriage of healthcare and IT has failed all of us (outside of the EMR companies). Unfortunately, divorce is now not an option so we must be be vigilant in exposing and ignoring the “solutions” that expose and ignore patients and physicians.

Bubba For President
Guest

This is a bit harsh. If you look at what Ashish has written over the past few months – you’ll see plenty of nuance, which is something that probably cannot be said about this post. His main point – which is that there is GOOD NEWS on adoption rates, is well worth talking about. As to your critique, reliable studies on the benefits of new technologies are NOTORIOUSLY DIFFICULT to pull off. Presumably you know this. While we’re throwing around charges making a federal case out of POOR USABILITY and DESIGN PHILOSOPHIES could be called alarmist. Lastly, technology companies are… Read more »

Steve Soumerai
Guest
Steve Soumerai

Those are reasonable points! As a research design educator, however, I have to say that the majority of poor studies were preventable. We don’t insist on RCTs.But there are some good ones- mostly negative. But please, measure some sort of change. Use longitudinal designs like interrupted time series (often those data are available) to control for pre-existing trends.Find a comparison series. I have seen too many correlational, cross-sectional studies passed off for cause and effect. And those studies almost always favor HIT. Naturally because the places that had it first are bigger, have higher volumes of procedures, are academic medical… Read more »

Dr. Rick Lippin
Guest
Dr. Rick Lippin

Scot-

Thanks for your leadership and activism. I also believe that once proven guilty of crimes including homocide that some of these HIT leaders should receive jail sentences. Lawsuits, fines and bad publicity are not enough and are not working.

Dr. Rick Lippin
Southampton,Pa

tschwiet MD
Guest
tschwiet MD

I am a fourth generation physician and was wisely told by my father and grandfather that medicine is part art and part science. Early in my career, I was naively biased toward the science aspects, and that data and logical decision-making on that data were the keys to great medicine. As I progressed in my career it became more apparent to me that the ‘art of medicine’ aspects prevailed as the dominant force to being a great primary care doc. Detecting subtleties, nuance, and seeing the underlying psychological factors, areas of resistance, and individual barriers gave me insights into what… Read more »

Steve Soumerai
Guest
Steve Soumerai

tschwiet: I guess the research supports the enormous value of both the science and art of medicine. We once were rightly criticized for comparing the quality of care based only on adherence to guidelines. We missed half the story. The studies and op-eds of other physicians frequently support your points. EHRs can interfere with doctor patient communication and can lead to simple minded and confusing clinical documentation. Nice point about patient centered care. Best, Steve Soumerai

S Silverstein MD
Guest
S Silverstein MD

A poorly designed but highly touted EHR led to the rather cruel injury and death to my mother – in the very hospital where I had done my residency in the mid 1980’s. I am postdoctorally trained in Medical Informatics as well. The events that occurred can happen anywhere and to anybody. May you and your family members be more fortunate. Good health IT is not what the industry is currently producing. What is the current standard is bad health IT. Definitions at http://www.ischool.drexel.edu/faculty/ssilverstein/cases . In essence, the health IT hyper-enthusiasts who ignore the current problems and push today’s bad… Read more »