More Work Is Needed on the Safety and Efficacy of Healthcare Information...

More Work Is Needed on the Safety and Efficacy of Healthcare Information Technology

41
SHARE

If one were writing about the improvement of gastronomy in America, one would probably not celebrate “over 300 billion hamburgers served.”  But that’s very much the type of success Dr. Ashish Jha is celebrating in last week’s piece on recent US healthcare IT sales. Unfortunately, the proliferation of Big Macs does not reflect superior cuisine, and healthcare IT (HIT) sales do not equate with better healthcare or with better health. Quantity does not equal quality of care.

To be sure, Dr. Jha acknowledges the challenges of rolling out HIT throughout US hospitals. And he should be strongly commended for his admission that HIT doesn’t capture care by many specialists and doesn’t save money. In addition, Dr. Jha points to the general inability of hospitals, outpatient physicians and laboratories to transfer data among themselves as a reason for HIT’s meager results.

But this is a circular argument and not an excuse. It is the vendors’ insistence on isolated proprietary systems (and the government’s acquiescence to the vendors) that created this lack of communication (non-interoperability) which so limits one of HIT’s most valuable benefits.

In our opinion, the major concern is that the blog post fails to answer the question we ask our PhD students:

So what? What is the outcome?

This entire effort is fueled by $29 billion in government subsidies and incentives, and by trillions of dollars spent and to be spent by hospitals, doctors and others [1].

So where is the evidence to back up the government’s and industry’s promises of lower mortality, improved health and lower health care costs?

Single studies tell us little. Sadly, as many as 90% of health IT studies fail the minimal criteria of the respected international literature syntheses conducted by the Cochrane Collaboration.

In other words, studies with weak methodology or sweetheart evaluation arrangements just don’t count as evidence.

Nevertheless, we know that healthcare IT can improve some processes of care, like reducing duplications of tests, prescribing better drugs for the elderly, and decreasing dosage errors, especially in integrated settings, like certain Kaiser systems.  But our and others’ research shows little or no evidence that such changes result in better health.

For example, all well-controlled studies in a recent Cochrane review show no effect of health IT on high blood pressure. The best data show no effects on healthcare costs although the administration and GAO promised us up to $100 billion. It turns out that the study promising those savings has now been debunked by the same group that made them, the Rand Corporation [2]. It also emerged that the original research was supported by two of the largest vendors of HIT, a fact not previously acknowledged.

Another difficulty with this research literature is the proliferation of undisclosed financial ties to the industry or with HIT operations. Given the vast sums involved, US policy should be based on rigorous syntheses of the entire literature by unbiased researchers.  We know financial and emotional ties are usually associated with finding more frequent and impressive HIT “successes [34].”

It is essential that all researchers disclose their conflicts of interest to the readers of this and other blogs, editorials, original research articles and other publications. Unfortunately, we have witnessed major financial ties, such as patents, board memberships, consultancies and research funding go undisclosed in influential studies. Why should health IT researchers and companies be treated any differently from drug companies?

We also have indisputable evidence that health IT can harm health. Dr. Jha, in fact, was a member of the federal committee that produced that report on HIT safety (not noted in his piece) [5] . The industry and the government agencies charged with promoting HIT sales aggressively propagandize any study with positive findings and condemn studies documenting no or adverse effects of HIT on health [6].

The tension between the industry’s sales  and patient safety concerns is reflected in the current efforts to prevent the FDA from exercising its regulatory authority over HIT. Both industry and its government promoters actively seek to avoid FDA oversight, citing the possible anti-innovative effects of regulation.

They’ve got a point, but so do those who worry about the innumerable errors enhanced by HIT’s clumsy, non-interoperable, and irrational user interfaces. We need a balance between regulation and freedom to innovate; there are valid concerns on both sides. Meanwhile, clinicians, scholars and the FDA itself have catalogued thousands of medication errors generated by these technologies  [7].

We agree with Dr. Jha that HIT offers many and significant advantages over paper. But if it had to meet the criteria required of medications, it would never be approved for human use. Even the best studies fail to support the advertised claims of improved health and lower costs. More important, there is overwhelming evidence of patient safety dangers and failures.

We also agree with Dr. Jha that the software will undoubtedly improve over the next decades. HIT is only about 40 years old, and has rocketed to widespread use in the last few years via the government’s incentive and subsidy programs. This has been a mixed blessing. We’ve learned a lot about its abilities and extraordinary potential, but we’ve also observed many of its challenges and errors.

We sincerely hope that HIT will achieve its currently unfulfilled promises of improved health outcomes and efficiency. But this will require policies focused on evidence-based analyses and data standards that allow interoperability, rather than a rush to obtain subsides for questionably effective functionalities designed to encourage sales in a compressed timetable.

Software sales must be secondary to rigorous and unbiased research examining whether such software represents good value for the trillions being spent by US hospitals and clinicians.

Stephen Soumerai, Sc. D., is Professor of Population Medicine at Harvard Medical School and Harvard Pilgrim Health Care Institute and Co-Chair of the Evaluative Sciences and Statistics concentration of the Harvard University-wide PhD Program in Health Policy.

Ross Koppel, PhD, FACMI, has taught sociology at the University of Pennsylvania for 21 years and is the PI on several studies at Penn’s Medical School.

Leave a Reply

41 Comments on "More Work Is Needed on the Safety and Efficacy of Healthcare Information Technology"


Guest
S Silverstein MD
Jul 24, 2013

New report on worsening of care quality due to today’s poorly conceived EHR systems, from the Cleveland Clinical Journal of Medicine.

Electronic siloing: An unintended consequence of the electronic health record
http://www.ccjm.org/content/80/7/406.full

Most of the observations in this new piece have been obvious to experienced clinicians for years, I add.

Guest
sleuth
Jul 21, 2013

OMG, I just listened to the Senate Finance Committee hearings on HITECH and the testimony of @farzad_onc. Losts of words but said nothing, just like most EHRs. The senators were impressed with the narcissistic approach.

Guest
legacyflyer
Jul 20, 2013

BobbyG,

You have challenged us “guys” – and by that you presumably mean those that are skeptical about the benefits of an EHR – to describe how a demonstration of safety and efficacy would work.

” What would comprise such a “demonstration”? It’ll never suffice for you guys.”

I am a practicing physician and not an expert on the design of research studies. But I will take your challenge.

1) Take 100 hospitals and divide them into two groups of 50
2) Require one group to adopt an EMR. No requirement for the other group. (Also no penalty for non-adoption)
3) Conduct a 6 month baseline evaluation following both groups for: complication rates, cost, length of stay, etc.
4) Follow with a 6 month test period during with one group adopts the EMR.
5) Again evaluate both groups for: complication rate, cost, length of stay, etc.

My prediction – those hospitals that have adopted the EMR will see an increased cost, BUT NO OTHER BENEFIT!

Conceptually this experiment does not seem all that difficult and given all the money being spent on EMRs should probably have been done before the requirements were rolled out. This could even be done on a smaller scale within hospitals chains that have multiple hospitals.

But of course we all KNEW how great EMRs were so we didn’t need to prove it. NOT

Guest
BobbyG
Jul 22, 2013

Now we’re getting somewhere. A reasonable proffer.

OK, but, as of today there are 318 ONC Certified inpatient “complete” systems.

– Would we have to multiply by some random sample fraction of those (say, maybe pick 10 systems randomly), and stratify? (e.g., those using EPIC ended up sucking, but those using McKesson or _________, _________, _________ did not). Or are we going to merely pit ONE system against paper? If so, which one would suffice as the generic proxy for all systems?

– would these be randomly chosen facilities (from among for-profit, community non-profit, and academic medical centers)?

– Given the materially differing sizes and case-mix characteristics of institutions, would we need more than just 50 vs 50?

– Would we allow for, say, at LEAST a 90 day post-implementation “shakedown/training” period prior to commencing the study, in order to keep from biasing the 6 months trial period (we already know that there’s a pretty substantial and to-be-expected post- go-live shakiness period)?

– Would time of year be a biasing factor one way or another? Geographic (incl sociodemographic) location?

N=50 in an experimental study cohort is pretty small even when you’re studying a single differential outcome variable. Where it encompasses the complexity inherent in this proposal, the sample size might not have sufficient “power,” once you get into inevitably having to stratify.

Thanks for your suggestion.

Guest
legacyflyer
Jul 22, 2013

Again, I am NOT an expert on experimental design but …

If the question we are seeking to answer is:

“Is an EMR better than paper?” – then I would suggest we let the decision makers in each of the hospitals choose which EMR they want – as is the case today.

“Is EPIC better than paper?” – then we would need to have all 50 hospitals use EPIC.

If EMR’s are really (take your pick); The Savior of Healthcare, The Best Invention Since Sliced Bread, The Bane of Our Existence, then the relatively small sample size should be enough.

On the other hand, if this experiment requires a huge sample and lots of data massage – well then I think that tells me something too.

Again, my prediction is that EMR will be shown to increase health care costs and offer no benefit to patients, physicians or nurses.

Prove me wrong.

Guest
BobbyG
Jul 22, 2013

Good stuff.

My speculation, though, is that the time window is too narrow. And, I am skeptical that costs will be materially reduced irrespective of HIT. Especially in across a six month period). Too many other confounding factors.

Consider the recent “Oregon Study” — 2 years, mildly “increased UTIL”, no “improvement in selected outcomes.”

OK, first, this was outpatient Medicaid, so there’s no direct tightly coupled analogy with inpatient. Even so, What In The Hell Were They Thinking?

Tangentially apropos, I’m 67, now a Medicare bene with the usual litany of Chronics. I’m a 99123 – 99214. I see my doc twice a year. That’s (n-1) df=3 across 2 years. You can’t conclude much in the way of outcomes in 2 years, as it pertains to — importantly — the A and P part of the SOAP that is in the doc’s brain and under his/her direction. Each tx is an “experiment” typically beset with a host of subsequent confounding factors and loosely coupled to eventual outcome.

“If EMR’s are really (take your pick); The Savior of Healthcare, The Best Invention Since Sliced Bread, The Bane of Our Existence,”

Anyone who reads my record knows that I am not cheerleading ANY of such unequivocal claims. Again, there is an efficacy distribution of Health IT (just as with any other technology) ranging from significantly value-adding to terrible and dangerous.

The HHS/ONC model envisions beginning to “bend the cost curve” starting around year 5 (MU Stage 3). We have thus far only had two years of “Meaningful Use” program compliance.

Guest
Doctor Mawrdough
Jul 20, 2013

Jha has been an unabashed cheerleader for HIT, when, he does not use the systems. I asked an HIT proponent kool aid server how to access a particular transition of care document and the response was, “my residents do it, I never did and do not know how”.

Jha should use the systems before pontificating how wonderful they are.

Koppel and Soumerai render a sobering and candid report. In order for any benefits to accrue, it behooves the lawmakers to know the adversity.

Guest
Caregiver, MD PhD
Jul 19, 2013

@koppel: “Please remember that both Steve and I teach research methods for a living.”

Wow, the HIT manufacturers should take lessons from you…but then again, they have HIMSS to spread the gospel and pay RAND to write favorable white papers.

Guest
ytisebo
Jul 19, 2013

Having observed from afar, the above commentary is long overdue. Kudos to the authors. Taxpayers are, indeed, being scammed and federal laws are being broken to enable the distribution and use of HIT sans application of the F D and C Act.

Guest
Whatsen Williams
Jul 18, 2013

Did anyone listen to the obtuse comments made by @farzad_onc at the Senate Finance Committee hearings on HIT? The also brought in a pediatrician who is high in command of CMS that insures old people. Hello?
The senators seemed to only be interested in numbers, a bit about privacy and hacking, some about care in the boon docks, and they all simply assumed that it was a safe building block for care. The spelling of patient has now become paytient. The computer is now the patient.

Guest
BobbyG
Jul 19, 2013

“The spelling of patient has now become paytient”
__

I like that. LOL.

But, that has long been the case.

Guest
Curly Harrison, MD
Jul 18, 2013

Pardon my naivete but what organization is providing assessment of the safety, efficacy, and usability of CPOE before it is sold and once it is being used?

From what little I know, CPOE and its associated decision support modules are serving to treat and mitigate disease, and thus, are medical devices.

Guest
Jul 19, 2013

“From what little I know”
__

Operative clause

Guest
ross koppel
Jul 18, 2013

Thanks for all of these comments. Most are stunningly supportive….and appreciated. I certainly agree that a randomized control double blind study of HIT is an absurd requirement. Can’t be done. But there is a lot of very obviously very bad research on HIT that is touted to justify spending on HIT. The perfect should not be the enemy of the good, but obviously inadequate research should not be used to justify locking us into a decade of mediocre-to-worse software.

Several points were raised: I’ll address a few:
with the forthcoming consolidation of the industry, the congress and/or the ONC or someone must immediately demand all patient data be secured in some independent facility so that those data are not lost when the companies go belly up. As part of my AHRQ study, we interviewed docs who were on their 5th EHR and who lost all data when three of the former companies went under. Increasingly also, large vendors are holding data hostage even from hospital systems. “Buy the upgrade or lose your data!!!” This is pure evil. I urge the ONC to demand that any vendor that does not protect patient data this minute should be prohibited from all commerce and decertified before the sun sets. Alas, even major vendors do this.

Next: Someone wrote: “throwing around charges making a federal case out of POOR USABILITY and DESIGN PHILOSOPHIES could be called alarmist.” I don’t know about design philosophies, but certainly concern for poor usability is not alarmist. I would say just the opposite: ignoring usability is ignoring the most basic patient safety needs. If the prescriber can’t see the other meds or the problem list or the allergies when ordering meds that’s just dangerous. I darn well hope someone is calling an alarm. In fact, I thought the comment made by mr “buba for president” was a joke. Usability is no joke! It’s real, and failure to have good usability kills patients and frustrates clinicians.

Dr. Jha is a serious scholar and I hope he reads our note in the spirit of scholarly concern. It’s certainly not intended as personal, and in fact we were too late to clarify some of that wording. We assume he and his coauthors are devoted to patient safety and are as enthusiastic about HIT as we are because of it’s patient safety potential and it’s promises to increase efficiency.

Please remember that both Steve and I teach research methods for a living. Very few physicians receive training in research design or in survey methods (the latter I’ve also been teaching for 35 years). I don’t attempt to do surgery or to diagnose diseases; but everyone thinks they can create, administer, and interpret a survey.
best,
Ross Koppel, Ph.D. FACMI

Guest
Steve Soumerai
Jul 18, 2013

Hi Ross: I should have read from the bottom up before commenting! At least we didn’t contradict each other! Nice note! Steve

Guest
S Silverstein MD
Jul 18, 2013

Re: ” This piece reads like a demand for … running 5+ year randomized clinical trials) ather giving us useful guide posts getting from where we are to where we want to be.”

I fail to understand, scientifically and ethically, why the research protocols of pharma and the medical device sector to avoid the debacles of the past (e.g., thalidomide and many others) should not apply to health IT, which is increasingly central to, and mediates, all transactions of care.

Guest
Bobby Gladd
Jul 18, 2013

Let’s hear your proposed HIT randomized trial study design, Scott. It would have to be a MANOVA on steroids, one stratified by all 70-odd medical specialties and every supported hardware system and firmware and OS release. I’m all for certification rigor. If you’ve published a design test proposal of such realism and stringency, let’s hear it. You certainly have the Sheet for it.

Calling for criminal prosecution of HIT vendors isn’t going to advance the cause one iota. First of all everyone has to sign Hold Harmless clauses. Second, the lawyers are wealthy enough.

Guest
S Silverstein MD
Jul 21, 2013

Actually, I don’t just “call” for prosecution, I actively educate and assist trial lawyers in prosecuting cases of EHR-related harms and evidence spoliation (tampering).

As opposed to your apparent belief in laissez-faire, unprecedented in healthcare since at least the FD&C Act, I call for robust post marketing empirical studies of health IT risks, done with transparency (as called for by IOM and others), as well as pre-marketing software validation such as spelled out in depth in documents such as “General Principles of Software Validation; Final Guidance for Industry and FDA Staff” and “Certification Processes for Safety-Critical and Mission Critical Aerospace Software.”

Guest
Steve Soumerai
Jul 19, 2013

Bobby: There are already a fair number of well designed, SIMPLE and large randomized clinical trials. If they are not dramatic they have less chance of publication in the major journals than a untrustworthy correlational study that “shows ” improved outcomes (there are data). Just one terrific example: the study led by Jerry Gurwitz, published in a geriatrics journal around 2007-8 (senior authored by David Bates). They studied an advanced CPOE system in a large number of horsing homes in Connecticut and found reduced drug interactions but zero effect on drug injuries–their main objective. Indisputably ineffective- so much so that this elegant cluster randomized trial didn’t get published in the top journal it deserved. Take a look on Google Scholar. Best, Steve Soumerai

Guest
BobbyG
Jul 20, 2013

ONE study, in CT “horsing homes,” published 5-6 years ago? Meaning the data collection and analysis was well before that?

Well, uh, OK… how timely.

Got a link?

Guest
Steve Soumerai
Jul 20, 2013

Shoot. That’s the review before their own RCT. Emaill me at ssoumerai@hms.harvard.edu on Tues when back and I’ll send you the article. My project manager has it. Best, S

Guest
BobbyG
Jul 20, 2013

Thanks, Steve.

Guest
Steve Soumerai
Jul 20, 2013

Hi Bobby: My MAC spell checker is the cause of too high HITECH and nursing homes for horses!! Another cool software error. I gotta lose it.

Here is the link. Regardless of the 2008 timing, they changed important interactions but no adverse events in a big trial.
http://www.ncbi.nlm.nih.gov/pmc/articles/PMC2359507/

If doesn’t work you have PMC number. S

Guest
Jul 18, 2013

The question is when will user centered modern interfaces attached to cloud based systems (what I call health 2.0) start sucking the data out of the ERP-type monstrosities that Steve, Ross & Scott criticize? When that happens, these systems will be easier to use, easier to change and also provide better access to data.

My sense is that the route to heaven is for Steve, Ross, Scott and the rest to start demanding the easy access to data, and the legislative demands for real inter-operability. which IS IN the meaningful use regs (however they actually get enforced).

This piece reads like a demand for a return to paper, (or at least to no new progress while we run 5+ year randomized clinical trials) rather giving us useful guide posts getting from where we are to where we want to be.

Guest
40yearold doc
Jul 19, 2013

“When that happens, these systems will be easier to use, easier to change and also provide better access to data”

As long as any system is designed for the benefit of CMS and large insurers (ICD, CPT, MU, P4P, and on and on), and not for the benefit of patients and physicians, nothing will get any better.

Guest
improveoutcomes
Jul 19, 2013

“As long as any system is designed for the benefit of CMS and large insurers (ICD, CPT, MU, P4P, and on and on), and not for the benefit of patients and physicians, nothing will get any better.”

Couldn’t agree more. This is it. It speaks directly to why the marriage of healthcare and IT has failed all of us (outside of the EMR companies). Unfortunately, divorce is now not an option so we must be be vigilant in exposing and ignoring the “solutions” that expose and ignore patients and physicians.

Guest

This is a bit harsh. If you look at what Ashish has written over the past few months – you’ll see plenty of nuance, which is something that probably cannot be said about this post. His main point – which is that there is GOOD NEWS on adoption rates, is well worth talking about.

As to your critique, reliable studies on the benefits of new technologies are NOTORIOUSLY DIFFICULT to pull off. Presumably you know this. While we’re throwing around charges making a federal case out of POOR USABILITY and DESIGN PHILOSOPHIES could be called alarmist.

Lastly, technology companies are technology companies. You can expect them to behave like technology companies. They’ll seek advantage. They’ll seek to control data. They’ll seek to retain their customer base. Just like pharma companies. Just like health plans. Just like pretty much every other actor in the system. When the market shifts, they’ll change their behavior. Until that happens, it ain’t going to happen …

Guest
Steve Soumerai
Jul 18, 2013

Those are reasonable points! As a research design educator, however, I have to say that the majority of poor studies were preventable. We don’t insist on RCTs.But there are some good ones- mostly negative. But please, measure some sort of change. Use longitudinal designs like interrupted time series (often those data are available) to control for pre-existing trends.Find a comparison series. I have seen too many correlational, cross-sectional studies passed off for cause and effect. And those studies almost always favor HIT. Naturally because the places that had it first are bigger, have higher volumes of procedures, are academic medical centers, have younger patients, etc., etc. The science is flawed in the pursuit of campaigns. Best, S

Guest
Dr. Rick Lippin
Jul 18, 2013

Scot-

Thanks for your leadership and activism. I also believe that once proven guilty of crimes including homocide that some of these HIT leaders should receive jail sentences. Lawsuits, fines and bad publicity are not enough and are not working.

Dr. Rick Lippin
Southampton,Pa

Guest
tschwiet MD
Jul 18, 2013

I am a fourth generation physician and was wisely told by my father and grandfather that medicine is part art and part science. Early in my career, I was naively biased toward the science aspects, and that data and logical decision-making on that data were the keys to great medicine. As I progressed in my career it became more apparent to me that the ‘art of medicine’ aspects prevailed as the dominant force to being a great primary care doc. Detecting subtleties, nuance, and seeing the underlying psychological factors, areas of resistance, and individual barriers gave me insights into what therapeutic approach, in line with evidence-based guidelines, made sense for that patient.
EMR’s today are unequivocally biased toward capturing and presenting data. Whether it is found in protocols, templates, auto-text, and cut-paste, this data overload leaves little time nor room for elements more related to the ‘art of medicine’ . Yes, you can color your notes with free text trying to instill some nuance and context to the documentation, but it simply gets lost amidst the avalanche of ACII symbols describing meaningless normal or negative findings.
In my opinion, the rush to electronic documentation methods have hi-jacked the clinical record away from being a scripted narrative of the patient condition useful to a doc and towards a billing-oriented, malpractice preventing, data repository useful to payers, billers, and regulators.
In an era where leaders are trying to improve patient-centricity, provider empathy, and patient-engagement, I am not sure a strongly data oriented approach is the right angle to take.

Guest
Steve Soumerai
Jul 18, 2013

tschwiet: I guess the research supports the enormous value of both the science and art of medicine. We once were rightly criticized for comparing the quality of care based only on adherence to guidelines. We missed half the story. The studies and op-eds of other physicians frequently support your points. EHRs can interfere with doctor patient communication and can lead to simple minded and confusing clinical documentation. Nice point about patient centered care. Best, Steve Soumerai

Guest
Jul 18, 2013

A poorly designed but highly touted EHR led to the rather cruel injury and death to my mother – in the very hospital where I had done my residency in the mid 1980’s. I am postdoctorally trained in Medical Informatics as well.

The events that occurred can happen anywhere and to anybody. May you and your family members be more fortunate.

Good health IT is not what the industry is currently producing. What is the current standard is bad health IT. Definitions at http://www.ischool.drexel.edu/faculty/ssilverstein/cases .

In essence, the health IT hyper-enthusiasts who ignore the current problems and push today’s bad health IT on non-consenting patients and clinicians are maiming and, in effect, contributing to murder in their zeal to “break a few eggs to improve healthcare” – or to make money.

I and like-minded colleagues will begin pushing for charges of gross or criminal negligence, for instance in injuries or deaths that might occur in cases such as I write about at “RNs Say Sutter’s New Electronic System Causing Serious Disruptions to Safe Patient Care at East Bay Hospitals”,http://hcrenewal.blogspot.com/2013/07/rns-say-sutters-new-electronic-system.html .

Perhaps the accountability of a possible jail sentence might temper the irrational exuberance for today’s bad health IT.