I’ve been talking in recent posts about how our typical methods of testing AI systems are inadequate and potentially unsafe. In particular, I’ve complainedthat all of the headline-grabbing papers so far only do controlled experiments, so we don’t how the AI systems will perform on real patients.
Today I am going to highlight a piece of work that has not received much attention, but actually went “all the way” and tested an AI system in clinical practice, assessing clinical outcomes. They did an actual clinical trial!
Big news … so why haven’t you heard about it?
The Great Wall of the West
Tragically, this paper has been mostly ignored. 89 tweets*, which when you compare it to many other papers with hundreds or thousands of tweets and news articles is pretty sad. There is an obvious reason why though; the article I will be talking about today comes from China (there are a few US co-authors too, not sure what the relative contributions were, but the study was performed in China).
China is interesting. They appear to be rapidly becoming the world leader in applied AI, including in medicine, but we rarely hear anything about what is happening there in the media. When I go to conferences and talk to people working in China, they always tell me about numerous companies applying mature AI products to patients, but in the media we mostly see headline grabbing news stories about Western research projects that are still years away from clinical practice.
This shouldn’t be unexpected. Western journalists have very little access to China**, and Chinese medical AI companies have no need to solicit Western media coverage. They already have access to a large market, expertise, data, funding, and strong support both from medical governance and from the government more broadly. They don’t need us. But for us in the West, this means that our view of medical AI is narrow, like a frog looking at the sky from the bottom of a well^.
It is easy for armchair activists to bash randomized controlled trials (RCTs) with clever methodological critiques. However, it takes a lot of effort and coordination to pull off an RCT successfully. In this episode of Radiology Firing Line, I speak with Dr. Mark Neuman and Lakisha Gaskins, principal investigator and research project manager of the REGAIN trial, respectively, about the logic, challenges and intricacies of conducting an RCT. The Regional versus General Anesthesia for Promoting Independence After Hip Surgery (REGAIN) trial is an ongoing pragmatic, multi-center RCT, funded by PCORI, which randomizes patients with hip fractures to regional or general anesthesia.
Guests: Mark Neuman MD MSc, is an Associate Professor of Anesthesiology and Critical Care at the University of Pennsylvania. He is a senior fellow at the Leonard Davis Institute of Health Economics. He’s a former RWJ Scholar. Lakisha Gaskins is a research coordinator with extensive experience recruiting patients for RCTs.
Listen to our conversation on Radiology Firing Line Podcast here.
Saurabh Jha is a contributing editor to THCB and host of Radiology Firing Line Podcast of the Journal of American College of Radiology, sponsored by Healthcare Administrative Partner.
NEHI recently convened a meeting on health care innovation policy at which the Harvard economist David Cutler noted that debate over innovation has shifted greatly in the last decade. Not that long-running debates about the FDA, regulatory approvals, and drug and medical device development have gone away: far from it.
But these concerns are now matched or overshadowed by demands for proven value, proven outcomes and, increasingly, the Triple Aim, health care’s analog to the “faster, better, cheaper” goal associated with Moore’s Law.
To paraphrase Cutler, the market is demanding that cost come out of the system, that patient outcomes be held harmless if not improved, and it is demanding innovation that will do all this at once. Innovation in U.S. health care is no longer just about meeting unmet medical need. It is about improving productivity and efficiency as well.
In this new environment it‘s the science-driven innovators (the pharma, biotech, and medtech people) who seem like the old school players, despite their immersion in truly revolutionary fields such as genomic medicine. It’s the tech-driven innovators (the healthcare IT, predictive analytics, process redesign, practice transformation and mobile health people) who are the cool kids grabbing the attention and a good deal of the new money.
To make matters worse for pharma, biotech and medtech, long-held assumptions about our national commitment to science-driven innovation seem to be dissolving. There’s little hope for reversing significant cuts to the National Institutes of Health. User fee revenues painstakingly negotiated with the FDA just last year have only barely escaped sequestration this year. Bold initiatives like the Human Genome Project seem a distant memory; indeed, President Obama’s recently announced brain mapping project seems to barely register with the public and Congress.