This is the part two of a three-part series. Catch up on Part One here.
Preetham Srinivas, the head of the
chest radiograph project in Qure.ai, summoned Bhargava Reddy, Manoj Tadepalli, and
Tarun Raj to the meeting room.
“Get ready for an all-nighter, boys,”
Qure’s scientists began investigating
the algorithm’s mysteriously high performance on chest radiographs from a new
hospital. To recap, the algorithm had an area under the receiver operating
characteristic curve (AUC) of 1 – that’s 100 % on multiple-choice question
“Someone leaked the paper to AI,”
“It’s an engineering college joke,”
explained Bhargava. “It means that you saw the questions before the exam. It
happens sometimes in India when rich people buy the exam papers.”
Just because you know the questions
doesn’t mean you know the answers. And AI wasn’t rich enough to buy the AUC.
The four lads were school friends from
Andhra Pradesh. They had all studied computer science at the Indian Institute
of Technology (IIT), a freaky improbability given that only hundred out of a
million aspiring youths are selected to this most coveted discipline in India’s
most coveted institute. They had revised for exams together, pulling
all-nighters – in working together, they worked harder and made work more fun.
No one knows who gave Rahul Roy
tuberculosis. Roy’s charmed life as a successful trader involved traveling in his
Mercedes C class between his apartment on the plush Nepean Sea Road in South
Mumbai and offices in Bombay Stock Exchange. He cared little for Mumbai’s weather.
He seldom rolled down his car windows – his ambient atmosphere, optimized for
his comfort, rarely changed.
Historically TB, or
“consumption” as it was known, was a Bohemian malady; the chronic suffering produced
a rhapsody which produced fine art. TB was fashionable in Victorian Britain, in
part, because consumption, like aristocracy, was thought to be hereditary. Even
after Robert Koch discovered that the cause of TB was a rod-shaped bacterium –
Mycobacterium Tuberculosis (MTB), TB had a special status denied to its immoral
peer, Syphilis, and unaesthetic cousin, leprosy.
TB became egalitarian in the early twentieth
century but retained an aristocratic noblesse oblige. George Orwell may have
contracted TB when he voluntarily lived with miners in crowded squalor to
understand poverty. Unlike Orwell, Roy had no pretentions of solidarity with
poor people. For Roy, there was nothing heroic about getting TB. He was
embarrassed not because of TB’s infectivity; TB sanitariums are a thing of the
past. TB signaled social class decline. He believed rickshawallahs, not
traders, got TB.
Despite an area under the ROC curve of 1, Cassandra’s
prophesies were never believed. She neither hedged nor relied on retrospective
data – her predictions, such as the Trojan war, were prospectively validated. In
medicine, a new type of Cassandra has emerged –
one who speaks in probabilistic tongue, forked unevenly between the
probability of being right and the possibility of being wrong. One who, by conceding
that she may be categorically wrong, is technically never wrong. We call these
new Minervas “predictions.” The Owl of Minerva flies above its denominator.
Deep learning (DL) promises to transform the prediction
industry from a stepping stone for academic promotion and tenure to something
vaguely useful for clinicians at the patient’s bedside. Economists studying AI believe that AI is revolutionary,
revolutionary like the steam engine and the internet, because it better predicts.
Recently published in Nature, a sophisticated DL algorithm was able to predict acute kidney injury (AKI), continuously, in hospitalized patients by extracting data from their electronic health records (EHRs). The algorithm interrogated nearly million EHRS of patients in Veteran Affairs hospitals. As intriguing as their methodology is, it’s less interesting than their results. For every correct prediction of AKI, there were two false positives. The false alarms would have made Cassandra blush, but they’re not bad for prognostic medicine. The DL- generated ROC curve stands head and shoulders above the diagonal representing randomness.
The researchers used a technique called “ablation analysis.”
I have no idea how that works but it sounds clever. Let me make a humble
prophesy of my own – if unleashed at the bedside the AKI-specific, DL-augmented
Cassandra could unleash havoc of a scale one struggles to comprehend.
Leaving aside that the accuracy of algorithms trained
retrospectively falls in the real world – as doctors know, there’s a difference
between book knowledge and practical knowledge – the major problem is the
effect availability of information has on decision making. Prediction is
fundamentally information. Information changes us.
How easy is it for physicians to choose wisely and reject low value care? Who decides what’s wise and what’s unwise? In this episode Saurabh Jha (aka @RogueRad) speaks with William Sullivan MD JD. Dr. Sullivan is an emergency physician and an attorney specializing in healthcare issues. Dr. Sullivan represents physicians and has published many articles on legal aspects of medicine. He is a past president of the Illinois College of Emergency Physicians and a past chair and current member of the American College of Emergency Physicians’ Medical Legal Committee.
On Episode 3 of HardCore Health, Jess & I start off by discussing all of the health tech companies IPOing (Livongo, Phreesia, Health Catalyst) and talk about what that means for the industry as a whole. Zoya Khan discusses the newest series on THCB called, “The Health Data Goldilocks Dilemma: Sharing? Privacy? Both?”, which follows & discuss the legislation being passed on data privacy and protection in Congress today. We also have a great interview with Paul Johnson, CEO of Lemonaid Health, an up-and-coming telehealth platform that works as a one-stop-shop for a virtual doctor’s office, a virtual pharmacy, and lab testing for patients accessing their platform. In her WTF Health segment, Jess speaks to Jen Horonjeff, Founder & CEO of Savvy Cooperative, the first patient-owned public benefit co-op that provides an online marketplace for patient insights. And last but not least, Dr. Saurabh Jha directly address AI vendors in health care, stating that their predictive tools are useless and they will not replace doctors just yet- Matthew Holt
Matthew Holt is the founder and publisher of The Health Care Blog and still writes regularly for the site.
Recently, my niece gingerly
confided that she was going to study engineering rather than medicine. I was
certain she’d become a doctor – so deep was her love for biology and her
deference to our family tradition. But she calculated, as would anyone with
common sense, that with an engineering degree and an MBA, she’d be working for
a multinational company making a comfortable income by twenty-eight. If she
stuck with tradition and altruism, as a doctor she’d still be untrained and
preparing for examinations at twenty-eight.
Despite the truism in India that
doctors are the only professionals never at risk of starving, the rational case
for becoming a physician never was strong. Doctors always needed a dose of the
irrational, an assumption of integrity and an unbridled goodwill to keep going.
Once, doctors commanded both the mystery of science and the magic of
metaphysics. As medicine became for-profit, the metaphysics slowly disappeared.
Indians are becoming more
prosperous. They’re also less fatalistic and expect less from their gods and
more from their doctors. In the beginning they treated their doctors as gods, now
they see that doctors have feet of clay, too. Doctors, who once outsourced the
limitations of medicine to the will of Gods, summarized by the famous Bollywood
line “inko dawa ki nahin dua ki zaroorat hai” (patient needs prayers not
drugs), now must internalize medicine’s limitations. And there are many –
medicine is still an imperfect science, a stubborn art, often an optimistic breeze
fighting forlornly against nature’s implacable gale.
In 1999, the Institute of Medicine (IOM) in their landmark report – To Err is Human – estimated that the number of deaths from medical errors is 44 ,000 to 98, 000. The report ushered the Quality and Safety Movement, which became a dominant force in all hospitals. Yet the number of deaths from medical errors climbed. It is now touted to be the 3rd leading cause of death. How easy is it to precisely quantify the number of deaths from medical errors? Not many physicians challenged the methodologies of the IOM report. Some feared that they’d be accused of “making excuses for doctors.” Many simply didn’t have a sufficient grip on statistics of measurement sciences. One exception was Rodney Hayward – who was then an early career researcher, a measurement scientist, who studied how sensitive the estimates of medical errors were to a range of assumptions.
Rod Hayward a Professor of Public Health and Internal Medicine at the University of Michigan and Co-Director of the Center for Practice Management and Outcomes Research at the Ann Arbor VA HSR&D. He received his training in health services research as a Robert Wood Johnson Clinical Scholar at UCLA and at the RAND Corporation, Santa Monica. His current and past work includes studies examining measurement of quality, costs and health status, environmental and educational factors affecting physician practice patterns, quality improvement, and physician decision making. His current work focuses on quality measurement and improvement for chronic diseases, such as diabetes, hypertension and heart disease.
Listen to their conversation on Radiology Firing Line Podcast here.
Can we reduce over diagnosis by re-naming disease to less anxiety-provoking makes? For example, if we call a 4.1 cm ascending aorta “ecstasia” instead of “aneurysm” will there be less over-treatment? In this episode of Radiology Firing Line Podcast, Saurabh Jha (aka @RogueRad) discusses over diagnosis with Ian Amber, a musculoskeletal radiologist at Georgetown University, Washington.
What does it take to create a decision rule? In this episode of Radiology Firing Line podcast Saurabh Jha (@RogueRad) has a discussion with Robert W. Yeh MD MBA about the deep thought and complex statistics involved in creating a decision rule to guide therapy which have narrow risk-benefit calculus, specifically a rule for how long patients should continue dual anti-platelet therapy after percutaneous coronary intervention. They also discuss the motivation behind the legendary, and satirical, parachute RCT published in the recent Christmas edition of the BMJ, which delighted satirists all over the world.
In this episode of Radiology Firing Line Podcast, I speak with Bishal Gyawali MD, PhD. Dr. Gyawali obtained his medical degree from Kathmandu. He received a scholarship to pursue a PhD in Japan. Dr. Gyawali’s work focuses on getting cheap and effective treatment to under developed parts of the world. Dr. Gyawali is an advocate for evidence-based medicine. He has published extensively in many high impact journals. He coined the term “cancer groundshot.” He was a research fellow at PORTAL. He is currently a scientist at the Queen’s University Cancer Research Institute in Kingston, Ontario.