Medicine Is a Profession That is Rapidly Losing Control of Its Tools.

Artificial Intelligence hype and reality are everywhere. However, the last month or two has seen some thoughtful reflection. HHS / ONC announced “Hype to Reality: How Artificial Intelligence (AI) Can Transform Health and Healthcare” referencing a major JASON report “Artificial Intelligence for Health and Health Care [PDF -817 KB],”. From a legal and ethical perspective, we have a new multinational program: “PMAIL will provide a comparative analysis of the law and ethics of black-box personalized medicine,…”. Another Harvard affiliate writes “Optimization over Explanation” subtitled “Maximizing the benefits of machine learning without sacrificing its intelligence”. Meanwhile, an investigative journalism report from the UK “Google DeepMind and healthcare in an age of algorithms”, “…draws a number of lessons on the transfer of population-derived datasets to large private prospectors, identifying critical questions for policy-makers, industry and individuals as healthcare moves into an algorithmic age.”

What sets these major papers apart from the constant AI and Big Data science or hype is that they are not particularly technical. Although long, they are accessible to pretty much anyone including physicians and patients. What shocks me, as a career medical technology engineer also trained as a physician, is how passive physicians and patients seem as medicine shifts from open public science to secret corporate black boxes.

Medicine is a profession that is rapidly losing control of its tools to corporate and political interests. First, they came for the information itself and introduced electronic health records that doctors do not purchase or control. Some doctors, frustrated by the loss of control, retired but most just complained and learned to type faster. Now they’re coming for the very core of the profession as a “learned intermediary” trusted by the patient. Black box AI may pay lip service to the physician as an intermediary but everyone understands that it’s the secret and proprietary machine that’s doing the heavy lifting. In machine we trust.

From a patient and physician perspective, two things define our loss of control and shifting trust: health records and secret methods. Patients have lost control over health records. Since 2002, corporations are able to use our health records without our consent. The trend to involuntary and hidden surveillance of everyone continues and grows to nationwide scale with the new HHS administration. This, we’re told, is the price we must pay to keep us safe and make research, including machine intelligence, effective.

Secret methods, we’re told, is the price we must pay to bypass the limitations of human understanding. Asking the AI to explain itself or making the AI an open public good like medicine used to be would simply slows progress and costs lives. Patients lose control of the records that train the AI. Physicians lose control of the open science that used to be the foundation of trust in the profession.

I firmly believe in the power and promise of technology, including AI, but we must not cede control out of greed and frustration.  As the JASON report clearly states, EHRs don’t have access to the full range of trusted information the AI needs. We can build a national health surveillance bureaucracy like European countries have but it’s unlikely to have access to our cell phones and the social determinants of health. We can also regulate black box AI at a corporate level but that takes the power of innovation and experimentation away from the licensed clinician.

AI is becoming an essential part of medicine. Patients need to trust their most sensitive data to the machines the way we used to trust our physicians before institutional EHRs and national-scale data breaches. Doctors need to remain the learned intermediary by keeping full transparency and interactivity with the machine. Secret software and black-boxes do not respect the role of physician as the root of trust and decentralized regulation. The current approach to AI is not in the public interest and the medical profession needs to snap out of its daze.

Livongo’s Post Ad Banner 728*90

Categories: Uncategorized

Leave a Reply

4 Comment threads
1 Thread replies
Most reacted comment
Hottest comment thread
5 Comment authors
William Palmer MDMatthew HoltDeborah_C_Peel_MDMargalit Gur-Ariepjnelson Recent comment authors
newest oldest most voted
William Palmer MD
William Palmer MD

If AI becomes its most glorious, it will be a “thinking” being. I.e. we shall be able to allow it to read Harrison’s Textbook of Internal Medicine and it will be able to suggest the symmetrical arthritis we are seeing in this particular patient is because of Mediterranean Fever. Wow. It may not be able to specify and print out for us its thinking, or algorithms, anymore than a creative person could tell us why she is making a chess move…or has a hunch there is a dangerous man in the elevator. But, as AI becomes a real being and… Read more »


The key thing I would add to Adrian’s brilliant essay is: AI does NOT have to be secret. In fact, secret AI should be illegal. And your pii should never be collected w/o meaningful informed consent, transparency and accountability, these are absolute requirements for trust in AI. If AI for ‘healthcare’ so great, everyone would agree to help. (key disclosure: Adrian Gropper MD is the CTO of Patient Privacy Rights) Autonomy, choice, and self-determination are the fundamental human rights required for freedom and Democracy, not just for the practice of Medicine. The context we live in today matters, every generation… Read more »

Matthew Holt

Oh man, this is an amazing day, I agree with (some of) what Deb Peel is saying. Those AI algorithms must be open for some kind of inspection/testing — otherwise we risk the Youtube algorithms phenomenon https://www.theguardian.com/technology/2018/feb/02/how-youtubes-algorithm-distorts-truth

Margalit Gur-Arie

Some of us have been complaining about the inherent surveillance mechanisms built into structured-data EHRs for a very, very long time, and sad to say, some of us have vague memories of being ridiculed and impatiently dismissed by people who now seem so woke to the dangers of what that surveillance net has begotten. The entire narrative about giving data to patients who control it and to doctors when they need it and such other nonsense is and always was a ruse. Data is exclusively owned and controlled by those who own the database where it resides, and those owners… Read more »


Let’s see: the Academy of Health is under the mantra of Congress, the National physician specialties evolve under the mantra of the Association of Medical Specialties; the legally separate Medical Staff of each hospital is under the mantra of their hospital’s parent enterprise; the medical schools survive under the mantra of the Association of American Medical Colleges; Primary Healthcare is under the mantra of it underfunded codependency with the sources of its financial support; Complex Healthcare is under the mantra of its institutional market-share; AND there is no connection between the availability and accessibility of Primary Healthcare and its needs… Read more »