Currently, three South Korean medical institutions – Gachon University Gil Medical Center, Pusan National University Hospital and Konyang University Hospital – have implemented IBM’s Watson for Oncology artificial intelligence (AI) system. As IBM touts the Watson for Oncology AI’s to “[i]dentify, evaluate and compare treatment options” by understanding the longitudinal medical record and applying its training to each unique patient, questions regarding the status and liability of these AI machines have arisen.

Given its ability to interpret data and present treatment options (along with relevant justifications), AI represents an interim step between a diagnostic tool and colleague in medical settings. Using philosophical and legal concepts, this article explores whether AI’s ability to adapt and learn means that it has the capacity to reason and whether this means that AI should be considered a legal person.

Through this exploration, the authors conclude that medical AI such as Watson for Oncology should be given a unique legal status akin to personhood to reflect its current and potential role in the medical decision-making process. They analogize the role of IBM’s AI to those of medical residents and argue that liability for wrongful diagnoses should be generally based on a medical malpractice basis rather than through products liability or vicarious liability. Finally, they differentiate medical AI from AI used in other products, such as self-driving cars.

Legal scholarship pertaining to artificial intelligence (AI), especially in light of its increasing sophistication and decision-making prowess, is still nascent. Courts have traditionally deemed it impossible for machines to have legal liability as they are not legal persons. Indeed, U.S. courts have previously outlined that “robots cannot besued”. However,some modern commentators argue that a rethinking of legal systems is necessary to deal with AI considering its growing capabilities.

Controversy about how to treat AI from a legal standpoint reflects deeper disagreements about how humans should interact with such technologies, with debate about AI’s potential tending to gravitate towards the extremes. Decades of popular culture depictions reflect this dichotomy. Isaac Asimov envisions a world of peaceful co-existence with sentient robots, with the First Law of Robotics dictating that robots may not injure a human or allow one to come to harm. At the other end of the spectrum, we have dystopias such as Terminator’s Skynet or The Matrix, wherein sentient machines overtake humanity as the world’s dominant species.

AI has not yet achieved sentience and presently exists only in the form of useful tools rather than as a race capable of truly interacting with humans – let alone challenging us for global dominance. However, this does not mean that AI is incapable of interpretation or giving advice. AI employing cognitive learning is regularly used to parse through data and to offer options to human operators.

IBM Watson for Oncology (hereinafter “Watson”) represents the most compelling usage of such technology to date. Clinicians and analysts train Watson, a so-called cognitive computing system, to “interpret cancer patients’ clinical information and identify individualized, evidence-based treatment options.” But does this indicate a machine’s ability to reason? And, if so, should machines be held liable for exercising poor judgment?

This paper highlights Watson’s duties and abilities to argue that existing legal regimes are sufficient to deal with AI’s (including Watson’s) “humanity” at its current stage of technological advancement, and to adequately attribute blame and liability for its errors. By adopting a practical and responsibility-based approach that assesses Watson’s influence on treatment outcomes, legal regimes should enjoy a flexible and workable framework within which to regulate existing AI responsibility and liability while a broader debate about the rights and role of AI in human society takes place.

What is Artificial Intelligence Anyway?

Artificial intelligence is commonly understood as intelligence displayed by machines, the ability and development of machines to perform tasks that normally require human intelligence, or some such variation. AI applications are already commonplace in our daily activities, such as when we ask Siri if it’s raining, or request that Alexa play some bluegrass music. These types of AI applications, focused on the execution of a single task, are referred to alternatively as “weak” or “narrow” AI.

In contrast, many people associate AI with Artificial General Intelligence (AGI), or “strong” AI, which does not yet exist. This refers to the achievement of machine sentience or consciousness; AGI computers would possess or display intelligence equivalent to that of humans in every respect. Various tests have been developed that would purportedly detect the existence of true AGI, such as the Turing Test – in which a human would be unable to distinguish the responses of an AGI and a human to various posed questions. Ray Kurzweil, Google’s Director of Engineering and a noted futurist with a track record for accurate predictions (he claims an 86% success rate out of his 147 predictions since the 1990s), has consistently predicted 2029 as the year an AI will first pass a valid Turing Test.3 AGI would be capable of recursive self-improvement, leading to the rapid emergence of Artificial Superintelligence (ASI), the limits of which are unknown – but which many predict will result in the so-called “technological singularity,” perhaps entailing the obsolescence of human beings.

For now, we have expert systems, which have existed and evolved since the 1970s: computer programs that utilize AI methodologies to solve problems within a specialized subject area. Such domains may include anything from logic problems and games like chess to financial investing, legal research, and of course, medical decision-making. Expert systems that extend beyond binary logical inquiries (yes/no, true/false) rely on “fuzzy logic,” which must process linguistic terms in conditions of imprecise knowledge.4

These philosophical categorizations of AI inform the discussion herein.

What is Watson for Oncology?

IBM’s AI software or “expert system”, Watson (named after IBM’s first CEO, industrialist Thomas J. Watson, and not Sherlock Holmes’ sidekick, as is popularly misattributed), arrived on the popular radar when it beat two of Jeopardy’s all-time champions in 2011. The following year, IBM announced Watson’s first practical collaboration – with the Cleveland Clinic, in hopes that the system’s ability to synthesize huge amounts of data and produce evidence-based hypotheses could aid clinicians and students in more accurately diagnosing and treating patients.

IBM wanted to create a “supercharged Siri for business” and it essentially has. Watson now uses the same “Deep QA” software it used to achieve quiz show glory to tackle some of the healthcare system’s most complex problems – including properly diagnosing and treating various forms of cancer. Deep QA, in the simplest terms, is the software architecture that “analyzes, reasons about, and answers the content fed into Watson.” Watson’s abilities in the oncology space are ever expanding. In 2014, IBM announced that physicians could start using Watson to connect genomic and medical data to help drive more personalized treatments, and Business Insider has predicted that Watson will eventually allow oncologists to “upload the DNA fingerprint of a patient’s tumor,” identify which genes are mutated, and then sort through thousands of mutations to identify which is driving the tumor and can be targeted through drug therapies.

Expert systems – with Watson on the cutting edge – comprise “the most visible effects of research on artificial intelligence,” containing a knowledge base and inference tools that allow the user to both ask questions in natural language and receive a reply in the same language.7 Nonetheless, expert systems solve problems only in “a narrow and well-defined area.”

While Watson is therefore a far cry from AGI or the replacement of doctors with cyborgs, IBM frequently (though not always) shies away from specifically calling Watson “AI”, favoring alternate nomenclatures such as “augmented intelligence,” or as in recent TV commercials, “the platform for cognitive business.” Despite Watson representing one of the most sophisticated uses of narrow AI to date, this is perhaps unsurprising given the above-discussed confusion surrounding the definitional tiers of AI. Indeed, IBM’s fine distinction between “cognitive computing” and “artificial intelligence” appears more designed to placate those wary of the decline of human centrality and agency in the healthcare decision-making process rather than an accurate representation of the intensive research and data gathering and interpretation work carried out by Watson – work which is key to outlining the options for treatment of human subjects. Already, Watson is being used to score and rank medical literature and summarize patient records – key tasks in helping to contextualize and justify treatment options. Watson currently draws on more data than any reasonable human can be expected to consult, including 300 medical journals, 200 textbooks, and nearly 15 million pages of text in order to present treatment alternatives, drug options, and instructions for therapeutic administration.

Rising concordance rates suggest that Watson is on the cusp of being viewed as a viable, reliable option for vetting and applying such vast quantities of information. In June 2017, data presented at a meeting of the American Society of Clinical Oncology suggested Watson was highly likely to reach the same treatment conclusions as human doctors. At India’s Manipal Comprehensive Cancer Center, there was agreement between Watson and doctors on the appropriate course of treatment for 96.4% of 112 cases of lung cancer. For other forms of cancer, concordance rates ranged from 81% to 92.7%.

Act only according to that maxim by which you can at the same time will that it should become a universal law.

Much like Descartes, Kant views humans as the only beings capable of comprehending, establishing and respecting CIs. The very concept of morality and responsible decision-making, according to Kant, is predicated on the conceit that the treatment of persons is an “end and never a means only.”21 In practical terms, that means difficult decisions can only arise from humans by humans as no other entity would have the necessary intrinsic qualities to filter human experience and values. That may be why Kant viewed human beings as “above all price.”

From a broader societal perspective, utilitarian philosophers such as Jeremy Bentham, John Stuart Mill, and David Hume advanced the notion that raising overall human happiness should be the primary goal of human decision-making. Their consequences-based approach and devotion to the “common good” is again a human-centric one and one that appears too nuanced for AI to understand. After all, even though AI excels at pure analytical calculations, how would it measure and compare intangible goods so that they can be weighed and pitted against one another?

These examples are not meant to be comprehensive but to demonstrate that morality is viewed as the key to principled decision-making among humans. Judging from the diversity and breadth of philosophical texts on morality, balancing individualistic and societal goals is an exceedingly difficult process involving a metaphysical arithmetic that is highly unsettled and characteristically ill-suited to rigid categorization and valuation.

For a decision to be viewed as good and principled, however, it will have to be consistent with at least some vision of human morality. And this proves to be the greatest difficulty for most when thinking about whether we should accept AI as a component of healthcare system – how can a machine devoid of intrinsic human sensations, perceptions, and intuition be trusted to make the ‘right’ decisions? Even more crucially, how can AI be trusted to choose the correct option in cases where there are competing moral views and interests?

To be fair, this issue is not specific to “robots” and perplexes human decision makers as well. Even a cursory review of Western moral philosophy shows a wide disparity in what people believe makes humans “special” and how humans should interpret their universe and interact with others. This demonstrates the ambiguity of the supposed morality that underpins modern society.

After all, even the weightiest human expression of values, our system of laws expressed through judicial systems, generally allows for some degree of variability in penalties and sentencing. Even when we have succeeded in establishing laws consistent with certain maxims (e.g. against murder), cases which offend these most basic maxims can involve lengthy debates about appropriate actions and mitigating circumstances.

For example, the interplay between reality and maxims is underscored by contemporary debates on euthanasia, where the social value of human life is weighed against the right of the individual to determine his or her own fate. There is no simple answer to this question, and to even begin to view this as a methodological or calculable issue among humans, let alone how to program such issues into AI language, engenders debate on the foundations of how to tackle such issues.

Further, it is unclear how humans can “teach” AI morality when we have failed to address glaring holes in our own extant theories of morality. For instance, regarding utilitarianism, the challenge of dealing with imprecise measurables in calculating the common good remains a challenge – not only for AI but for humans. As critiqued by political philosopher John Rawls, utilitarianism conflates “all systems of desires” into a singular conception of desirable social outcomes.  Thus, the separateness of persons is sacrificed in favor of the utilitarian conceit that there is an impartial truth that maximizes human happiness.

This is problematic because ascertaining how to achieve the greatest net balance of satisfaction is not only difficult but can also obscure questions regarding the “source or quality” of such impetuses. After all, racism may make the majority happier overall but deprive a minority of their safety and liberty. Rawls notes that such a trade-off satisfies the central tenets of utilitarianism but still represents a lack of justice.24 Of course, this begs the question of what justice entails and who gets to define the concept. As one can imagine, this is a difficult conceptual conversation for humans – let alone for a programmer to code for the benefit of AI systems.

It is clear then that morality plays a key role in defining “appropriate” decisions among humans. As moral agents, we are not only called upon to make decisions but also pressured to be able to justify actions to both ourselves and others.Whether the influence is philosophical, religious, spiritual, or simply “intuitive,” human decision-making is assumed to be superior, paradoxically, because of our innate desire and ability to measure and compare incalculable moral considerations.

Can AI Make “Good” Decisions?

Morality may lie at the center of human decision-making but is it a necessity for AI such as Watson, particularly as it relates to current usage in medicine? While the answer to that question may become complex as AI capabilities evolve, with respect to contemporary technological usage and implications, the answer is no.

There are many prerequisite steps before humans can apply moral principles to a decision and not all decisions made in our day-to-day lives require the same level of complexity. For instance, the decision to drink water when thirsty is much more elemental than whether to prolong the lives of terminally ill patients in pain.

This reflects the fact that there is a hierarchy to thought and learning. The work of Benjamin Bloom and his colleagues in 1956 outlined learning theory that classified functional learning into three domains – knowledge (thinking) skills, psychomotor (motor) skills, and affective (behavioral) skills – with graduated categories of learning from the simplest (concrete) to most complex (abstract) in each domain.

Current AI technologies such as Watson exist to mimic human behavior in the knowledge, also referred to as cognitive, domain. For Watson to reach par with the decision-making capability of adult humans, it stands to reason that it would be able to learn in a manner consistent with Bloom’s knowledge taxonomy. To learn on par with humans, AI would have to demonstrate achievement of each of the following steps:

(1)  Knowledge: The ability to recall.

(2)  Comprehension: The ability to understand and interpret a surface-level issue.

(3)  Application: The ability to apply abstractions, general principles and methods to concrete problems.

Bloom’s taxonomy is a practical one and still widely used in the educational realm. Its appeal in this context derives from the fact that it separates out the ability to gauge mastery over increasingly complex tasks and gives us a clear way to judge AI capabilities in a functional and outcome-driven, rather than theoretical, manner.

According to IBM, Watson mimics human problem-solving ability by respecting the following steps28:

Watson does this by employing Deep QA software for the purposes of:

(1)  storing and updating a “corpus of knowledge” in the form of medical texts,

(2)  curating the content (with human intervention) by culling through relevant medical texts and discarding those which are not relevant,

(3)  ingesting the content by creating indices and other metadata that make working with the data more efficient (including creating knowledge graphs),

(4)  learning from human experts who facilitate machine learning by uploading question/answer pairs for the purposes of exposing Watson to linguistic patterns,

(5)  continuing the learning process indefinitely with periodic review by human experts,

(6)  identifying parts of speech in a question or inquiry,

(7)  generating hypotheses,

(8)  searching for evidence to support or refute the hypotheses,

(9)  scoring each hypothesis based on statistical modeling for each piece of weighted evidence, and

(10) performing evidence scoring and rating.

In applying Bloom’s taxonomy to Watson, we see that it excels in certain domains – to the point of far exceeding human capacity – but lags in others. Indeed, it has been argued that current AI machines “have very limited learning abilities” equivalent to only the third level of Bloom’s knowledge taxonomy.

With respect to Watson, this argument appears correct. Regarding the “knowledge” level of Bloom’s taxonomy, we have noted that Watson can store and recall information from a staggering number of medical texts. With regard to “comprehension” and “application”, Watson’s ability to understand inquiries as well as search through relevant data in order to organize information necessary to present logical treatment options is, at this point, indisputable given high concordance rates.

But in marketing Watson, IBM overhypes its capabilities. Unlike humans, Watson still does not and likely cannot perform a key function – to decide on conclusions or courses of action. While Watson excels at understanding human questions, the question and answer-based model of Watson’s programming means that it cannot independently “analyze,” “synthesize,” or “evaluate” medical issues, or independently conceive of and search for new and relevant information without human stewardship or guidance.

For true analysis, synthesis, or evaluation to take place, there must be the ability to choose between a range of competing options. While one may argue that Watson can do that to some extent, in that answers to hypotheses can be scored and ranked, Watson still cannot do anything with this information. Watson’s role is as a passive assistant that can only interpret and answer external inputs.  It does not have its own capacity or desire to formulate the question itself or to independently test and definitively prove or invalidate hypotheses.