By David Shaywitz, MD, PhD

The idea that physicians are going to be far less important in the medicine of the future seems to be a central assumption of many next-generation health companies, an assertion that, like undergraduate Shakespeare productions set in the present day, may once have felt daring and original, but now seems merely tedious.

The logic goes something like is: Patients are accustomed to seeking insight from their doctors but doctors are far less good at providing this advice than most patients realize. As more consumer-based tools for managing health become available, patients will recognize that they now have the means and the motivation to take care of their health better than their physicians, and medical care will move directly into their hands.

Arguably, a form of this already happens today, as patients make extensive use of non-prescription products (e.g. the Vitamin Shoppe reported net sales of $750 million in fiscal year 2010), non-traditional practitioners (e.g. total revenue received by chiropractors is estimated by Hoovers to be about $10 billion), and seek medical advice from friends on Facebook (which may have directly saved at least one life).

What these data don’t convey, however, is something I’ve had the privilege to experience first-hand: Doctors enjoy an exceptionally durable bond with patients — especially those patients with chronic illnesses. The level of trust reported by patients for their physicians is remarkable, and the role of physician as trusted adviser is difficult to overstate. It’s a huge burden to manage disease on one’s own, and it’s generally reassuring to know your physician is with you at every step — something I believe still happens, by the way, although obviously not in every case.

The problem is, while patients may feel better by believing their physicians are delivering excellent care, this confidence may not always be warranted: The reality is that the care provided by doctors is uneven and inconsistent. While the importance of “discrepancies from standard of care” (a common performance metric) has probably been overstated (the evidence base for many guidelines can be pretty thin), it’s also likely that a lot of patients would enjoy improved health if their doctors were simply better; look no further than the difficult time Peter Pronovost and colleagues have had getting physicians to do things as basic as washing hands and following simple checklists (see Better, by Atul Gawande, as well). And every medical resident knows about “gome docs,” physicians who seem to provide unusually poor care, and yet are beloved by their patients. Some physicians are clearly much better than others, a phenomenon discussed with characteristic eloquence by Gawande in this wonderful essay focused on the care of children with cystic fibrosis.

So perhaps there are really two questions here: First, will doctors become obsolete? I doubt it, and suspect that reports of their demise have been greatly exaggerated. Second — and arguably more interestingly: Should (most) doctors become obsolete — or less provocatively, does the practice of medicine need to change? Here, the answer must be yes. We urgently need to track and review outcomes that can already be measured, and we must dramatically improve our ability to measure patient health and real-world effectiveness, so that physicians can get better, and patient care can improve.

This is a distinctly non-trivial undertaking. Medicine is a profoundly conservative discipline. Change comes slowly, and with great resistance. There’s also a tremendous stake in maintaining a form of the status quo; I’ve also heard my share of stories about the risks of keeping score — I’ve heard of hospitals trying to maintain their favorable mortality statistics by transferring sick patients before they die, and I know of the pressures on physicians to give patients the suboptimal medicines they demand or risk a scathing patient review; and I appreciate as much as anyone how metrics can deceive, how algorithms can stultify (see also here and here), and I know how difficult it can be to capture some of the most vitally important aspects of patient care.

I am reminded of the famous quotation by management guru W. Edwards Deming — a quote often rendered, incorrectly, as “you can’t manage what you don’t measure.” In fact, he wrote, “the most important figures that one needs for management are unknown or unknowable, but successful management must nevertheless take account of them.”

Similarly, the challenge for medicine is not only to utilize patient-focused technologies to measure more precisely and more meaningfully the quality of care our patients receive, and the quality of life they are able to enjoy, but also to ensure that we recognize the inevitable limitations of measurement, and the unquantifiable benefit that can come from a reassuring nod, from a hand on the shoulder, from an empathetic doctor. “The good physician treats the disease,” Sir William Osler said. “The great physician treats the patient who has the disease.”

My hope is that, informed by metrics but driven by heart, the doctor of the future will be uniquely empowered to treat both.

David Shaywitz is co-founder of the Harvard PASTEUR program, a research initiative at Harvard Medical School. His a strategist at a biopharmaceutical company in South San Francisco. You can follow him at his personal website. This post originally appeared in the Atlantic.