Tag: Metrics

We Are Not A Dashboard: Contesting The Tyranny Of Metrics, Measurement, And Managerialism

May 13, 2019• 2

By DAVID SHAYWITZ

The dashboard is the potent symbol of our age. It offers the elegant visualization of data, and is intended to capture and represent the performance of a system, revealing at a glance current status, and pointing out potential emerging concerns. Dashboards are a prominent feature of most every “big data” project I can think of, offered by every vendor, and constructed to provide a powerful sense of control to the viewer. It seemed fitting that Novartis CEO Dr. Vas Narasimhan, a former McKinsey consultant, would build (then tweet enthusiastically about) “our new ‘control tower’” – essentially a multi-screen super dashboard – “to track, analyse and predict the status of all our clinical studies. 500+ active trials, 70+ countries, 80 000+ patients – transformative for how we develop medicines.” Dashboards are the physical manifestation of the ideology of big data, the idea that if you can measure it you can manage it.

I am increasingly concerned, however, that the ideology of big data has taken on a life of it’s own, assuming a sense of both inevitability and self-justification. From measurement in service of people, we increasingly seem to be measuring in service of data, setting up systems and organizations where constant measurement often appears to be an end in itself.

My worries, it turns out, are hardly original. I’ve been delighted to discover over the past year what feels like an underground movement of dissidents who question the direction we seem to be heading, and who’ve thoughtfully discussed many of the issues that I stumbled upon. (Special hat-tip to “The Accad & Koka Report” podcast, an independent and original voice in the healthcare podcast universe, for introducing me to several of these thinkers, including Jerry Muller and Gary Klein.)

Obsessive Measurement Disorder: Etiology of an Epidemic

Feb 13, 2019• 6

By KIP SULLIVAN JD

Review of The Tyranny of Metrics by Jerry Z. Muller, Princeton University Press, 2018

In the introduction to The Tyranny of Metrics, Jerry Muller urges readers to type “metrics” into Google’s Ngram, a program that searches through books and other material published over the last five centuries. He tells us we will find that the use of “metrics” soared after approximately 1985. I followed his instructions and confirmed his conclusion (see graph below). We see the same pattern for two other buzzwords that activate Muller’s BS antennae – “benchmarks,” and “performance indicators.” [1]

Muller’s purpose in asking us to perform this little exercise is to set the stage for his sweeping review of the history of “metric fixation,” which he defines as an irresistible “aspiration to replace judgment based on personal experience with standardized measurement.” (p. 6) His book takes a long view – he takes us back to the turn of the last century – and a wide view – he examines the destructive impact of the measurement craze on the medical profession, schools and colleges, police departments, the armed forces, banks, businesses, charities, and foreign aid offices.

Foreign aid? Yes, even that profession. According to a long-time expert in that field, employees of government foreign aid agencies have “become infected with a very bad case of Obsessive Measurement Disorder, an intellectual dysfunction rooted in the notion that counting everything in government programs will produce better policy choices and improved management.” (p. 155)

Muller, a professor of history at the Catholic University of America in Washington, DC, makes it clear at the outset that measurement itself is not the problem. Measurement is helpful in developing hypotheses for further investigation, and it is essential in improving anything that is complex or requires discipline. The object of Muller’s criticism is the rampant use of crude measures of efficiency (cost and quality) to dish out rewards and punishment – bonuses and financial penalties, promotion or demotion, or greater or less market share. Measurement can be crude because it fails to adjust scores for factors outside the subject’s control, and because it measures only actions that are relatively easy to measure and ignores valuable but less visible behaviors (such as creative thinking and mentoring). The use of inaccurate measurement is not just a waste of money; it invites undesirable behavior in both the measurers and the “measurees.” The measurers receive misleading information and therefore make less effective decisions (for example, “body count” totals tell them the war in Viet Nam is going well), and the subjects of measurement game the measurements (teachers “teach to the test” and surgeons refuse to perform surgery on sicker patients who would have benefited from surgery).

What puzzles Muller, and what motivated him to write this book, is why faith in the inappropriate use of measurement persists in the face of overwhelming evidence that it doesn’t work and has toxic consequences to boot. This mulish persistence in promoting measurement that doesn’t work and often causes harm (including driving good teachers and doctors out of their professions) justifies Muller’s harsh characterization of measurement mavens with phrases like “obsession,” “fixation,” and “cult.” “[A]lthough there is a large body of scholarship in the fields of psychology and economics that call into question the premises and effectiveness of pay for measured performance, that literature seems to have done little to halt the spread of metric fixation,” he writes. “That is why I wrote this book.” (p. 13)

Building Better Metrics: Immunizations and Asking the Right Question(s)

Oct 5, 2017• 14

By NIRAN AL-AGBA, MD

As policy experts cling to pay-for-performance (P4P) as an indicator of healthcare quality and shy away from fee-for-service, childhood immunization rates are being utilized as a benchmark. At first glance, vaccinating children on time seems like a reasonable method to gauge how well a primary care physician does their job. Unfortunately, the parental vaccine hesitancy trend is gaining in popularity. Studies have shown when pediatricians are specifically trained to counsel parents on the value of immunizations, hesitancy does not change statistically.

Washington State Law allows vaccine exemptions on the basis of religious, philosophical, or personal reasons; therefore, immunizations rates are considerably lower (85%) compared to states where exemptions rules are tighter. Immunization rates are directly proportional to the narrow scope of state vaccine exemptions laws. Immunization rates are used to rate the primary care physician despite the fact we have little influence on the outcome according to scientific studies. Physicians practicing in states with a broad vaccine exemption laws is left with two choices: refuse to see children who are not immunized in accordance with the CDC recommendations or accept low quality ratings when caring for children whose parents with beliefs that may differ from our own.

Building Better Metrics: Patient-Driven Metrics

Oct 5, 2016• 7

By NIRAN AL-AGBA, MD

Recently I wrote about empowerment and the importance of letting patients make their own health care decisions. Our job is to make sure patients are given information and then allowed to choose the best option for them. Maybe we should even embolden patients; give them confidence and encourage them to take more control. Physicians tend to feel more comfortable advising according to the “standard of care” and we struggle handing over the reins when we believe we “know” the safest path to take.

Every time I talk about building better metrics, I emphasize the significance of evaluating something physicians can change or control. The intent behind measuring patient satisfaction was likely to increase patient autonomy, however, as with many things; the devil was in the details. It turns out chasing higher patient satisfaction scores can result in higher costs and increased mortality. Overall, the most satisfied patients were more likely to be admitted to the hospital and total health-care costs were 9% higher. Most strikingly, for every 100 people who died over a four year period in the least satisfied group, 126 people died in the most satisfied group. At least they died happy and satisfied right? That notion can be difficult for some physicians to accept but might be more important than we realize.