By KIP SULLIVAN JD
Review of The Tyranny of Metrics by Jerry Z. Muller, Princeton University Press, 2018
In the introduction to The Tyranny of Metrics, Jerry Muller urges readers to type “metrics” into Google’s Ngram, a program that searches through books and other material published over the last five centuries. He tells us we will find that the use of “metrics” soared after approximately 1985. I followed his instructions and confirmed his conclusion (see graph below). We see the same pattern for two other buzzwords that activate Muller’s BS antennae – “benchmarks,” and “performance indicators.” [1]
Muller’s purpose in asking us to perform this little exercise is to set the stage for his sweeping review of the history of “metric fixation,” which he defines as an irresistible “aspiration to replace judgment based on personal experience with standardized measurement.” (p. 6) His book takes a long view – he takes us back to the turn of the last century – and a wide view – he examines the destructive impact of the measurement craze on the medical profession, schools and colleges, police departments, the armed forces, banks, businesses, charities, and foreign aid offices.
Foreign aid? Yes, even that profession. According to a long-time expert in that field, employees of government foreign aid agencies have “become infected with a very bad case of Obsessive Measurement Disorder, an intellectual dysfunction rooted in the notion that counting everything in government programs will produce better policy choices and improved management.” (p. 155)
Muller, a professor of history at the Catholic University of America in Washington, DC, makes it clear at the outset that measurement itself is not the problem. Measurement is helpful in developing hypotheses for further investigation, and it is essential in improving anything that is complex or requires discipline. The object of Muller’s criticism is the rampant use of crude measures of efficiency (cost and quality) to dish out rewards and punishment – bonuses and financial penalties, promotion or demotion, or greater or less market share. Measurement can be crude because it fails to adjust scores for factors outside the subject’s control, and because it measures only actions that are relatively easy to measure and ignores valuable but less visible behaviors (such as creative thinking and mentoring). The use of inaccurate measurement is not just a waste of money; it invites undesirable behavior in both the measurers and the “measurees.” The measurers receive misleading information and therefore make less effective decisions (for example, “body count” totals tell them the war in Viet Nam is going well), and the subjects of measurement game the measurements (teachers “teach to the test” and surgeons refuse to perform surgery on sicker patients who would have benefited from surgery).
What puzzles Muller, and what motivated him to write this book, is why faith in the inappropriate use of measurement persists in the face of overwhelming evidence that it doesn’t work and has toxic consequences to boot. This mulish persistence in promoting measurement that doesn’t work and often causes harm (including driving good teachers and doctors out of their professions) justifies Muller’s harsh characterization of measurement mavens with phrases like “obsession,” “fixation,” and “cult.” “[A]lthough there is a large body of scholarship in the fields of psychology and economics that call into question the premises and effectiveness of pay for measured performance, that literature seems to have done little to halt the spread of metric fixation,” he writes. “That is why I wrote this book.” (p. 13)
Continue reading…
