My daughter is an accountant. She took a statistics class in high school, and another as a requirement for her major. My son has taken a statistics course, and he is an English Literature major. I was a chemistry major in college and have an an MD and have never taken a statistics course. I don’t even recall a lecture on statistics in medical school. Mark Twain quoted Disraili as saying, “There are three kinds of lies: lies, damned lies and statistics.” Reading medical journal articles reporting on the benefits and lack of benefits when reported statistically can be really challenging. Reading a report of these, or worse listening to an interested party, like a sales rep or sponsored speaker talk about a study, requires being a skeptic. Here are some examples of how true statistics can be worse than a lie, and how what would seem to be common sense does not pay off.
Relative risk reduction vs. absolute risk reduction : The benefits of an intervention, say a medication or a procedure is usually reported as the desired outcome seen with the intervention as compared to a baseline seen with placebo or a control group. It can be reported as relative risk reduction, or absolute risk reduction. Relative risk reduction can seem very impressive, but it is key to look at absolute risk reduction also. An example of this is the reported benefit of reduction of risk of hip fractures with a bisphosphonate called Fosamax. A report in JAMA showed a reduction in hip fractures over 4 years of use in women with osteoporosis from 2.2% to 1.0%. This is an absolute reduction of 1.2% over 4 years, or 0.3% annually. This is reported by the makers of Fosamax accurately as a 56% reduction in risk, which is true but misleading. A more helpful way to look at this is the NNT (Number needed to treat, defined in statistics using the formula 100/%reduction). In this case 100/1.2% =88. So to prevent 1 hip fracture it would be necessary to treat 88 women for 4 years. Sounds less impressive than a 56% reduction to me.
Graphs can also exaggerate benefits or risks: Many people are visual thinkers, People who make graphs manipulate them to prove their point. By changing the scale on the X or Y axis of a graph, the extent of a trend can be visually exaggerated or minimized. A good article discussing this is Does Graph Design Matter To CPAs And Financial Statement Readers? I couldn’t find a good reference to this discussing medicine. Apparently misleading readers about their money is more newsworthy than misleading them about their health.
Extrapolation of Data: Another problem with using evidence from studies is the lure of using proven benefits in one circumstance and assuming thte data must also apply to a somewhat different circumstance. An example: Since treating a person who has had a stroke with an aspirn daily has been proven to reduce the risk of another stroke significantly, then using an aspirin before having a stroke must be even better, prevent the first stroke. As logical as this seems, data does not support this corollary to the proven theorem. See my prior post Aspirin – Should you take one a day? to see a discussion of primary vs. secondary prevention, and the real and significant differences in where the data leads you.
For a nice series of articles using similar discussions of statistical lies and aberrations see Bittersweet Medicine by Dr Lemmon (love the blog site name given the author’s name) and his series on the most overrated treatments.
Maybe medical schools should require pre-med students to take a course in statistics and require one less course in the physical sciences.
Ed Pullen, MD, is a board certified family physician practicing in Puyallup, WA. Dr. Pullen shares his viewpoints on medical news and policy from a primary care physician’s perspective at his blog,DrPullen.com.
Categories: Uncategorized
I never doubted that the Government might stoop to slanting statistics in a manner to support their agendas, but in the medical field? This is not tolerable, those that we need to trust in most, those medical professionals that hold are very lives in their hands. We seem to have more integrity in medical supply sales than they do in applying and administering such devices.
Stats, just like words (and images and all other forms of media for that matter), is a key tool of persuasion. Both can be taken out of context and can be spun to whatever angle needed. At the same time, both can be used to reveal a more realistic view of the world. Aren’t the most successful people in the world essentially great storytellers who use these tools well?
What I’ve also found is that especially in healthcare, we have either too much or too little data. It’s either overly-complicated or over-simplified. This only adds to the problem of “statistics” that Dr. P talks about above. One solution is developing better 2.0-type tools that level the playing field – allowing all people to have access to the data. May the best data slicer/dicer win.
jme made an excellent observation, which I would rephrase to be more useful to those planning medical education (or their continuing medical education): clinicians need to be knowledgable about data and decision making. This includes, of course, statistics (which can help us understand the data). But it also includes where to find data, how to evaluate data, how (and when) to manipulate data, how to describe data (verbally and graphically). Some of these have statistical pieces, but defining the issue as ‘clinicians don’t understand statistics’ is tremendously limiting, perhaps akin to saying that photographers need to understand contrast.
In addition to the math and data skills partially listed above, one needs to understand human decision making, which is not built for rational and dispassionate assessment and often long term or delayed action in complex and uncertain settings. Read Kahnemann and Tversky, for example (Judgement Under Uncertainty). More accessible and a great read for clinicians is ‘Inevitable Illusions’ by Piatelli-Palmarini.
What I find most intriguing about this post (I am actually a statistician) is that none of the examples given demonstrate an understanding of what the field of statistics actually is.
Now, these three examples are certainly demonstrating important aspects of using and interpreting data. And I agree with each of them.
I’d summarize these three skills as (1) Choosing the correct quantity to summarize data, (2) Accurately and fairly displaying data visually and (3) Using care to avoid inappropriate generalizations.
IMHO, if you asked actual statisticians to describe what statistics is all about, none of these three things would be at the top of the list. That’s not to say they aren’t important aspects of data analysis, but they just don’t represent “statistics”.
The fundamental task in statistics is measuring and quantifying variability in data. So it was telling, to me, that Dr. Cullen’s first example treating estimates of a 56% reduction in risk and a NNT of 88 as simply true, because that’s what the summary of the paper claimed.
The important statistics piece here is that all of these estimates have errors associated with them. Studies that estimate 56% reduction in risk will (or at least ought to) report a confidence interval as well.
Using statistics correctly isn’t simply picking the correct number to use. That’s just basic numeracy. I’ll consider physicians adept with statistics when they are able to use not just the correct numerical quantity but also the associated error estimate to help guide their medical decision making.
More examples can be found in the blog post of Jean-Luc Doumont a professional science communication trainer. The blog post is called: Lies, damned lies, and visual lies
http://www.treesmapsandtheorems.com/blog/X0028.php
And even scientists for oversimplifying problems:
“Unless we concentrate very hard, we often simplify a problem, because our minds routinely do so without knowing it.
Many people confuse the statement that “almost all drugs are small molecules” with “almost all small molecules are drugs”. Assume that the first statement is true, that 96 percent of drugs are small molecules (not counting biotech, or nutraceutical drugs). This would mean that only about 0.01 percent of small molecules are drugs, since there are more than 24 million physically existing small molecules and only 4600 drugs, roughly one in five thousand. So the logical mistake makes you (unconsciously) overestimate the odds of a randomly drawn small molecule being a drug by more than five thousand times !” http://miningdrugs.blogspot.com/2009/03/open-innovation-in-drug-design-i-do-not.html
HOW THE HARVARD LAW WAS CAUGHT IN A LIE
Can statistics lie? Is OWE-bama BANKRUPTING the USA?
http://nyti.ms/ax4BW4
How many personal bankruptcies might be avoided is unpredictable, as it is not clear how often medical debt plays a back-breaking role. There were 1.1 million personal bankruptcy filings in 2008, including 12,500 in Nashville, and more are expected this year.
Last summer, Harvard researchers published a headline-grabbing paper that concluded that illness or medical bills contributed to 62 percent of bankruptcies in 2007, up from about half in 2001. More than three-fourths of those with medical debt had health insurance.
But the researchers’ methodology has been criticized as defining medical bankruptcy too broadly and for the ideological leanings of its authors, some of whom are outspoken advocates for nationalized health care.
Nov. 2, this HARVARD LAW LIE gets CORRECTED. The hard way — by unemploying the DEADWOOD (D).
Thanks for the above comments. Not hearing the drug reps at all is the best option, but whereever we turn we hear speakers quote stats to make their points, so it’s not just hired PR guys talking to us. MD as Hell, I want a doctor who spends my money wisely, and cares. Let’s me be informed as to likelihoods, etc. Not just one who doesn’t care about $.
Another post where there will be chest thumping and a lot of lather, but who cares?
If there is a 99% chance you don’t have a condition, but if you do, you will die if we don’t find it, who do you want deciding what to do, some bean counter or the doctor looking at you live and in the flesh?
“You can’t have the cardiac cath, Mrs. Smith, because you don’t risk stratify.” That is the problem with someone else holding your money. If you hold the money, then you can decide what risks are worth it to you and what risks are not worth saving the money.
In the exam room care is not about statistics. Either the patient has it or they don’t.
The ultamate healthcare freedom is from keeping the money at home for your own use.
When pharmacy reps bring in their studies I ask how many patients were in the study and who funded it. I don’t go by what the reps say, I go by what I see in my patients. That’s the most important factor for me. I did take a statistics class in my master’s degree for family nurse practitioner and it has helped me to understand that some studies are not what they seem on the surface.
Yes and no. It certainly would be desirable if every physician could correctly interpret statistics, and I agree that medical schools and residency programs should try harder to at least cover basics and to promote understanding of the basics of the above tricks.
But is it realistic to have every MD understand stats well, and would it solve many of the connected health care problems? I doubt it.
When pharm reps can make improper claims in docs offices, it’s not just a failure of stats knowledge, but rather:
-failure of peers who propagate those or similar claims
-lack of interest in practising cost efficient medicine
-the system that allows these reps to be active (and to pay speakers, invite to dinners etc.)
You don’t need to be a stats expert to be skeptical of pharm reps claims, to follow evidence based guidelines and to consult evidence based databases (which do the stats for you). But unfortunately, this is not high on the priority list for a lot of physicians.
Anon, I think you are right … the whole medical education has to focus more on what matters. Residents get praised to consider the most remote, untreatable differential diagnoses in an elderly patient, while they are allowed to keep the same patient on superfluous polypharmacotherapy, making the patient drowsy, fall and brake his/her hip.
Good post,and right on target with the lack of medical education in this area. It is crucial for the ability to critically evaluate peer reviewed articles,for instance; to say nothing of fending off pharma detail people.
Only trouble is,I did take statistics in college, got an A in it,and it didn’t help that much. I think a course specifically for medical students, as you recommend, is best.
Hi,
I’m a first-year medical student with a mixed background, including some studies in mathematics and biostatistics that were undertaken purely out of personal interest. I could not agree more with what’s being said here. I would also like to add that I personally feel that the required statistical baggage for boards is not only insufficient, but moreover may give some students the impression that they know enough statistics to correctly understand research.
I agree that physicians need basic training in statistics and how to interpret them (the AMA Manual of Style provides useful background on med/sci studies and statistics, also covered in a number of books). Journalists also need better training to avoid describing results in misleading terms to sound “newsier,” or amplifying misinterpretations placed in front of them. On media coverage of healthcare stats, I recommend Gary Schwitzer’s HealthNewsReview blog:
http://www.healthnewsreview.org/blog/
It’s unconscionable that we force them to waste their time memorizing all twelve cranial nerves, but never teach them basic statistics or things like process flow analysis. And don’t get me started on how insane it is that an information industry not only doesn’t use IT, but takes young people who “get” IT and tells them to put all that useful knowledge aside.
We’re still acting like the central challenge facing our doctors is leading the charge in the application of the scientific method in a world that still uses witchcraft.
Doctors have a totally new set of challenges, because the practice of medicine has advanced. By failing to teach them statistics, organizational management, and the new pitfalls created by IT management, and instead focusing all out effort on the science they’re ALREADY pretty strong on, we’re doing a terrible disservice to our health care system.