OP-ED

Statistics – Using the Truth to Mislead

My daughter is an accountant.  She took a statistics class in high school, and another as a requirement for her major.  My son has taken a statistics course, and he is an English Literature major.  I was a chemistry major in college and have an an MD and have never taken a statistics course. I don’t even recall a lecture on statistics in medical school.  Mark Twain quoted Disraili as saying, “There are three kinds of lies: lies, damned lies and statistics.”  Reading medical journal articles reporting on the benefits and lack of benefits when reported statistically can be really challenging. Reading a report of these, or worse listening to an interested party, like a sales rep or sponsored speaker talk about a study, requires being a skeptic.  Here are some examples of how true statistics can be worse than a lie, and how what would seem to be common sense does not pay off.

Relative risk reduction vs. absolute risk reduction : The benefits of an intervention, say a medication or a procedure is usually reported as the desired outcome seen with the intervention as compared to a baseline seen with placebo or a control group. It can be reported as relative risk reduction, or absolute risk reduction.  Relative risk reduction can seem very impressive, but it is key to look at absolute risk reduction also.  An example of this is the reported benefit of reduction of risk of hip fractures with a bisphosphonate called Fosamax.  A report in JAMA showed a reduction in hip fractures over 4 years of use in women with osteoporosis from 2.2% to 1.0%.  This is an absolute reduction of 1.2% over 4 years, or 0.3% annually.  This is reported by the makers of Fosamax accurately as a 56% reduction in risk, which is  true but misleading.  A more helpful way to look at this is the NNT (Number needed to treat, defined  in statistics using the formula  100/%reduction).  In this case 100/1.2% =88. So to prevent 1 hip fracture it would be necessary to treat 88 women for 4 years.  Sounds less impressive than a 56% reduction to me.

Graphs can also exaggerate benefits or risks:  Many people are visual thinkers, People who make graphs manipulate them to prove their point.   By changing the scale on the X or Y axis of a graph, the extent of a trend can be visually exaggerated or minimized.  A good article discussing this is  Does Graph Design Matter To CPAs And Financial Statement Readers? I couldn’t  find a good reference to this discussing medicine.  Apparently misleading readers about their money is more newsworthy than misleading them about their health.

Extrapolation of Data: Another problem with using evidence from studies is the lure of using proven benefits in one circumstance and assuming thte data must also apply to a somewhat different circumstance.  An example:  Since treating a person who has had a stroke with an aspirn daily has been proven to reduce the risk of another stroke significantly, then using an aspirin before having a stroke must be even better, prevent the first stroke.  As logical as this seems, data does not support this corollary to the proven theorem.   See my prior post  Aspirin – Should you take one a day? to see a discussion of primary vs. secondary prevention, and the real and significant differences in where the data leads you.

For a nice series of articles using similar discussions of statistical lies and aberrations see Bittersweet Medicine by Dr Lemmon (love the blog site name given the author’s name) and his series on the most overrated treatments.

Maybe medical schools should require pre-med students to  take a course in statistics and require one less course in the physical sciences.

Ed Pullen, MD, is a board certified family physician practicing in Puyallup, WA. Dr. Pullen shares his viewpoints on medical news and policy from a primary care physician’s perspective at his blog,DrPullen.com.

Livongo’s Post Ad Banner 728*90

Categories: OP-ED

Tagged as: ,

14
Leave a Reply

14 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
12 Comment authors
SteveTony at FDAzilla.compheskijmeJoerg Kurt Wegner Recent comment authors
newest oldest most voted
Steve
Guest

I never doubted that the Government might stoop to slanting statistics in a manner to support their agendas, but in the medical field? This is not tolerable, those that we need to trust in most, those medical professionals that hold are very lives in their hands. We seem to have more integrity in medical supply sales than they do in applying and administering such devices.

Tony at FDAzilla.com
Guest

Stats, just like words (and images and all other forms of media for that matter), is a key tool of persuasion. Both can be taken out of context and can be spun to whatever angle needed. At the same time, both can be used to reveal a more realistic view of the world. Aren’t the most successful people in the world essentially great storytellers who use these tools well? What I’ve also found is that especially in healthcare, we have either too much or too little data. It’s either overly-complicated or over-simplified. This only adds to the problem of “statistics”… Read more »

pheski
Guest
pheski

jme made an excellent observation, which I would rephrase to be more useful to those planning medical education (or their continuing medical education): clinicians need to be knowledgable about data and decision making. This includes, of course, statistics (which can help us understand the data). But it also includes where to find data, how to evaluate data, how (and when) to manipulate data, how to describe data (verbally and graphically). Some of these have statistical pieces, but defining the issue as ‘clinicians don’t understand statistics’ is tremendously limiting, perhaps akin to saying that photographers need to understand contrast. In addition… Read more »

jme
Guest
jme

What I find most intriguing about this post (I am actually a statistician) is that none of the examples given demonstrate an understanding of what the field of statistics actually is. Now, these three examples are certainly demonstrating important aspects of using and interpreting data. And I agree with each of them. I’d summarize these three skills as (1) Choosing the correct quantity to summarize data, (2) Accurately and fairly displaying data visually and (3) Using care to avoid inappropriate generalizations. IMHO, if you asked actual statisticians to describe what statistics is all about, none of these three things would… Read more »

Joerg Kurt Wegner
Guest

More examples can be found in the blog post of Jean-Luc Doumont a professional science communication trainer. The blog post is called: Lies, damned lies, and visual lies http://www.treesmapsandtheorems.com/blog/X0028.php And even scientists for oversimplifying problems: “Unless we concentrate very hard, we often simplify a problem, because our minds routinely do so without knowing it. Many people confuse the statement that “almost all drugs are small molecules” with “almost all small molecules are drugs”. Assume that the first statement is true, that 96 percent of drugs are small molecules (not counting biotech, or nutraceutical drugs). This would mean that only about… Read more »

Karl
Guest

HOW THE HARVARD LAW WAS CAUGHT IN A LIE Can statistics lie? Is OWE-bama BANKRUPTING the USA? http://nyti.ms/ax4BW4 How many personal bankruptcies might be avoided is unpredictable, as it is not clear how often medical debt plays a back-breaking role. There were 1.1 million personal bankruptcy filings in 2008, including 12,500 in Nashville, and more are expected this year. Last summer, Harvard researchers published a headline-grabbing paper that concluded that illness or medical bills contributed to 62 percent of bankruptcies in 2007, up from about half in 2001. More than three-fourths of those with medical debt had health insurance. But… Read more »

Dr. Pullen
Guest

Thanks for the above comments. Not hearing the drug reps at all is the best option, but whereever we turn we hear speakers quote stats to make their points, so it’s not just hired PR guys talking to us. MD as Hell, I want a doctor who spends my money wisely, and cares. Let’s me be informed as to likelihoods, etc. Not just one who doesn’t care about $.

MD as HELL
Guest
MD as HELL

Another post where there will be chest thumping and a lot of lather, but who cares? If there is a 99% chance you don’t have a condition, but if you do, you will die if we don’t find it, who do you want deciding what to do, some bean counter or the doctor looking at you live and in the flesh? “You can’t have the cardiac cath, Mrs. Smith, because you don’t risk stratify.” That is the problem with someone else holding your money. If you hold the money, then you can decide what risks are worth it to you… Read more »

Jennifer Scott ARNP, FNP-BC
Guest

When pharmacy reps bring in their studies I ask how many patients were in the study and who funded it. I don’t go by what the reps say, I go by what I see in my patients. That’s the most important factor for me. I did take a statistics class in my master’s degree for family nurse practitioner and it has helped me to understand that some studies are not what they seem on the surface.

rbar
Guest
rbar

Yes and no. It certainly would be desirable if every physician could correctly interpret statistics, and I agree that medical schools and residency programs should try harder to at least cover basics and to promote understanding of the basics of the above tricks. But is it realistic to have every MD understand stats well, and would it solve many of the connected health care problems? I doubt it. When pharm reps can make improper claims in docs offices, it’s not just a failure of stats knowledge, but rather: -failure of peers who propagate those or similar claims -lack of interest… Read more »

bev M.D.
Guest
bev M.D.

Good post,and right on target with the lack of medical education in this area. It is crucial for the ability to critically evaluate peer reviewed articles,for instance; to say nothing of fending off pharma detail people.
Only trouble is,I did take statistics in college, got an A in it,and it didn’t help that much. I think a course specifically for medical students, as you recommend, is best.

Hypotyposis
Guest
Hypotyposis

Hi,
I’m a first-year medical student with a mixed background, including some studies in mathematics and biostatistics that were undertaken purely out of personal interest. I could not agree more with what’s being said here. I would also like to add that I personally feel that the required statistical baggage for boards is not only insufficient, but moreover may give some students the impression that they know enough statistics to correctly understand research.

KayJay
Guest
KayJay

I agree that physicians need basic training in statistics and how to interpret them (the AMA Manual of Style provides useful background on med/sci studies and statistics, also covered in a number of books). Journalists also need better training to avoid describing results in misleading terms to sound “newsier,” or amplifying misinterpretations placed in front of them. On media coverage of healthcare stats, I recommend Gary Schwitzer’s HealthNewsReview blog:
http://www.healthnewsreview.org/blog/

anon
Guest
anon

It’s unconscionable that we force them to waste their time memorizing all twelve cranial nerves, but never teach them basic statistics or things like process flow analysis. And don’t get me started on how insane it is that an information industry not only doesn’t use IT, but takes young people who “get” IT and tells them to put all that useful knowledge aside. We’re still acting like the central challenge facing our doctors is leading the charge in the application of the scientific method in a world that still uses witchcraft. Doctors have a totally new set of challenges, because… Read more »