Torture the Data Until it Confesses

flying cadeuciiDid you ever hear the old joke where the boss says floggings will continue until morale improves? Torturing the data until results improve…or the data confesses…is not uncommon. Which is a pity.

In my career I’ve worked with companies with over 100k covered lives the claim costs of which could swing widely, from year to year, all because of a few extra transplants, big neonatal ICU cases, ventricular assist cases, etc.

Here are just a few of the huge single case claims I’ve observed in recent years:

  • $3.5M    cancer case
  • $6M       neonatal intensive care
  • $8M       hemophilia case
  • $1.4M    organ transplant
  • $1M       ventricular assist device

This is not a complaint. After all this is what health insurance should be about, huge unbudgetable health events.

All plans have one organ transplant every 10k life years or so, most of which will cost about $1M over 6 years. A plan with 1k covered lives will have such an expense on the average of every 10 years. Of course the company may have none for 15 years and two in the 16th year. The same goes for $500k+ ventricular assist device surgeries.

Looking at claims data for small groups is perilous, sometimes for large groups as well. Because of the high cost and relative infrequency of so-called “shock” claims, those over $250k, you need about 100k life years for the claims data to be even approximately 75% credible. When a group with 5k lives said they did something that cut the claims costs, they can’t really know if the change made a significant difference for a couple of decades.

Here is an example. A smallish group, about 3k covered lives asked me to help calculate how much their wellness plan was saving. They had all employees listed in three tiers: active wellness participants, moderate participants, and non-participants. I warned them they didn’t have enough data to be credible but they proceeded anyway. They expected active users would have the lowest claim costs and so on. When the data were reviewed, there was perfect reverse correlation. Active wellness users had the highest claim costs, moderate users had the next highest costs, and non-participants the lowest. In their final report, which I had nothing to do with preparing, and from which I had recused myself, they subtracted out big claims by the active and moderate users to get the results they wanted. In short they flogged the data until it confessed. Alas.

One large company claimed huge reductions in plan costs by adding a wellness program. It turns out during that period in question they also implemented an “early out” incentive. Upon examination, the early out program resulted in a big reduction in the number of older employees which more than accounted for the reduction in claims costs.

Here is yet another example. I was in a conference a few years ago in which a presenter from a small company, about 1k covered lives, claimed to have kept their health costs flat for five years through wellness initiatives. While he got a big ovation, his numbers just didn’t add up. I asked him a few questions after his speech about what other changes he made during that period. He said they lowered their “stop loss” limit from $100k to $50K a few of years earlier. Then he admitted to excluding his stop loss premium costs, which were skyrocketing, from his presentation. With a little bit of mental arithmetic I added that back in, which revealed his company’s total health costs were going up at the same rate as everyone else’s, perhaps even a little higher. Hmmm. I don’t think he deliberately mislead the audience. He just didn’t know better. When you hear boasts of big short-term impacts of wellness programs, beware of confirmation bias.

When a company claims they implemented something that caused their health plan costs to drop 15% or so, ask a few questions:

  1. The big question is did the company adjust for plan design changes, such as raising deductibles and copays, that merely shifted costs to employees?
  2. Did the changes really save claim dollars?
  3. Did they factor in stop loss premiums?
  4. How many life years of data did they observe?
  5.  Did the company exclude large or “shock” claims? (This is not uncommon, especially among wellness vendors.)
  6. Did it experience any big changes in demographics, such as through implementation of an early retirement program or layoffs that impacted older workers the worst?

When I’ve asked those kinds of questions, I’ve almost never seen a big claim of cost reductions by a small company hold up under scrutiny, and same for some big companies too.

Today torturing the data to get the desired results is all too common. That’s no surprise. They keep catching academics and big pharma doing the same thing. Skepticism is in a good thing.

Tom Emerick is the President of Emerick Consulting and co-founder of Edison Health.

Categories: Uncategorized

Tagged as:

9 replies »

  1. One needs to have successors who will be firm on applying living wills.



  2. Peter, I agree with your comment about how public discussions about end-of-life questions quickly devolve into demagoguery. Alas.

    I’ve made my living will crystal clear. Too bad but I’ve been an eye witness to providers who try to thwart living wills.I’ve even seen them be successful. That seems perfectly unethical to me.



  3. “We need to have an adult conversation in America about spending a million dollars to keep someone alive an extra 21 to 365 days.”

    Tom, I’ve already had the adult conversation with my wife and myself and secured it in a living will and health care POA. However I don’t think it’s possible to have this conversation nationally given the political polarization and the “death panel” group. We seem to prefer our politics in the extremes where middle ground is a four letter word.

    Certainly the conversation may be easier if we could also have an intelligent conversation on medical prices and cost controls.

    I also fear that the “intelligent” conversation will be forced on Medicaid patients or those with excruciating co-pays and deductibles or without insurance at all.

  4. Peter, you asked a good question. One VAD case I saw recently was $500k and the pt survived 21 days. VADs tend to extend death vs “save” lives. For a good read on this, I’d strongly recommend Nortin Hadler’s “Rethinking Aging”. We need to have an adult conversation in America about spending a million dollars to keep someone alive an extra 21 to 365 days.


    Tom Emerick

  5. “the patient survived for 24 hours”

    Is this predicable, or at least x% survived versus x% did not?

  6. Check out ECMOs, extracorporeal membrane oxygenators. These are devices used to bridge a patient into a lung transplant.

  7. Nortin, Thank you very much for your comment. Regarding the $1M ventricular assist device, the patient survived for 24 hours. In short, the cost to keep the patient alive one more day was one million dollars. What a waste. Yes, the real question is why. Alas.

  8. I have learned a great deal from you over the years about “benefits” and their provision. The observations set forth in this blog post take their place among the most telling and most valuable. Thank you.

    Let me try to return the favor. We could engender an animated discussion of the lack of transparency regarding the pricing of the “huge single case claims” you list. But no one can argue that if the costly intervention offered no benefit to the claimant, it is overpriced at any price. The next time you are asked to indemnify a “ventricular assist device”, please ask to what end?