Tech

Meet Propeller Health: Digital Health’s Poster Child for Invalid Savings Reporting

We’ve seen shorter abstracts, and we’ve seen abstracts with more curious findings, but we’ve never seen a shorter abstract with more curious findings than this one, done by Dignity Health and Dr. Rajan Merchant, and financed by the California Healthcare Foundation, evaluating a gadget made by Propeller Health.

The study group’s use of inpatient care for asthma declined by a whopping 62% vs. the control group.  You might think this result violates Dr. John Ioannidis’ well-known conclusion that large treatment effects are usually wrong, but you’d be mistaken.  You see, there was no treatment here.

There was only an effect.  Dr. Ioannidis’ result applies only to actual comparisons of effects due to different treatments, not to random changes in effects using the same treatment.  In this study, the actual treatment protocol was the same and the inhalers were the same.

The only thing different was frequency of drug use.  Whereas the conventional wisdom for disease management states that hospitalizations can be avoided by more adherence and hence more drug use, in this case the study group used less medication than the control group, reaching for their rescue inhalers 25% less– once every 6.3 days vs. every 4.7 days for the control group.

The inevitable conclusion would be that the study group needed far fewer inpatient days and considerably fewer rescue puffs because their asthma was controlled much better.  And that brings us to the most counterintuitive and impressive finding of all.

Yes, the most impressive part of the study was that this 62% reduction in inpatient days and 25% reduction in rescue drug usage was achieved even though overall asthma control improved similarly[1] in both groups. Specifically, actual patient scores on the Asthma Control Test improved only by a statistically insignificant 2% in the study groups vs. the control groups.

In other words, the Propeller Health intervention had no effect on relative asthma control scores.

Putting all this information in a table makes it clear that Dr. Ioannidis is going to have to re-examine his conclusions because this Propeller Health study shows that it is indeed possible to achieve a massive reduction in events and costs even if nothing else changes except frequency of drug use, which changes in the “wrong” direction:

Nonetheless, let’s give the Propeller Health intervention credit for that 2% improvement in control scores.  Avid readers of our wellness postings  and Why Nobody Believes the Numbers may recall that the ratio of event reduction to quality score improvement is called the “Wishful Thinking Multiplier,” and Propeller Health’s Multiplier of 31x just shattered Viverae’s record of 17x.

(The real answer, of course, is that on a good day event changes mirror quality improvement changes. That itself could be wishful thinking, and there are many reasons why even a 1x Wishful Thinking Multiplier is difficult to achieve, as described in the book.)

We’re also not sure where exactly Dr. Merchant found these asthmatics but, wow, are they ever sick.  During the baseline period they were running 326 asthma inpatient days per 1000 asthmatics.  A typical group of 1000 asthmatics will incur about a quarter that many inpatient days.

Perhaps that’s because when these folks show up in the ER they almost always get admitted: Whereas the typical asthmatic has a ratio of about 5-to-2 of ER visits to inpatient days, these folks were averaging less than 1-to-2.  Neither of these statistical anomalies seemed to bother Dr. Merchant enough to explain why his population might be such an outlier.

All this leads up to the savings figures.  Annualized gross savings were $688 in the study group.  Assuming a typical commercial health plan’s typical 4% asthma rate, $688 in savings/asthmatic equates to about a $27 per-member-per-year (PMPY) savings.  Nice work when you consider that we’ve never seen a commercial health plan that spends more than $20 PMPY on total asthma ER and IP events, and most spend about $15.

And if readers learn only one thing from our books and columns, it’s that you can’t reduce a number by more than 100%, no matter how hard you try.

Two postscripts are in order.  First, no one has ever accused us of being nuanced, so for a more nuanced view (but nonetheless a view generally consistent with ours) of Propeller Health, we might suggest David Shaywitz’ column in Forbes. Shaywitz points out, among other things, that the results are still preliminary.

Shaywitz is right, but Propeller isn’t letting such subtleties as a study which only has 4 months’ data and impossible results spoil their party:  their home page announces that “patients on Propeller save more than $600 a year.”  It further suggests that visitors “find out how” but no need – you now already know.

Second, we hope that the individuals involved with this study, who fortunately for readers on THCB turned down our offer to critique their study privately, decide after reviewing this posting to make their final results actually appear even slightly plausible, as opposed to epidemiologically and mathematically impossible, or at least explain how they achieved such massive savings and event reductions in the absence of increased drug use and asthma control scores.

The bad news, though, is that recent history shows reaction to our column on the part of perpetrators is exactly the reverse.  Recently Al and I outed Aetna for pitching questionable obesity drugs to people who aren’t even sick and didn’t request them.  Shortly thereafter, the Journal of the American Medical Association piled on with a “special communication” pointing out that these drugs are so unsafe they aren’t even approved in Europe.

Faced with that information, instead of backing down Aetna doubled down: a Tom Sawyeresque $300 will afford you the privilege of watching Aetna sell this program via a webinar.

We wonder if Propeller Health will be “hovering” on this webinar to figure out how they too can charge $300 to sell their program to attendees who haven’t read this column.

Vik Khanna is a St. Louis-based independent health consultant with extensive experience in managed care and wellness.  An iconoclast to the core, he is the author of the Khanna On Health Blog.  He is also the Wellness Editor-At-Large for THCB.

Al Lewis is the author of Why Nobody Believes the Numbers, co-author of Cracking Health CostsHow to Cut Your Company’s Health Costs and Provide Employees Better Care, and president of the Disease Management Purchasing Consortium.


[1] Of course that “improvement” was simply regression to the mean, perhaps with some Hawthorne Effect thrown in, due to the selection criteria favoring previously high utilizers.  However, we are focusing on the differences between the groups, not the groups compared to the baseline.   Nonetheless we would wonder why a principal investigator can’t distinguish between regression to the mean and actual improvement.

 

Livongo’s Post Ad Banner 728*90

26
Leave a Reply

16 Comment threads
10 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
14 Comment authors
Al LewisSarahthe social network latinoVince KuraitisEric Recent comment authors
newest oldest most voted
Sarah
Guest
Sarah

Hi Al!

Can I ask where you get the 4% commercial health plan asthma rate and $20PMPY total asthma ER and events figures? Is the 2nd averaged across all patients (ie. takes into account patients that do not go to the ER at all)? Or is that the amount for uncontrolled patients that do go to the ER? Just trying to get a better understanding of the numbers and math! (For the record I’m a student doing research on this subject).

Thanks

Al Lewis
Member

Hi Sarah, just seeing this now. That is asthma ER and IP events. You can go to the HCUP database and see the event counts (492xx is the ICD9 if memory serves), add them up and apply a cost factor. It actually works out to much less than $15 PEPY. I was being generous.

the social network latino
Guest

Recommend a client to hook up these with somebody within your interact.
It is not a formal relaxing meditation, but
a concerning-going awareness mind-calming exercise or continously allowing get out of fury,
fear, despair, disappointment and loneliness regarding stands in the form
of ones internal silence. Now, establishing one
social media marketing presence is no longer
an option for any service intention concerning creating an internet presence, and that includes writers searching to create a lover base.

Matthew Holt
Guest

Fair enough, Al, but really that’s not necessary. The nature of THCB is that they get to respond in the comments and usually in the main flow as well (see Farzad’s recent response to a bunch of critical comments about his ACO studies) There’s nothing wrong with Propeller saying “the kids liked it, the moms liked it, the docs (like Rajan) thought it improved their view of the care they delivered, and very preliminary there is a suggestion that it improved care and saved a bit of money–we’ll tell you more when the study is done, done” This is basically… Read more »

Al Lewis
Guest
Al Lewis

Matt, Eric, John, let me make a proposal that i think would both solve this problem AND make for more readers and a better debate. Anytime you receive a posting (whether from me&Vik or anyone else) that has someone’s NAME in it , that person gets the material X amount of time in advance and is told when it will be posted so that they can be ready with responses from themselves and their allies. If it is a “breaking news” story that interval might be 24 hours, otherwise 72. And it could be noted that the person/people in question… Read more »

Al Lewis
Guest
Al Lewis

A response, thank you! Well, someone seems to have provided the data to “Proppeller” because they are bragging about it on their screen shot above. It does seem like the study is well-designed in that you’ve managed to double-blind a device, which couldn’t have been easy, and kudos for that. So it should turn out plausibly. However, there is nothing plausible about massive reductions in inpatient utilization, a huge unexplained disparity in admit rates from the ER, drug use going in the opposite direction than expected, and savings on asthma events greater than every commercial health plan spends on asthma… Read more »

Rajan K. Merchant MD
Guest
Rajan K. Merchant MD

As the PI of this study we are seriously looking into the benefits of technology and the potential cost savings that are associated with this. The study design was to look at standard care of asthma vs. standard care of asthma plus information from Proppeller Device with our population. The premise that information about medication use has no value is incorrect. The study has not completed and full results to be available for review and publication by the end of the year. We provided prelimary results which was submitted for abstract publication. The Cost savings Data come from our Parent… Read more »

Matthew Holt
Guest

Eric–you should tell everyone more….or are you just assuming from Al/Vik’s line about “turned down offer to hear critique privately” that it was a “pay us to keep quiet” offer/threat. After all transparency about virtually everything is the idea behind lots of the Health 2.0 ideals.

However, I refer you to my comments to suggest that boasting about the savings was still the wrong approach

Eric
Guest
Eric

Al, thanks for the response. I think a simple technique is to give the company a chance to respond to your analysis and include their response in your post. Offering to have them pay you may come across as a veiled threat, both to the company in question and any health IT CEO reading this blog post that you have an interaction with in the future. Matt, my interpretation is that, despite the fact that Al/Vik are clearly smart people with a great skill set, they are using H2.0 as a vehicle to raise their profile using fear. A less… Read more »

Al Lewis
Guest
Al Lewis

I think that is an excellent idea! Consider it done for the future. Thank you.

Matthew Holt
Guest

Eric
a) THCB does not equal Health 2.0 and if you say it does, Indu Subaiya (Health 2.0’s CEO) will come around and put you on time out. I own THCB but John Irvine runs it and he’s the one who’s been allowing Al & Vik to run rampant! I’m lucky if I get to post my own stuff! Health 2.0 is a separate company (albeit one I own half of) and Al & Vik have nothing to do with it.

b) There is a place for Propeller Health to respond and it’s here in these comments.

Eric
Guest
Eric

My apologies, Matt. Thanks for clarifying.

Letting Propeller respond in the comments will come across as defensive rather than a debate. Show their response in the body of the post, as I think Al is suggesting he will do in the future.

Al, great. Including their response (or their refusal to respond!) will be perceived as much more fair.

Al Lewis
Guest
Al Lewis

I’ll leave the details up to you but remember how Jane Curtin and Dan Ackroyd used to debate on SNL? We wouldn’t remember if it was just one of them or the other. It would take THCB to the next level for buzz. And Vik and I — who would probably be on the receiving end more than others — welcome debate. Our frustration is quite the opposite. We put up a posting once “British Petroleum Wellness Program Is Spewing Invalidity,” which made Mercer and Staywell look like they had conspired to defraud BP, personally urged both to respond —… Read more »

Vince Kuraitis
Guest

I can see this quickly devolving into the next reality TV show, “Health 2.0 Cheaters”. Al, get your video camera ready.

Disappointed in Vik and Al
Guest
Disappointed in Vik and Al

Al and Vik,

So you are publicly submitting your business model as one of extortion? You offer to do a critique privately and then, if they don’t pay you, you submit the critique through a popular website?

you are attempting to brand yourselves as people who care about improving accuracy of research. Your willingness to accept to critique in “private” in exchange for cash undermines your messaging and credibility.

I have read and learned from your books and posts. I’m disappointed to learn that you are so easily bought.

Eric

Al Lewis
Guest
Al Lewis

Eric, that’s a really good point,. The problem is that when we’ve done the opposite on many occasions, people would say: “Why didn’t you call use first? We would have hired you to help us.” So we lose either way. What we ARE doing is disclosing that we are offering in advance to work for the people in question. If we thought there was anything untoward about it, we wouldn’t have brought it up at all on this post. And while there have been groups (household name groups) that as Matthew says below “pay us to keep quiet” because they… Read more »

Al Lewis
Guest
Al Lewis

Matt, good points as always. I think what’s happened is that so many people are trying to compete on ROIs and savings and so few benefits consultants and buyers can distinguish real value-added from fluff, that companies feel forced to make up savings numbers. If they don’t, they don’t have a chance.

Matthew Holt
Guest

Man is this depressing….. Now Al & Vik are not going after some 1980s-style wellness screener vendor, they’re going after a wonderful Health 2.0 tech company poster child, and yes it’s David van Sickle and his colleague’s fault. They’ve done 2 dumb things lately 1) change their name from Asthmapolis — which explicitly said what their thing does (maps out where asthmatics use their inhalers) to Propeller Health which is 3 letters from Propeller Head, with all the practical value that implies 2) Focus on cost in their first study (and then make the balls up of it that leaves… Read more »

Vik Khanna
Guest

Matt: we aren’t waiting for them at the bottom. We are waiting for at the top, becuase these guys, and anyone who follows them, is simply duplicating the wellness industry’s idiotic and simplistic meme that claiming a cost reduction is the most important thing that they can do. They lack the creativity, insight, intelligence, experience, and actual content knowledge to claim that they can improve care or position that improvement in a fundamentally useful way. Apparently, digital health people have learned nothing from the past 20 years of experience with managed care and that last 10 years of experience with… Read more »

Mitch Collins
Guest
Mitch Collins

Matt: You raise a host of good points, especially around the Health 2.0 point. I live in Philadelphia and have attended quite a few start-up events and seen everything from apps to platforms. When I ask what does your product do? What problem does it solve? Who will pay for it-who, specifically? The typical answer is something like “It is social-mobile-gamified-app-technology-disruptive and cool. And we are not health care IT guys. They are all dumb.” Really? The graveyard of these companies is filling rapidly, and there are plenty of near-dead zombie companies as well. Al nailed it when he pointed… Read more »

Al lewis
Guest
Al lewis

I saw a bag of almonds on JetBlue that said: “Studies suggest but do not prove that almonds reduce your cardiac event risk.” A very impressive piece of honesty Answer: Propeller is not likely to respond. We always give perps the opportunity to be coached in lieu of a public posting (albeit not for free) and they almost always decline. And they’d be nuts (no pun intended) to respond since this is their own information. All we did was reformat it 🙂 And the more visibility the debate gets the higher the google ranking. This is also a perfect example… Read more »

@BobbyGvegas
Guest

“their home page announces that “patients on Propeller save more than $600 a year.”
__

Well, we have also long know that “Clinical Studies at The Ponds Institute PROVE that our expensive goop will make you look younger.”

Gotta love this stuff.

Sam Lippe
Guest
Sam Lippe

Ben, you must be a newbie on this site. The “perps” never respond. That would just raise the visibility of the dialog and embarrass them further. And what can Propeller say?

Benjamin F. Troxell
Guest
Benjamin F. Troxell

Given the methodological issues of the study discussed in the article above, it would seem that a reply from the researchers would be in order but something tells me that such a thing is unlikely.

Al Lewis
Guest
Al Lewis

Thanks, Mr. (Ms.?) Datum. Good points. We should be a little more explicit. Propeller has a “smart” meter that in theory gives you more insight into when you should take your “rescue meds” to forestall serious attacks, based on your measured functional lung capacity. (Curiously, the study group ended up taking fewer rescue meds than the control group, though neither group took many puffs at all. This was not explained.) “Shaky” would be a polite word for these results. The results self-invalidate. There is also a major integrity issue in their screenshot. Not just that they say they save an… Read more »