We’ve seen shorter abstracts, and we’ve seen abstracts with more curious findings, but we’ve never seen a shorter abstract with more curious findings than this one, done by Dignity Health and Dr. Rajan Merchant, and financed by the California Healthcare Foundation, evaluating a gadget made by Propeller Health.
The study group’s use of inpatient care for asthma declined by a whopping 62% vs. the control group. You might think this result violates Dr. John Ioannidis’ well-known conclusion that large treatment effects are usually wrong, but you’d be mistaken. You see, there was no treatment here.
There was only an effect. Dr. Ioannidis’ result applies only to actual comparisons of effects due to different treatments, not to random changes in effects using the same treatment. In this study, the actual treatment protocol was the same and the inhalers were the same.
The only thing different was frequency of drug use. Whereas the conventional wisdom for disease management states that hospitalizations can be avoided by more adherence and hence more drug use, in this case the study group used less medication than the control group, reaching for their rescue inhalers 25% less– once every 6.3 days vs. every 4.7 days for the control group.
The inevitable conclusion would be that the study group needed far fewer inpatient days and considerably fewer rescue puffs because their asthma was controlled much better. And that brings us to the most counterintuitive and impressive finding of all.
Yes, the most impressive part of the study was that this 62% reduction in inpatient days and 25% reduction in rescue drug usage was achieved even though overall asthma control improved similarly”[1] in both groups. Specifically, actual patient scores on the Asthma Control Test improved only by a statistically insignificant 2% in the study groups vs. the control groups.
In other words, the Propeller Health intervention had no effect on relative asthma control scores.
Putting all this information in a table makes it clear that Dr. Ioannidis is going to have to re-examine his conclusions because this Propeller Health study shows that it is indeed possible to achieve a massive reduction in events and costs even if nothing else changes except frequency of drug use, which changes in the “wrong” direction:
Nonetheless, let’s give the Propeller Health intervention credit for that 2% improvement in control scores. Avid readers of our wellness postings and Why Nobody Believes the Numbers may recall that the ratio of event reduction to quality score improvement is called the “Wishful Thinking Multiplier,” and Propeller Health’s Multiplier of 31x just shattered Viverae’s record of 17x.
(The real answer, of course, is that on a good day event changes mirror quality improvement changes. That itself could be wishful thinking, and there are many reasons why even a 1x Wishful Thinking Multiplier is difficult to achieve, as described in the book.)
We’re also not sure where exactly Dr. Merchant found these asthmatics but, wow, are they ever sick. During the baseline period they were running 326 asthma inpatient days per 1000 asthmatics. A typical group of 1000 asthmatics will incur about a quarter that many inpatient days.
Perhaps that’s because when these folks show up in the ER they almost always get admitted: Whereas the typical asthmatic has a ratio of about 5-to-2 of ER visits to inpatient days, these folks were averaging less than 1-to-2. Neither of these statistical anomalies seemed to bother Dr. Merchant enough to explain why his population might be such an outlier.
All this leads up to the savings figures. Annualized gross savings were $688 in the study group. Assuming a typical commercial health plan’s typical 4% asthma rate, $688 in savings/asthmatic equates to about a $27 per-member-per-year (PMPY) savings. Nice work when you consider that we’ve never seen a commercial health plan that spends more than $20 PMPY on total asthma ER and IP events, and most spend about $15.
And if readers learn only one thing from our books and columns, it’s that you can’t reduce a number by more than 100%, no matter how hard you try.
Two postscripts are in order. First, no one has ever accused us of being nuanced, so for a more nuanced view (but nonetheless a view generally consistent with ours) of Propeller Health, we might suggest David Shaywitz’ column in Forbes. Shaywitz points out, among other things, that the results are still preliminary.
Shaywitz is right, but Propeller isn’t letting such subtleties as a study which only has 4 months’ data and impossible results spoil their party: their home page announces that “patients on Propeller save more than $600 a year.” It further suggests that visitors “find out how” but no need – you now already know.
Second, we hope that the individuals involved with this study, who fortunately for readers on THCB turned down our offer to critique their study privately, decide after reviewing this posting to make their final results actually appear even slightly plausible, as opposed to epidemiologically and mathematically impossible, or at least explain how they achieved such massive savings and event reductions in the absence of increased drug use and asthma control scores.
The bad news, though, is that recent history shows reaction to our column on the part of perpetrators is exactly the reverse. Recently Al and I outed Aetna for pitching questionable obesity drugs to people who aren’t even sick and didn’t request them. Shortly thereafter, the Journal of the American Medical Association piled on with a “special communication” pointing out that these drugs are so unsafe they aren’t even approved in Europe.
Faced with that information, instead of backing down Aetna doubled down: a Tom Sawyeresque $300 will afford you the privilege of watching Aetna sell this program via a webinar.
We wonder if Propeller Health will be “hovering” on this webinar to figure out how they too can charge $300 to sell their program to attendees who haven’t read this column.
Vik Khanna is a St. Louis-based independent health consultant with extensive experience in managed care and wellness. An iconoclast to the core, he is the author of the Khanna On Health Blog. He is also the Wellness Editor-At-Large for THCB.
Al Lewis is the author of Why Nobody Believes the Numbers, co-author of Cracking Health Costs: How to Cut Your Company’s Health Costs and Provide Employees Better Care, and president of the Disease Management Purchasing Consortium.
[1] Of course that “improvement” was simply regression to the mean, perhaps with some Hawthorne Effect thrown in, due to the selection criteria favoring previously high utilizers. However, we are focusing on the differences between the groups, not the groups compared to the baseline. Nonetheless we would wonder why a principal investigator can’t distinguish between regression to the mean and actual improvement.
Categories: Uncategorized
Hi Sarah, just seeing this now. That is asthma ER and IP events. You can go to the HCUP database and see the event counts (492xx is the ICD9 if memory serves), add them up and apply a cost factor. It actually works out to much less than $15 PEPY. I was being generous.
Hi Al!
Can I ask where you get the 4% commercial health plan asthma rate and $20PMPY total asthma ER and events figures? Is the 2nd averaged across all patients (ie. takes into account patients that do not go to the ER at all)? Or is that the amount for uncontrolled patients that do go to the ER? Just trying to get a better understanding of the numbers and math! (For the record I’m a student doing research on this subject).
Thanks
Recommend a client to hook up these with somebody within your interact.
It is not a formal relaxing meditation, but
a concerning-going awareness mind-calming exercise or continously allowing get out of fury,
fear, despair, disappointment and loneliness regarding stands in the form
of ones internal silence. Now, establishing one
social media marketing presence is no longer
an option for any service intention concerning creating an internet presence, and that includes writers searching to create a lover base.
I can see this quickly devolving into the next reality TV show, “Health 2.0 Cheaters”. Al, get your video camera ready.
I’ll leave the details up to you but remember how Jane Curtin and Dan Ackroyd used to debate on SNL? We wouldn’t remember if it was just one of them or the other.
It would take THCB to the next level for buzz. And Vik and I — who would probably be on the receiving end more than others — welcome debate. Our frustration is quite the opposite. We put up a posting once “British Petroleum Wellness Program Is Spewing Invalidity,” which made Mercer and Staywell look like they had conspired to defraud BP, personally urged both to respond — as did many of the commenters — and couldn’t get boo out of them.
Fair enough, Al, but really that’s not necessary. The nature of THCB is that they get to respond in the comments and usually in the main flow as well (see Farzad’s recent response to a bunch of critical comments about his ACO studies)
There’s nothing wrong with Propeller saying “the kids liked it, the moms liked it, the docs (like Rajan) thought it improved their view of the care they delivered, and very preliminary there is a suggestion that it improved care and saved a bit of money–we’ll tell you more when the study is done, done”
This is basically what Sense Health did in their study in NYC–said it has promising results but not going crazy about savings http://mobihealthnews.com/31238/startup-bolsters-medicaid-care-managers-with-two-way-text-messaging/
Full Disclosure; my company Health 2.0 is managing the Pilot Health Tech program that this Sense Health pilot is part of (funded by the NYC Economic Development Corporation), but no we don’t manage Sense Health’s press & marketing outreach!
My apologies, Matt. Thanks for clarifying.
Letting Propeller respond in the comments will come across as defensive rather than a debate. Show their response in the body of the post, as I think Al is suggesting he will do in the future.
Al, great. Including their response (or their refusal to respond!) will be perceived as much more fair.
Matt, Eric, John, let me make a proposal that i think would both solve this problem AND make for more readers and a better debate. Anytime you receive a posting (whether from me&Vik or anyone else) that has someone’s NAME in it , that person gets the material X amount of time in advance and is told when it will be posted so that they can be ready with responses from themselves and their allies.
If it is a “breaking news” story that interval might be 24 hours, otherwise 72. And it could be noted that the person/people in question declined to respond.
Eric
a) THCB does not equal Health 2.0 and if you say it does, Indu Subaiya (Health 2.0’s CEO) will come around and put you on time out. I own THCB but John Irvine runs it and he’s the one who’s been allowing Al & Vik to run rampant! I’m lucky if I get to post my own stuff! Health 2.0 is a separate company (albeit one I own half of) and Al & Vik have nothing to do with it.
b) There is a place for Propeller Health to respond and it’s here in these comments.
I think that is an excellent idea! Consider it done for the future. Thank you.
Al, thanks for the response. I think a simple technique is to give the company a chance to respond to your analysis and include their response in your post. Offering to have them pay you may come across as a veiled threat, both to the company in question and any health IT CEO reading this blog post that you have an interaction with in the future.
Matt, my interpretation is that, despite the fact that Al/Vik are clearly smart people with a great skill set, they are using H2.0 as a vehicle to raise their profile using fear. A less one-sided approach might have the same exact result but would go a long way towards making these articles educational, which is what we all come here for.
A response, thank you!
Well, someone seems to have provided the data to “Proppeller” because they are bragging about it on their screen shot above.
It does seem like the study is well-designed in that you’ve managed to double-blind a device, which couldn’t have been easy, and kudos for that. So it should turn out plausibly.
However, there is nothing plausible about massive reductions in inpatient utilization, a huge unexplained disparity in admit rates from the ER, drug use going in the opposite direction than expected, and savings on asthma events greater than every commercial health plan spends on asthma events. Also, calling regression to the mean (and Hawthorne effect) “improvement” doesn’t inspire confidence.
It is certainly possible that information about medication use has value. We never said it didn’t . However, taking these results literally, a little extra information about medication has far more impact on asthma events than any medication that has ever been tested in clinical trials, except possibly the very expensive Advair. Surely you don’t think that’s plausible?
Nonetheless, we appreciate your response and wish you the best of luck in your future study. We look forward to seeing how this posting and comments inform your final writeup.
Eric, that’s a really good point,. The problem is that when we’ve done the opposite on many occasions, people would say: “Why didn’t you call use first? We would have hired you to help us.” So we lose either way.
What we ARE doing is disclosing that we are offering in advance to work for the people in question. If we thought there was anything untoward about it, we wouldn’t have brought it up at all on this post.
And while there have been groups (household name groups) that as Matthew says below “pay us to keep quiet” because they simply want to rip off HR directors, most organizations that hire us genuinely want to learn how to measure population health outcomes. It’s a topic that is not covered in PhD biostatistics classes.
What would you propose we do — the surprise method or the “would you like us to consult with you privately?” method.
PS And if as Matthew says you “boast about savings” that don’t exist, you really can’t expect a free private consult.
As the PI of this study we are seriously looking into the benefits of technology and the potential cost savings that are associated with this. The study design was to look at standard care of asthma vs. standard care of asthma plus information from Proppeller Device with our population. The premise that information about medication use has no value is incorrect. The study has not completed and full results to be available for review and publication by the end of the year. We provided prelimary results which was submitted for abstract publication. The Cost savings Data come from our Parent Organization Dignity Health. We looked at total cost of treatment/utilization for all individuals the year prior to patients enrolling. We then looked at what utilization/cost was incurred during the study. That is how the cost saving was calculated. This is not information that is being supplied by Proppeller. Only when the study is fully completed will we know the extent or lack of benefit. Anything prior is pure speculation.
Eric–you should tell everyone more….or are you just assuming from Al/Vik’s line about “turned down offer to hear critique privately” that it was a “pay us to keep quiet” offer/threat. After all transparency about virtually everything is the idea behind lots of the Health 2.0 ideals.
However, I refer you to my comments to suggest that boasting about the savings was still the wrong approach
Al and Vik,
So you are publicly submitting your business model as one of extortion? You offer to do a critique privately and then, if they don’t pay you, you submit the critique through a popular website?
you are attempting to brand yourselves as people who care about improving accuracy of research. Your willingness to accept to critique in “private” in exchange for cash undermines your messaging and credibility.
I have read and learned from your books and posts. I’m disappointed to learn that you are so easily bought.
Eric
Matt:
You raise a host of good points, especially around the Health 2.0 point. I live in Philadelphia and have attended quite a few start-up events and seen everything from apps to platforms. When I ask what does your product do? What problem does it solve? Who will pay for it-who, specifically? The typical answer is something like “It is social-mobile-gamified-app-technology-disruptive and cool. And we are not health care IT guys. They are all dumb.”
Really?
The graveyard of these companies is filling rapidly, and there are plenty of near-dead zombie companies as well.
Al nailed it when he pointed out “Technology is not psychology. It is all about motivation.” And technology has yet to crack that.
Matt, good points as always. I think what’s happened is that so many people are trying to compete on ROIs and savings and so few benefits consultants and buyers can distinguish real value-added from fluff, that companies feel forced to make up savings numbers. If they don’t, they don’t have a chance.
Matt: we aren’t waiting for them at the bottom. We are waiting for at the top, becuase these guys, and anyone who follows them, is simply duplicating the wellness industry’s idiotic and simplistic meme that claiming a cost reduction is the most important thing that they can do. They lack the creativity, insight, intelligence, experience, and actual content knowledge to claim that they can improve care or position that improvement in a fundamentally useful way.
Apparently, digital health people have learned nothing from the past 20 years of experience with managed care and that last 10 years of experience with wellness. Gizmos, gadgets, lotions, and potions are all well and good, but until you have a content-driven message rather than concocted cost claims, all you have is a trinket that will eventually be forgotten. As for me and Al, we like playing whack-a-mole.
Man is this depressing…..
Now Al & Vik are not going after some 1980s-style wellness screener vendor, they’re going after a wonderful Health 2.0 tech company poster child, and yes it’s David van Sickle and his colleague’s fault. They’ve done 2 dumb things lately
1) change their name from Asthmapolis — which explicitly said what their thing does (maps out where asthmatics use their inhalers) to Propeller Health which is 3 letters from Propeller Head, with all the practical value that implies
2) Focus on cost in their first study (and then make the balls up of it that leaves Al & Vik with a simple tap-in to the open goal).
Their tool/system is great–believe me! It uses a clever GPS gizmo on the inhaler to track each “rescue use” This leads to 2 wonderful clinical things
a) it tells where, when and whether a patient (usually a kid) took their medication, which can provide lots of useful information about what triggers an asthma attack, and therefore can provide lots of useful information to change that kid’s care for the better (possibly why they used it less, such as “stop taking the path to school past the cattle feed lot as that kicks off your attacks). Not to mention figuring out patient compliance with correct use of an inhaler, etc which helps in education, behavior change, etc etc
b) it creates a great population map of asthma attack inducers which is of tremendous potential benefit to public health researchers, and citizens.
Instead Propeller went straight to the end goal–cost (“we save $600!!”).
Why do they have to do this? It’s a trap and no one else in health care does it. Does Epic say, “put in our EMR for $1 billion and you’ll save money”? No. They do say (and Kaiser’s more or less proved) that if you put it in and make a bunch of changes, you’ll deliver better care. Do stent makers say “you’ll save money using our device”? C’mon, 6+ years after the COURAGE trial we are using more stents than before… even though the trial should it’s not only more expensive but clinically worse care.
No they don’t, they say that they have a tool for improving care, (even if it’s not true)
Why oh why is everyone in the Health 2.0/digital health world heading down this saving money rabbit hole? Because when they screw up Al & Vik are waiting for them at the bottom….and it’ll be tough to change their messaging later.
I saw a bag of almonds on JetBlue that said: “Studies suggest but do not prove that almonds reduce your cardiac event risk.” A very impressive piece of honesty
Answer: Propeller is not likely to respond. We always give perps the opportunity to be coached in lieu of a public posting (albeit not for free) and they almost always decline. And they’d be nuts (no pun intended) to respond since this is their own information. All we did was reformat it 🙂
And the more visibility the debate gets the higher the google ranking.
This is also a perfect example of our mantra: in population health, you don’t have to challenge the data to invalidate it. You merely have to read the data. It will invalidate itself.
“their home page announces that “patients on Propeller save more than $600 a year.”
__
Well, we have also long know that “Clinical Studies at The Ponds Institute PROVE that our expensive goop will make you look younger.”
Gotta love this stuff.
Ben, you must be a newbie on this site. The “perps” never respond. That would just raise the visibility of the dialog and embarrass them further. And what can Propeller say?
Given the methodological issues of the study discussed in the article above, it would seem that a reply from the researchers would be in order but something tells me that such a thing is unlikely.
Thanks, Mr. (Ms.?) Datum. Good points. We should be a little more explicit. Propeller has a “smart” meter that in theory gives you more insight into when you should take your “rescue meds” to forestall serious attacks, based on your measured functional lung capacity. (Curiously, the study group ended up taking fewer rescue meds than the control group, though neither group took many puffs at all. This was not explained.)
“Shaky” would be a polite word for these results. The results self-invalidate. There is also a major integrity issue in their screenshot. Not just that they say they save an impossible amount based on 4 months’ data for 40 people. They didn’t bother to subtract the $200+ that the control group saved, from the “more than $600 a year” claim. So they did a so-called controlled study and then ignored the results in the control group.
And yes, HR people are the Mikeys of this field. They’ll sign up for anything that says it saves money. Brokers and benefits consultants enable this addiction because they make more money too.
Interestiing. my only comments: I wish you had introduced propeller health for the benefit of those of us not intimately familiar with the company and its product. The fact that the research the company is trotting out to back up its claims is shaky may come as a surprise to a few HR types and naive investors, but it is nothing new for the rest of us.
This is only the tip of the iceberg, with digital health routinely breaking out the data to validate their models, thereis much work like this to be done.
Note: the only thing you have not shown (and cannot show) is that the company’s technology does not work ..