
When report cards of performance became available, cardiac surgeons in New York and Pennsylvania avoided high risk patients. Could something similar happen, nationally, after the forthcoming revolution in transparency inspired by Propublica’s data release?
Take two fictional orthopedic surgeons, Cherry Picker MD and Morbidity Hunter MD.
Cherry Picker lives in the Upper East Side of New York. His patients give him great reviews on Yelp. His patients read every comment on Yelp before making any decision. Cherry Picker has a beautiful family. When he smiles, light refracts from his shiny teeth.
Cherry regularly appears on TV. He writes for the sleek, metrosexual publication, FHM. Cherry specializes in knee injuries in weekend warriors. His patients often call him from the ski slopes in Colorado, Whistler and Zermatt. Cherry is good at his craft. But his patients are even better at their craft – post-operative recovery. Cherry doesn’t actively seek such patients. His patients are selected for him by his zip code, reputation, long waiting list and Yelp.
Morbidity Hunter’s real name is Harjinder Singh. He migrated from Punjab and works in a safety net hospital in North Philadelphia. Singh wanted to work in Beverley Hills, but to convert his J1-visa to a green card, he had to work in an area of need. Once he started working, he liked his job. His daughters liked their school and his wife liked the house they bought. Singh doesn’t have shiny teeth. He hasn’t appeared on TV, although his daughters tease that he can play Sonny from Exotic Marigold Hotel.
Singh’s colleagues named him Morbidity Hunter because he operates regardless of how sick his patients are. He never says no. Nearly all his patients are obese and diabetic. The School of Public Health sends students to shadow him to learn about polypharmacy. The hospital went on a spree of hiring hospitalists when Singh started.
His patients, straddling the Federal Poverty Limit, don’t rate him on Yelp. His patients don’t use Yelp. Even if they were informed consumers they would have to choose Singh, because there are very few orthopedic surgeons who are willing to operate on them in that zip code. His patients haven’t heard of Cherry Picker. They don’t ski, ballroom dance or run half marathons.
Singh, too, is good at his craft. Technically excellent, to be precise. You wouldn’t know that from looking at the rates of readmission, infection, and deep vein thrombosis in his patients. But the staff in the operating room know that, as do his colleagues, whom he has often helped out in tough operations. Even Cherry admires him.
Singh is not in for the money. He doesn’t make as much money as Cherry, but makes enough. He doesn’t operate for glory. He operates for professional pride – an ethereal concept that eludes some health economists.
It’s hard to zap the morale of this sturdy lad from the Punjab. But the data transparency movement achieved that. He always knew that operating on the sickest, poorest and most disenfranchised section of society was not going to be lucrative. But he never knew he was going to be made the captain of their ship – he was happy to captain the placement of their total hip – but what happened before or after they entered the operating room was not his fault, he felt.
People began to call Singh an incompetent surgeon. He objected, but he could not understand the logic behind the numbers which were incriminating him. His complication rates were the highest in Philadelphia. Numbers don’t lie, supposedly. This was too much for him to bear. He didn’t mind losing the pitiful bonuses that CMS was withholding from him, but the reason broke his heart – his poor quality.
Singh was puzzled by people who claimed to lose sleep over the poor. The chasm between their sentimentality and actions baffled him. Punjab began to make more sense than Philadelphia. But then Cherry invited Singh to join his practice in New York. Cherry promised Singh that he could operate on technically challenging patients. Grudgingly, Singh accepted the offer, which made his wife very excited about shopping for Indian food in Queens. She insisted, though, that Singh had to see a dentist first.
Homo sapiens have always sought redemption. Today it is through data. Numbers have replaced Yahweh and Indra. But, just like the old gods were, numbers can be moody, arbitrary and, occasionally, downright unfair. Numbers are a human construct, after all.
My favorite is Simpson’s paradox – where the conclusions are actually, and precisely, the opposite of what is inferred from the data. That is, for example, when a study shows the superiority of an inferior treatment, and vice versa.
You can imagine the God of the Old Testament yelling as Abraham was about to sacrifice Isaac, “Stop that’s Simpson’s paradox. Now I know you fear God. Put the knife down, slowly. Isaac, promise daddy you will learn your times tables.”
The data release by Propublica is a reservoir of Simpson’s paradox. This means when the data says “bad surgeon”, the surgeon might, in fact be a Top Gun – a technically-gifted, Morbidity Hunter – the last hope of the poor and sick.
Aren’t you intrigued and perturbed by this paradox? This means that data may not be just telling half-truths, but flat out lying. I thought we were done with burning innocents at Salem.
This graph made by John Alan Tucker PhD (@JohnTuckerPhD), an analyst, is instructive. Let’s assume that a complication rate of 2.5 % is the national benchmark of surgeons. Let’s say that a surgeon, Good Enough MD, in reality meets the benchmark. The probability that the data will show a complication rate of 4 % in Good Enough – that is 60 % over the benchmark – is nearly 40 %, if we sample his last 35 procedures. Forty percent! Would you accept that degree of uncertainty for a new statin in the market? Are surgeons more expendable than Lipitor?
Chance, it seems, favors the high-volume surgeon. But don’t get too excited. If Good Enough’s last 200 procedures are recorded, the chances that the data will show a 4 % complication rate, when his true rate is actually 2.5 %, are 15 %. This is not as high as forty percent, but hardly respectable.
This figure, in which Tucker included the much touted risk-adjustment, is no more reassuring. You can see that the complication rates of hip surgeons in California that fall within the 95 % confidence interval. This means that the difference between the nearly worst and nearly best hip surgeon could be the roll of a dice. Of course, that’s not true. The technical skills of surgeons lie in a spectrum. The numbers do not replicate the spectrum.
This highlights a curious ethical issue. If a patient has a right to know about a surgeon’s performance, the point estimate, should they not be informed about the methodological limitations of that measurement, the confounders, the confidence interval – which is as large as the elephant in the room?
No, some will say, it’s information overload. But limiting the information we give patients just because we think they can’t handle the information has a name. What’s it called again? Ah yes, paternalism.
We are at the brink of a revolution in accountability in medicine. We are at the brink of some revolution, or other, a dozen times a day. The future of the transparency movement is bright. The future for patients who need Morbidity Hunter MD is not so good. Harjinder from Punjab won’t look after the poorest and sickest – even for a green card – if you call him a bad surgeon.
Categories: Uncategorized
Looks to me that if the BEST hip surgeon in CA has a complication rate of 0.8% and the WORST has a complication rate of 4.0%, the people in California are pretty damn lucky to have plenty of good orthopedic surgeons. What the heck are they cryin about?
We have nationalized healthcare – it’s called Medicare.
There are ways to make it work – both transparency and appropriate services for high risk patients. Perhaps not all high risk patients should have joint replacements – if they knew the realistic risks of it and had to pay for part of their care if they do not follow necessary follow up instructions, everyone would benefit. Having an elective surgery essentially end your life because of death or disabling complications isn’t good for anyone, and as a society we just can’t afford it any longer. We should follow the example of other countries, I think, who limit publicly/communally funded healthcare.
The fight will be between the managers (read: masterminds) and their immediate subordinates (who lazily assuage their guilt by empowering government to dictate more and more of our lives).
The proletariat have more pressing things to do than fight a revolution, such as subsisting … oh yeah, and sitting on the couch watching the Kardashians, Netflix, and ESPN. Panem et circenses.
If we will not rule over both ourselves and our government, government and others will rule over us.
“Let perfect not be the enemy of good.”
“All models are wrong, some are useful.”
Risk adjustment is not perfect, but the question is how imperfect and in which group is this model, and yes it’s only a model, the wrongest?
Propublica should let independent group run sensitivity analyses and these should be published.
Not sure the consumer is informed about the limitations of methodology – they should be.
Marx was right. But he was wrong about who would fight who. It’ll be between the managers and their immediate subordinates. The proletariat have more pressing things to do than fight a revolution, such as subsisting.
Paying for outcomes and punishing docs for not avoiding the sick who are also poor, will lead to an interesting cognitive dissonance. I am keen to see how we resolve it.
I don’t think I’d get the job as a bot! The problem in the media is homogeneity of opinion. It’s a serious problem. Blogs can mitigate it, but only to an extent.
Thanks. That’s exactly right Allan. Risk is uneven, money is uneven – paying for performance doesn’t even pretend to align the two.
There is a cultural shift that is going to be required of everyone in health care toward more openness and transparency. If the measures we have aren’t right, let’s get more data to fix them, but let’s do it in the open. If Morbidity Hunter’s patients are more complex, then let’s find a measure for that rather than closing our eyes to what data we have. Be part of the solution. Burying heads in the sand won’t get us anywhere. Also, if Morbidity Hunter has made it this far, I suspect he’ll have greater mental fortitude than to storm out of the country if someone presents data that says his outcomes aren’t so great. First, he should know! Second, he can have input on what to measure differently and maybe even what he, his hospital or his patients could do to get better outcomes. How is that bad? If it takes bruising a few egos for a few lives, it’s worth it.
A fun read but…
Haven’t you answered your own questions? Re Simpson’s, the data have to be partitioned such that the high risk, obese, diabetic, high sed rate, high CRP, smokers, drug users, high cholesterol, bad zip code folks, blah blah blah are identified and surgical outcomes calculated and shown for just these folks? Then, Dr. Morbidity would shine and Dr. Cherry would be revealed as average or whatever.
Re low N statistics, don’t you have to go ahead and tell the patient plus or minus confidence limits whether he wants the information or not?
apropos (?), read “The end of capitalism has begun.” http://www.theguardian.com/books/2015/jul/17/postcapitalism-end-of-capitalism-begun
Yes, among the questions raised:
Who will take on the tough cases? Accept the most desperate of all patients? Perform the riskiest of procedures? And how will such physicians gain the needed experience and even come into being in the first place, with such disincentives in place?
Furthermore, will doctors game the system in response, as you have reported? Will they be assigned patients from risk pools to even things out?
With each program and initiative, these holier-than-thou, good-government activists create new problems, which is to be expected: they are only capable of Stage One Thinking. One hasty intervention leads to another, and the natural endpoint is nationalized healthcare.
Of course, not to encourage them in these efforts, such outcomes rankings will have to be adjusted by propensity scoring, patients’ socioeconomic status, and regression models to be meaningful. Propublica claims to be doing this, but how effectively?
Lastly, the president recently talked about our current “corrupt” model which pays surgeons to amputate limbs, but not to keep patients intact. So be forewarned: they intend to impose this model not just to *reporting outcomes* but directly to actual *physician compensation*. When that happens, the unintended consequences will be orders of magnitude worse.
Oh heck I just assumed it was Narrative Science writing again, that bot that ProPublica, Forbes and many others use to write their data news today, saves time and bots also help rig the news as well. I try to watch where I comment as well so I’m not writing a comment to a news story written by a bot, as bots don’t care what you say..it gets tagged and run through for distribution, though and all the way in Facebook today:) 60% of what’s out there is written by bots today so do yourself a favor and look for better quality stuff that human journalists are putting out. I think we are all tired of “Stat Rat Fever” but HHS and CMS has not caught on to that yet either:) News Rigging has arrived..here’s what some of it is with fake grass root campaigns and how the bots get the job done, so watch for good stuff that humans are still writing.
Again with data stories, the bots really excel at this type of story telling. I cross over and talk to the nerdy financial guys here and there and about a year ago we were talking about “news rigging” so just getting bigger. It’s not the media’s fault either, but rather the cash rich cheap public companies that rely on the news to move their stocks that won’t fork out a few dollars to support the very industry that moves and sells their stock…what a fluke.
http://ducknetweb.blogspot.com/2015/04/news-rigging-has-arrived-astroturf-and.html
Excellent, something that should appear in the lay press over and over again. Every time a good medical center flies someone out to Mayo and keeps those too sick to fly one should think that the ability to fly elsewhere means a higher survival rate for the recipient hospital and a lower survival rate for the hospital that sent the patient..