OP-ED

Getting Quality Right: Exercise Due Caution When Grading Hospitals, Schools and Doctors

If Americans judged the quality of hospital care the way Newsweek judges high schools, we would soon be inundated with “charter hospitals” that only treat healthy patients.

As reported in The New York Times, thirty-seven of Newsweek’s top 50 high schools have selective admission standards, thereby enrolling the cream of the eighth grade crop. That means that when these high scoring eighth graders reach eleventh grade, they’ll be high scoring eleventh graders, helping the school move up the Newsweek rankings. These selective admission schools simply have to avoid screwing up their talented students.

That’s no way to determine how good a school is. The measure of a good education should be to assess how well students did in that school compared to how they would have been predicted to do if they had gone to other schools.

Imagine two liver transplant programs, one whose patients experience 90% survival in the year following their transplant and the other whose patients experience only a 75% survival rate. Based on that information, the former hospital looks like the place to go when your liver fails. But aren’t you curious about the kind of patients that receive care in these two hospitals? Wouldn’t you want to know whether that first hospital was padding its statistics by selectively transplanting relatively healthy patients?

When hospitals are judged by patient outcomes, savvy hospital administrators find ways to bolster their statistics. That’s why, according to a 2005 JAMA article, when New York State began reporting mortality rates and complication rates for patients undergoing cardiac surgery, hospitals in that state began to game the system. They found ways to avoid patients who were less likely to survive after surgery, minority patients, for example, or patients with lots of other illnesses, what doctors call co-morbidities.

New transplant programs, eager to qualify for Medicare reimbursement, work hard to bolster their transplant survival statistics, because Medicare looks for proof of success before counting a program as being eligible for reimbursement. The best way to achieve good survival statistics of course is to transplant the healthiest candidates. Heck, if you really want good survival rates, you should transplant me—I’m perfectly healthy (aside from a few worn-out joints)!

For the last two years, I have taught an undergraduate health policy course at Duke University with the assistance of Public Policy teaching assistants who have interests in education policy. These TAs often remark on the parallels between education policy and health policy. The challenges of measuring health care quality, for example, closely parallel the difficulties of assessing the quality of an educational institution. In healthcare, we “risk adjust” our measures, to even out the playing field between hospitals that care for otherwise different populations. These risk adjustments are not perfect by any means. But researchers are slowly improving their statistical measures.

Education policy needs to aggressively adopt the same kind of risk adjustment measures, if we hope to identify which high schools are truly doing the best job of educating their students and preparing them for the future.

Peter Ubel is a physician, behavioral scientist and author of Pricing Life: Why It’s Time for Health Care Rationing and Free Market Madness. He teaches business and public policy at Duke University. Peter’s new book, Critical Decisions will be available in the fall of 2012. You can follow him on his personal blog.

Livongo’s Post Ad Banner 728*90

16
Leave a Reply

11 Comment threads
5 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
10 Comment authors
Shimon Waldfogelmaithrism2012Maithrism2012 Recent comment authors
newest oldest most voted
Shimon Waldfogel
Guest

Peter, Margalit, Brian, … What if we re-framed the question regarding hospital quality and transparency? What if rather than focusing on comparing hospitals based on various measures, we challenge ourselves and ask what can we do to make all hospitals top hospitals? What if rather than listing all the barriers we accept that public reporting is as The Society of Thoracic Surgeons “believes the public has a right to know the quality of surgical outcomes and considers public reporting an ethical responsibility of the specialty”. http://www.sts.org/quality-research-patient-safety/sts-public-reporting-online What if rather than hospitals spending millions of dollars advertising their US News ranking… Read more »

Peter1
Guest
Peter1

“What if”

Yes Shimonoseki, what if indeed. What if patients designed health care and not corporate America.

maithri
Guest

sm2012-My point wasn’t to discount the value of certification. It was to say that measure of competence is a complex thing to measure past a certain point, especially as we start to narrow the focus of our assessment (be it certification for health care providers of specific specialties, health care providers who are entrenched in the system for a certain number of years, etc.). I’m aware that there are systems in place to encourage professional education and growth over the years… we might do well to better understand the ‘soundness’ of these particular systems…the incentive to make them so across… Read more »

Peter1
Guest
Peter1

“There is a huge amount of testing and recredentialing and clinical skills maintenance and ongoing knowledge assessment that is mandated as part of these various certification processes.” sm2012, are you saying all docs are created equal? Not sure if you’ve ever needed a doc for a complicated operation but would you or how would you seek out the best qualified surgeon? And if this “certification” process is so flawless then why is it so hard to get rid of incompetent docs? Teachers go through certification and re-evaluation as well, what’s your opinion of the teaching profession? “Just like airline pilots… Read more »

sm2012
Guest

Maithri – if certification is useless, then get rid of the board certification process and all of the accompanying organizations. Go back to the days when anyone could hang up a shingle and call themselves whatever they wanted. But that would be counter-productive to say the least. There is value in all of these systems. There is a huge amount of testing and recredentialing and clinical skills maintenance and ongoing knowledge assessment that is mandated as part of these various certification processes. Just like airline pilots requiring regular testing and updating of abilities, so do physicians and nurses have continuing… Read more »

Maithri
Guest

“What internal system exists to evaluate doctor competency?”

Good point–past the MCAT, or the boards, dare I say-how sound is certification? Who is putting forth the effort to make certification (which I suppose is a measure of ‘competence’) sound?

Margalit Gur-Arie
Guest

I think this obsessive-compulsive need to score and rate everything, as in top 10 this or 10 worse that, is becoming quite ridiculous, and counter productive.
Data is fine, transparency is fine, but manipulating and processing everything to death, does not necessarily shed any more light on the situation, and with a little bit of skill, may be used for the exact opposite.

sm2012
Guest

I’m not clear why some people are objecting to this post. What is Peter saying really? Providers and administrators are sometimes gaming the system because there are challenges to risk adjusting accurately and the education system needs similar risk adjustment to be able to make fair comparisons. Those both seem like very fair points. Brian, yes, analytics tools do incorporate for risk adjustment but as an analyst and consultant to hospitals changing their processes to accommodate public reporting, there is no question in my mind that accurately capturing co-morbidities on the front end is challenging and many people will just… Read more »

Peter1
Guest
Peter1

What internal system exists to evaluate doctor competency?

Austin Anderson
Guest
Austin Anderson

What a remarkable specimen of progressive intellectual virtuosity: take a perfectly pedestrian and noncontroversial staple of statistical technique (“Hey, let’s risk adjust those mortality statistics—brilliant!”) and then tendentiously extend the application to a witheringly more complex social experiment—inner city charter schools—as a means of impugning the latter. Yep, if you “risk adjust” in just the right way you’ll definitely be able to “prove” that that elitist charter school is actually no better than the wretched public school across the street, awash as it is in a sea of dysfunction and disadvantage. (Indeed, on a properly risk adjusted basis, that wretch… Read more »

Peter1
Guest
Peter1

“But would the good doctor really choose the wretched school for his own children, provided that on a risk adjusted basis it is no worse than the charter school across the street?”

No he wouldn’t, because the public school is not necessary to the success of his children who have everything they need to succeed via their doctor parent. The analogy may not be a good one but if you’re evaluating teachers, as opposed to doctors, the public ones should be getting paid twice what the charter ones are – too bad that doesn’t happen.

Margalit Gur-Arie
Guest

Let’s be honest for a minute….
If the data were adjusted for risk and socio-economic status and a variety of other things, and a school where 2% of kids go on to college, ended up being scored as excellent because of those adjustments, would you send your kids to that school?
If a hospital, had an absolute transplant survival rate of 10%, but after all the adjustments for treating very large numbers of homeless and poor people who are very sick and mostly hopeless, was rated as high quality, would you go there?

Peter1
Guest
Peter1

” would you go there?”

Good point, but if I was judging (without prejudice) on lots of experience treating difficult diagnosis then it may be my best option if I was a difficult case.

Schools are more about the right associations, prestige and bragging rights. Even poor parents want to send their kids to the rich school with already motivated and easily taught kids – less bad influences.

Brian Klepper
Guest

Peter, The failure to understand and adjust for risk levels has been around certainly for as long as I’ve been an analyst. All credible analytics tools now incorporate them. The larger and more shameful problem, though, is the failure of physicians, hospitals, health plans and everyone else in health care to make data available. Medicare physician data is still locked from public view, even though physicians taking Medicare assignment are vendors taking public dollars. Nearly all health plans treat their claims data as proprietary, and require clients receiving their own data to sign agreements that they won’t use it to… Read more »

southern doc
Guest
southern doc

Brian,

In primary care, at least, there are so few quality measures that hold up to any kind of rigorous analysis. Any thoughts on how to separate the good aples from the bad?

docster
Guest

great post.

short. sweet. and to the point.

also correct.