Uncategorized

Not All Ratings Are Equal

Earlier this month USNews and World Report released their annual list of America’s Best Hospitals. This list is terribly misleading and is a disservice to the readers of that magazine, in my opinion. The fine print is revealing:

“Central to understanding the rankings is that they were developed and the specialties chosen to help consumers determine which hospitals provide the best care for the most serious or complicated medical conditions and procedures—pancreatic cancer or replacement of a heart valve in an elderly patient with co- morbidities, for example. Medical centers that excel in relatively commonplace conditions and procedures, such as noninvasive breast cancer or uncomplicated knee replacement, are not the focus.”

Since when did breast cancer and knee replacements become so commonplace that they didn’t matter? On July 19, The New York Times published Doubt About Pathology Opinions for Early Breast Cancer, suggesting that diagnosing Stage 0 breast cancer was fairly difficult. And what is the bright-line test between “uncomplicated” and “complicated” knee surgery?

Research by Dr. Ash Sehgal recently published in the Annals of Internal Medicine has shown that the predictive nature of the USNews results is most closely tied to the opinions of the doctors surveyed over a three year period. About 100 doctors influence each of the 16 specialties annually. Four specialties of the 16 are 100% doctor opinion. In addition, only 1,892 of the 4,852 hospitals are even considered for ranking, and that only 152 get ranked (Read: that’s 3%)! Really, what if my pick-up truck won’t get there from here? What about the other 4,700 hospitals? Don’t they matter, or doesn’t the magazine care?

Full disclosure: I create health care ratings for a living, and that is my bias! I do it to help consumers find the most appropriate and best care for themselves and their families. I also do it to challenge the industry to improvement to reduce mortality, reduce complication rates, improve patient safety, increase efficiency and reduce costs, improve patient experience and finally to improve the functional status outcomes…the end result of recovering from illness. Once again, this all benefits people like you and me.I do this commercially for a fee, as I prefer to see foundation money used for medical innovation. I have been doing it longer than USNews, Thomson-Reuters, HealthGrades, WebMD, The Dartmouth Atlas, JDPower Healthcare, NCQA, Leapfrog Group, and dozens of Quangos and other self-proclaimed experts. With another round of rankings and ratings, it is time again to comment about our progress and purpose.

The U.S. News Most Serious or Complicated rating is a serious errata data. Now, if the magazine cover or name was America’s Best Hospitals for Most Serious or Complicated Care, then we could pause here. There is a huge difference (Read: statistically significant) between looking at only serious and complicated care and studying how hospitals do with the high volume of less serious and complex routine care! The vast majority of care delivered by the roughly 4,852 Medicare certified hospitals in this country is routinely complex! AND, plenty of things go wrong with routine care that we should all be concerned with. But does that make for good headlines?

Similarly, many policymakers and all members of the press other than Reed Abelson and Gardner Harris of the New York Timies are apparently unaware that The Dartmouth Atlas studies only Medicare Part A & B (i.e., no Medicare Advantage or Part D) beneficiaries’ costs during the last two years of life, for only those who ultimately died (Read: they don’t look at the success of those who survived). Perhaps calling it The Dartmouth Atlas of The Cost of Mortality would make the statistical relevance of the researchers’ important works even more evident. That, of course, would probably make The Dartmouth Atlas less popular in policy-making circles in Washington, D.C. if last summer’s talk of “death panels” is any indication. At least we would all know what bias there is in the denominator of the study group, without having to reference an antiquated footnote about prior research here, here and here. Note to researchers: If you are only going to look at high cost outliers at the end of life, you should not ignore the high cost outliers at the beginning of life too.

And then there is the just released Hospitals & Health Networks Magazine: 100 Most Wired Hospitals list. I guess if you are an HIT vendor you might be proud of a few clients who made it here or here. But the problem is, from my own analysis of the hospitals on the list (of which there are more than 100), less than 40% of them fall in the top quartile of performance on a variety of metrics relevant to hospital performance that those same information systems are supposed to help, such as patient experience, affordability and efficiency, patient safety, CMS Core Measures, 30-day re-admissions or 30-day mortality rates! So, if fewer than 40% of the “Most Wired” are in the top quartile of performance on the most basic metrics of quality and efficiency, what does that suggest for the likely success of HITECH and Meaningful Use and EMRs? If Hospitals & Health Networks Magazine and the CHIME board that selected these hospitals are adept at identifying the best technology, then the curious and the skeptics might ask what that technology is accomplishing for those institutions. If 10% of the “Most Wired” rank worse than the top 2,000 nationally (out of the same 4,852 hospitals) on the most important metrics of quality, safety, experience, efficiency and outcomes, the folks who think HITECH is a key part of reform may be in for a big surprise!

Certainly every researcher or magazine publisher would like to defend their works, but also every researcher, magazine publisher and licensor of researched methodology from academically funded research should embrace transparency, welcome constructive criticism and work to improve their works…else they aren’t really objective researchers seeking a solution to the problems we face.

In my opinion, the commercial world has done a better job of creating rating, ranking and review systems for hospitals and doctors than those in research or those affiliated with media outlets. WebMD, HealthGrades, DataAdvantage, Vitals.com, and Thomson-Reuters to name just a few, are all productively contributing toward moving the needle on provider performance and helping consumers in the end. Pick your poison, it isn’t an easy choice selecting healthcare, but if you stick with the ratings that are most transparent and functional, then at least you will know what you are reading. We have a long way to go to get it right…but just like in healthcare, we need to weed out the hype of those who are misleading and misinterpreting the data to sell a magazine or to influence health policy or to export the methodology to foreign corporations.

Let’s get with the guidelines folks; the consumer world is anxiously waiting our accurate help.

John R. Morrow has founded, created and contributed to a variety of national ratings programs including; 100 Top Hospitals : Benchmarks for Success℠ a Thomson-Reuters product, The Patient Satisfaction Index™ a National Research Corporation product, The Hospital Value Index™ a Press Ganey & Associates property, and is currently in Beta with Distinguished Doctor™, a new doctor profiling initiative. Morrow was a Principal at HCIA/Solucient, CEO of CHKS Ltd, SVP at HealthGrades and is Principal at The Ratings Guy LLC. John welcomes all comments.

Livongo’s Post Ad Banner 728*90

Categories: Uncategorized

Tagged as:

22
Leave a Reply

22 Comment threads
0 Thread replies
0 Followers
 
Most reacted comment
Hottest comment thread
20 Comment authors
burberryJohn Morrow:The Ratings GuySusen Rasmussenbeautriellisbeautrillis Recent comment authors
newest oldest most voted
burberry
Guest

Nice!! Great Ifo. Great People. Great Blog.and also share our website to you:Welcome to our burberry outlet store.Here you can enjoy discount Burberry products and burberry scarves with top quality. Burberry outlet online store,which leads the trend,can give you best shopping experience.

John Morrow:The Ratings Guy
Guest

@Susen, First, I hope that you would keep to your knitting and do the right thing for patients. Focusing on safe, efficient, low varaition outcomes would be in general a good theme for any provider. Doing the right things will always win. Also, realize that you are somewhat a victim of your market sociodemographics. Some hospitals just can’t dig out of being a “knife and gun club” location, being a major teaching hospital or Criticl Access Hospital…all of which profile less optimally from a ratings perspective, regardless of risk and severity models When you are doing the right things, benchmark… Read more »

Susen Rasmussen
Guest

So here’s a question. Assuming I live for ratings and want to move ahead in the US News & World variety, how would I best do that as a provider? HealthGrades and others sell consulting services, often addressing coding and clinical enhancements. Is my best bet working the earned media and communication channels to get my service (assuming it is indeed deserving) to register in the minds of physicians across the US? Are there actual consultants who carve this enhancement out as one of their key area of specialty? I ask because I have been asked this very question and… Read more »

beautriellis
Guest
beautriellis

You hit the nail on the head John. If we had more strategic thinkers like you, we might actually get somewhere. You get the big picture and it is refreshing to know there is still somebody out there who cares more about what the FACTS are and what the DATA SAYS than taking the word of the spinsters!

beautrillis
Guest
beautrillis

You hit the nail on the head John. Maybe you should lead the way…

juicy couture handbags
Guest

Juicy Couture bags are known for its very girly, bright design and affordable prices, this is perhaps why his creations are usually known by the younger decorated. In addition, those who still want to go, was to see a very young women to use their designs.
http://www.juicy-bags-sale.com

Clive from France
Guest
Clive from France

These are really important issues that you have raised and it particularly brings to mind the formula that I read recently in Freakonomics.
Risk = Hazard + Outrage (or in this case visibility)
We need worldwide, a better approach to collecting and analysing data which reduces the weight associated with outrage. This would provide a better handle on the most common interventions, their costs and outcomes as well as the impact of technological innovation.
We might then have a rational basis for policy making and for making individual decisions.

arun
Guest

It is thought provocating.It is really difficult to get unbiased rating.

Ruth
Guest

Property Magazines contain facts that would definitely be helpful for you to be familiar with the maze of property investment.

xocoslim
Guest
xocoslim

You are so right in your comment! I think it is quite impossible to get a real unbiased rating system for hospitals.But besides that,what worries me the most is that only 1,892 of the 4,852 hospitals are even considered for ranking.How many people believe what’s in The U.S. News? Too many I guess

Paul Levy
Guest

I agree with the above response to Dr. Millis. I can’t imagine how administrators could interfere with the peer review process in the way suggested, and I can’t imagine why any administrator would ever want to do so. That would be incredibly destructive of the process and undermine the underlying mission of a hospital. It would also quickly discredit that person’s position in the organization.

bev M.D.
Guest
bev M.D.

Dr. Millis;
Your allegation is a common one,but I don’t see how it can happen since the peer review process has to be documented at each step and reasons for sweeping something under the rug would have to be given for when JC comes to inspect. Do you have direct evidence that this is going on?
I am concerned because if it really is,then amalgamating docs and hospitals into ACO’s will exacerbate this problem. Do we need legislation excluding CEO’s from peer review or something?

Ohio Health Exchange
Guest

Maybe we need a “Consumer Reports” type of study. Of course, maybe they aren’t impartial any more either.

Harvey Millis, M.D.
Guest
Harvey Millis, M.D.

Doctors see what is wrong. They want to report. Peer review is run by hospital adminstrators. The complainers are silenced. The press does not publish the truth because of the heavy advertising budget of the hospitals and fear of suit. The following events occurred at a US News high rated hospital that advertises its ratings on TV, on full page newpaper ads, on buses, wherever:http://www.pittsburghlive.com/x/pittsburghtrib/news/pittsburgh/s_685271.html

Gary Lampman
Guest
Gary Lampman

The Facts are providers are use to keeping Patients in the Dark! They deal with given and taking Life.Thus they are accustom to hiding deep dark secrets from the Patients and Family. It is inconceivable to believe that Health Professionals would chose to be rated.As they have their own Peer review Group. Fellow practioners that share the cult of privledge and secrecy. Transparency is too close to their pocket books when it comes to rating a persons Knowledge and abilities. Their status of MD puts them above the scrutiny of their purveyors,members and patients.Like any pyrimid scheme,Spreading the wealth is… Read more »