Not All Ratings Are Equal

Earlier this month USNews and World Report released their annual list of America’s Best Hospitals. This list is terribly misleading and is a disservice to the readers of that magazine, in my opinion. The fine print is revealing:

“Central to understanding the rankings is that they were developed and the specialties chosen to help consumers determine which hospitals provide the best care for the most serious or complicated medical conditions and procedures—pancreatic cancer or replacement of a heart valve in an elderly patient with co- morbidities, for example. Medical centers that excel in relatively commonplace conditions and procedures, such as noninvasive breast cancer or uncomplicated knee replacement, are not the focus.”

Since when did breast cancer and knee replacements become so commonplace that they didn’t matter? On July 19, The New York Times published Doubt About Pathology Opinions for Early Breast Cancer, suggesting that diagnosing Stage 0 breast cancer was fairly difficult. And what is the bright-line test between “uncomplicated” and “complicated” knee surgery?

Research by Dr. Ash Sehgal recently published in the Annals of Internal Medicine has shown that the predictive nature of the USNews results is most closely tied to the opinions of the doctors surveyed over a three year period. About 100 doctors influence each of the 16 specialties annually. Four specialties of the 16 are 100% doctor opinion. In addition, only 1,892 of the 4,852 hospitals are even considered for ranking, and that only 152 get ranked (Read: that’s 3%)! Really, what if my pick-up truck won’t get there from here? What about the other 4,700 hospitals? Don’t they matter, or doesn’t the magazine care?

Full disclosure: I create health care ratings for a living, and that is my bias! I do it to help consumers find the most appropriate and best care for themselves and their families. I also do it to challenge the industry to improvement to reduce mortality, reduce complication rates, improve patient safety, increase efficiency and reduce costs, improve patient experience and finally to improve the functional status outcomes…the end result of recovering from illness. Once again, this all benefits people like you and me.I do this commercially for a fee, as I prefer to see foundation money used for medical innovation. I have been doing it longer than USNews, Thomson-Reuters, HealthGrades, WebMD, The Dartmouth Atlas, JDPower Healthcare, NCQA, Leapfrog Group, and dozens of Quangos and other self-proclaimed experts. With another round of rankings and ratings, it is time again to comment about our progress and purpose.

The U.S. News Most Serious or Complicated rating is a serious errata data. Now, if the magazine cover or name was America’s Best Hospitals for Most Serious or Complicated Care, then we could pause here. There is a huge difference (Read: statistically significant) between looking at only serious and complicated care and studying how hospitals do with the high volume of less serious and complex routine care! The vast majority of care delivered by the roughly 4,852 Medicare certified hospitals in this country is routinely complex! AND, plenty of things go wrong with routine care that we should all be concerned with. But does that make for good headlines?

Similarly, many policymakers and all members of the press other than Reed Abelson and Gardner Harris of the New York Timies are apparently unaware that The Dartmouth Atlas studies only Medicare Part A & B (i.e., no Medicare Advantage or Part D) beneficiaries’ costs during the last two years of life, for only those who ultimately died (Read: they don’t look at the success of those who survived). Perhaps calling it The Dartmouth Atlas of The Cost of Mortality would make the statistical relevance of the researchers’ important works even more evident. That, of course, would probably make The Dartmouth Atlas less popular in policy-making circles in Washington, D.C. if last summer’s talk of “death panels” is any indication. At least we would all know what bias there is in the denominator of the study group, without having to reference an antiquated footnote about prior research here, here and here. Note to researchers: If you are only going to look at high cost outliers at the end of life, you should not ignore the high cost outliers at the beginning of life too.

And then there is the just released Hospitals & Health Networks Magazine: 100 Most Wired Hospitals list. I guess if you are an HIT vendor you might be proud of a few clients who made it here or here. But the problem is, from my own analysis of the hospitals on the list (of which there are more than 100), less than 40% of them fall in the top quartile of performance on a variety of metrics relevant to hospital performance that those same information systems are supposed to help, such as patient experience, affordability and efficiency, patient safety, CMS Core Measures, 30-day re-admissions or 30-day mortality rates! So, if fewer than 40% of the “Most Wired” are in the top quartile of performance on the most basic metrics of quality and efficiency, what does that suggest for the likely success of HITECH and Meaningful Use and EMRs? If Hospitals & Health Networks Magazine and the CHIME board that selected these hospitals are adept at identifying the best technology, then the curious and the skeptics might ask what that technology is accomplishing for those institutions. If 10% of the “Most Wired” rank worse than the top 2,000 nationally (out of the same 4,852 hospitals) on the most important metrics of quality, safety, experience, efficiency and outcomes, the folks who think HITECH is a key part of reform may be in for a big surprise!

Certainly every researcher or magazine publisher would like to defend their works, but also every researcher, magazine publisher and licensor of researched methodology from academically funded research should embrace transparency, welcome constructive criticism and work to improve their works…else they aren’t really objective researchers seeking a solution to the problems we face.

In my opinion, the commercial world has done a better job of creating rating, ranking and review systems for hospitals and doctors than those in research or those affiliated with media outlets. WebMD, HealthGrades, DataAdvantage, Vitals.com, and Thomson-Reuters to name just a few, are all productively contributing toward moving the needle on provider performance and helping consumers in the end. Pick your poison, it isn’t an easy choice selecting healthcare, but if you stick with the ratings that are most transparent and functional, then at least you will know what you are reading. We have a long way to go to get it right…but just like in healthcare, we need to weed out the hype of those who are misleading and misinterpreting the data to sell a magazine or to influence health policy or to export the methodology to foreign corporations.

Let’s get with the guidelines folks; the consumer world is anxiously waiting our accurate help.

John R. Morrow has founded, created and contributed to a variety of national ratings programs including; 100 Top Hospitals : Benchmarks for Success℠ a Thomson-Reuters product, The Patient Satisfaction Index™ a National Research Corporation product, The Hospital Value Index™ a Press Ganey & Associates property, and is currently in Beta with Distinguished Doctor™, a new doctor profiling initiative. Morrow was a Principal at HCIA/Solucient, CEO of CHKS Ltd, SVP at HealthGrades and is Principal at The Ratings Guy LLC. John welcomes all comments.

Spread the love

Categories: Uncategorized

Tagged as:

22 replies »

  1. @Susen,
    First, I hope that you would keep to your knitting and do the right thing for patients. Focusing on safe, efficient, low varaition outcomes would be in general a good theme for any provider. Doing the right things will always win. Also, realize that you are somewhat a victim of your market sociodemographics. Some hospitals just can’t dig out of being a “knife and gun club” location, being a major teaching hospital or Criticl Access Hospital…all of which profile less optimally from a ratings perspective, regardless of risk and severity models
    When you are doing the right things, benchmark against other best practices and against other top performing peers. Don’t just look at outcomes or Core Process Measures, you need to develop a full dashboard of metrics to monitor all aspects of performance; mortality, complications, patient safety indicators, re-admission rates, patient satisfaction, employee satisfaction, and begin to measure functional status of patients 30-60 and 90 days after discharge. An ACO will be doing all of these things well.
    If you simply want to make it on a rating firm’s list, study their methodology closely and clearly understand what the data periods are, what the sources are and how each metric is calculated. In so doing you may find that coding optimization will help, you may also find what the medical executive committee already knows. Not all team members deliver the same results. EG, 92% of doctors believe they perform better than average…of course you know that to be statistically impossible. Most good firms will tell you how far off you are for making thie list. Obviously if a selection criteria excludes community hospitals, and you are one of those, that isn’t a recognition worth placing much value upon.
    Find your own sweet spot. Find an area of success and work to spread that success to other areas or service lines. Not every hospital does everything really well, but most everyone does something worth promoting. The rating vendors can usually help with that as well. Or EMail me and I will hook you up with someone independent who can.
    Most importantly, do it for the right reasons…not because it is a marketing strategy…but because it will matter to your community.

  2. So here’s a question. Assuming I live for ratings and want to move ahead in the US News & World variety, how would I best do that as a provider? HealthGrades and others sell consulting services, often addressing coding and clinical enhancements. Is my best bet working the earned media and communication channels to get my service (assuming it is indeed deserving) to register in the minds of physicians across the US? Are there actual consultants who carve this enhancement out as one of their key area of specialty? I ask because I have been asked this very question and really don’t have an answer. I am a purist who would prefer that actions speak louder than ratings. Mortality, bedside staffing and error rates are very clinical/outcomes based and it seems a clinical operational consultant or internal quality department may be the best resource. I would love your thoughts on the matter.

  3. You hit the nail on the head John. If we had more strategic thinkers like you, we might actually get somewhere. You get the big picture and it is refreshing to know there is still somebody out there who cares more about what the FACTS are and what the DATA SAYS than taking the word of the spinsters!

  4. These are really important issues that you have raised and it particularly brings to mind the formula that I read recently in Freakonomics.
    Risk = Hazard + Outrage (or in this case visibility)
    We need worldwide, a better approach to collecting and analysing data which reduces the weight associated with outrage. This would provide a better handle on the most common interventions, their costs and outcomes as well as the impact of technological innovation.
    We might then have a rational basis for policy making and for making individual decisions.

  5. You are so right in your comment! I think it is quite impossible to get a real unbiased rating system for hospitals.But besides that,what worries me the most is that only 1,892 of the 4,852 hospitals are even considered for ranking.How many people believe what’s in The U.S. News? Too many I guess

  6. I agree with the above response to Dr. Millis. I can’t imagine how administrators could interfere with the peer review process in the way suggested, and I can’t imagine why any administrator would ever want to do so. That would be incredibly destructive of the process and undermine the underlying mission of a hospital. It would also quickly discredit that person’s position in the organization.

  7. Dr. Millis;
    Your allegation is a common one,but I don’t see how it can happen since the peer review process has to be documented at each step and reasons for sweeping something under the rug would have to be given for when JC comes to inspect. Do you have direct evidence that this is going on?
    I am concerned because if it really is,then amalgamating docs and hospitals into ACO’s will exacerbate this problem. Do we need legislation excluding CEO’s from peer review or something?

  8. Doctors see what is wrong. They want to report. Peer review is run by hospital adminstrators. The complainers are silenced. The press does not publish the truth because of the heavy advertising budget of the hospitals and fear of suit. The following events occurred at a US News high rated hospital that advertises its ratings on TV, on full page newpaper ads, on buses, wherever:http://www.pittsburghlive.com/x/pittsburghtrib/news/pittsburgh/s_685271.html

  9. The Facts are providers are use to keeping Patients in the Dark! They deal with given and taking Life.Thus they are accustom to hiding deep dark secrets from the Patients and Family.
    It is inconceivable to believe that Health Professionals would chose to be rated.As they have their own Peer review Group. Fellow practioners that share the cult of privledge and secrecy.
    Transparency is too close to their pocket books when it comes to rating a persons Knowledge and abilities. Their status of MD puts them above the scrutiny of their purveyors,members and patients.Like any pyrimid scheme,Spreading the wealth is paramont. While any rating or bad press would dispell the illusion that all is well in La La land.

  10. Ranking a doctor is like ranking an artist or a chef, because medicine is more art than science.
    Who are the physicians of last resort?
    Who are the physicians who instill the most confidence?
    Who deploys the best strategies to maximize quantity and quality of life?
    You can even get a clunker at the great “clinic” of the world, and being wired is no proof of excellence: http://www.pittsburghlive.com/x/pittsburghtrib/news/pittsburgh/s_692683.html
    Want to see a reduced central line infection rate? When the hospital administrators pay the infection control director enough, all infections will “disappear”.

  11. I knew that I loved you the moment I read ‘Since when did breast cancer and knee replacements become so commonplace that they didn’t matter?’
    Reminds me of one definition of minor surgery I heard: Something that happens to someone else.

  12. After looking at a wide variety of quality and popularity measures for almost 20 years now, I have come to the conclusion that all hospitals must be renamed Lake Wobegon Memorial Medical Center, where all physicians good looking, all the nurses are tall and all the patients are above average.
    Americans still have more and better information in selecting a new dishwasher or automobile than a physician, surgeon, or hospital.
    What I do know is hospital administrators do pay attention to these scorecards which is one indication that something has changed.

  13. It is extremely helpful and beneficial to have health care facilities rated. It is not easy choosing a health care provider. Better choices are made through
    Informed, and an educated decision making process.
    In my city there is a lot of weight put on the individual physician as opposed to the facility. I believe that is because of the size of my town, and a reputation precedes itself.
    Our hospital is part of a much larger organization with what the call telemedicine. Whereas the physicians here call and consult physicians in another much larger city.

  14. Amen. Magazines are in the business of selling magazines; not necessarily looking out for the public health.
    Rankings go back to the argument of making consumer decisions based on health grades and scorecards. This too though, is frought with confusion, bias, uneven playing fields and the like. However, I’ll be happy to wade through the mountains of interpretive data rather than pick up a news or financial rag for a microwave ready pick of where to go for my laminectomy or pulmonary resection.
    I also think the consumer makes many of their medical choices based on physician reputation, rather than facility reputation. I’m not underselling the latter, I just feel physicians carry a lot of weight in the minds of patients when they make their healthcare choices.
    Obviously, my hospital administrators and their marketing agencies will differ from that opinion. But if I have confidence in my cardiothoracic surgeon, I’ll go where he/she has privileges, even if it’s at a place that didn’t make the US News Top 100.

  15. All good points John. As you correctly indicate, comparative analysis of hospital performance, which for many years (decades?) served as a basis for improving outcomes and lowering cost, has now become a media circus. As we devolve into a environment with guidelines for legitimacy dictated by TMZ and People Magazine, we wind up with a headline-driven mentality, rather than one which fosters innovation and improvement in care and management of costs. Keep writing John.

Leave a Reply

Your email address will not be published.