Why do star ratings?
Now we’re giving star ratings to hospitals? Does anyone think this is a good idea? Actually, I do. Hospital ratings schemes have cropped up all over the place, and sorting out what’s important and what isn’t is difficult and time consuming. The Centers for Medicare & Medicaid Services (CMS) runs the best known and most comprehensive hospital rating website, Hospital Compare. But, unlike most “rating” systems, Hospital Compare simply reports data on a large number of performance measures – from processes of care (did the patient get the antibiotics in time) to outcomes (did the patient die) to patient experience (was the patient treated with dignity and respect?). The measures they focus on are important, generally valid, and usually endorsed by the National Quality Forum. The one big problem with Hospital Compare? It isn’t particularly consumer friendly. With the large number of data points, it might take consumers hours to sort through all the information and figure out which hospitals are good and which ones are not on which set of measures.
To address this problem, CMS just released a new star rating system, initially focusing on patient experience measures. It takes a hospital’s scores on a series of validated patient experience measures and converts them into a single star rating (rating each hospital 1 star to 5 stars). I like it. Yes, it’s simplistic – but it is far more useful than the large number of individual measures that are hard to follow. There was no evidence that patients and consumers were using any of the data that were out there. I’m not sure that they will start using this one – but at least there’s a chance. And, with excellent coverage of this rating system from journalists like Jordan Rau of Kaiser Health News, the word is getting out to consumers.
Our analysis
In order to understand the rating system a little bit better, I asked our team’s chief analyst, Jie Zheng, to help us better understand who did well, and who did badly on the star rating systems. We linked the hospital rating data to the American Hospital Association annual survey, which has data on structural characteristics of hospitals. She then ran both bivariate and multivariable analyses looking at a set of hospital characteristics and whether they predict receiving 5 stars. Given that for patients, the bivariate analyses are most straightforward and useful, we only present those data here.
Our results
What did we find? We found that large, non-profit, teaching, safety-net hospitals located in the northeastern or western parts of the country were far less likely to be rated highly (i.e. receiving 5 stars) than small, for-profit, non-teaching, non-safety-net hospitals located in the South or Midwest. The differences were big. There were 213 small hospitals (those with fewer than 100 beds) that received a 5-star rating. Number of large hospitals with a 5 star rating? Zero. Similarly, there were 212 non-teaching hospitals that received a 5-star rating. The number of major teaching hospitals (those that are a part of the Council of Teaching Hospitals)? Just two – the branches of the Mayo Clinic located in Jacksonville and Phoenix. And safety net hospitals? Only 7 of the 800 hospitals (less than 1%) with the highest proportion of poor patients received a 5-star rating, while 106 of the 800 hospitals with the fewest poor patients did. That’s a 15-fold difference. Finally, another important predictor? Hospital margin – high margin hospitals were about 50% more likely to receive a 5-star rating than hospitals with the lowest financial margin.
Here are the data:
Interpretation
There are two important points worth considering in interpreting the results. First, these differences are sizeable. Huge, actually. In most studies, we are delighted to see 10% or 20% differences in structural characteristics between high and low performing hospitals. Because of the approach of the star ratings, especially with the use of cut-points, we are seeing differences as great as 1500% (on the safety-net status, for instance).
The second point is that this is only a problem if you think it’s a problem. The patient surveys, known as HCAHPS, are validated, useful measures of patient experience and important outcomes unto themselves. I like them. They also tend to correlate well with other measures of quality, such as process measures andpatient outcomes. The star ratings nicely encapsulate which types of hospitals do well on patient experience, and which ones do less well. One could criticize themethodology for the cut-points that CMS used for determining how many stars to award for which scores. I don’t think this is a big issue. Any time you use cut-points, there will be organizations right on the bubble, and surely it is true that someone who just missed being a 5 star is similar to someone who just made it. But that’s the nature of cut-points – and it’s a small price to pay to make data more accessible to patients.
Making sense of this and moving forward
CMS has signaled that they will be doing similar star ratings for other aspects of quality, such as hospital performance on patient safety. The validity of those ratings will be directly proportional to the validity of the underlying measures used. For patient experience, CMS is using the gold standard. And the goals of the star rating are simple: motivate hospitals to get better – and steer patients towards 5-star hospitals. After all, if you are sick, you want to go to a 5-star hospital. Some people will be disturbed by the fact that small, for-profit hospitals with high margins are getting the bulk of the 5 stars while large, major teaching hospitals with a lot of poor patients get almost none. It feels like a disconnect between what we thinks are good institutions and what the star ratings seem to be telling us. When I am sick – or if my family members need hospital care, I usually choose these large, non-profit academic medical centers. So the results will feel troubling to many. But this is not really a methodology problem. It may be that sicker, poor patients are less likely to rate their care highly. Or it may be that the hospitals that care for these patients are generally not as focused on patient-centered care. We don’t know. But what we do know is that if patients start really paying attention to the star ratings, they are likely to end up at small, for-profit, non-teaching hospitals. Whether that is a problem or not depends wholly on how you define what is a high quality hospital.
Ashish Jha, MD is a professor at the Harvard School of Public Health and the Director of the Harvard Public School of Health.
Categories: Uncategorized
Great points, though you also have to account for he findings of the Fenton study, which found the opposite association between Healy outcomes and satisfaction. I agree that the medical profession does not pay enough attention to the patient experience, but that doesn’t mean that HCAHPS, and the star system are the right way to do it.
http://archinte.jamanetwork.com/article.aspx?articleid=1108766
“Patients can be very satisfied and dead an hour later. Sometimes hearing bad news is not going to result in a satisfied patient, yet the patient could be a well-informed, prepared patient.”
Any and all crashes of the EHR ought to be part of the safety ratings of the hospitals.
Excellent post, Dr. Jha. The analysis was very interesting and your comments were insightful.
Patient satisfaction scores may not be “right” — just as few of us take care of our health as well as we could/should. But that doesn’t mean we shouldn’t be listening to patients, and that we shouldn’t try to be better patients.
I used some of your analysis & comments in a related piece I wrote: http://kimbellardblog.blogspot.com/2015/04/does-patient-satisfaction-matter.html
Glad you posted this Bobby.
I dislike many “opinion” surveys because they ask questions which have no depth or are irrelevant to the experience and place everybody in the same box.
Was the nurse polite, was the furniture comfortable, were you seen on time?
Is there a single physician who posts here who would choose a hospital based on patient satisfaction scores?
Excerpt I posted on my blog:
__
The Problem With Satisfied Patients
A misguided attempt to improve healthcare has led some hospitals to focus on making people happy, rather than making them well.
ALEXANDRA ROBBINS APR 17 2015
When healthcare is at its best, hospitals are four-star hotels, and nurses, personal butlers at the ready—at least, that’s how many hospitals seem to interpret a government mandate.
When Department of Health and Human Services administrators decided to base 30 percent of hospitals’ Medicare reimbursement on patient satisfaction survey scores, they likely figured that transparency and accountability would improve healthcare. The Centers for Medicare and Medicaid Services (CMS) officials wrote, rather reasonably, “Delivery of high-quality, patient-centered care requires us to carefully consider the patient’s experience in the hospital inpatient setting.” They probably had no idea that their methods could end up indirectly harming patients.
Beginning in October 2012, the Affordable Care Act implemented a policy withholding 1 percent of total Medicare reimbursements—approximately $850 million—from hospitals (that percentage will double in 2017). Each year, only hospitals with high patient-satisfaction scores and a measure of certain basic care standards will earn that money back, and the top performers will receive bonus money from the pool.
Patient-satisfaction surveys have their place. But the potential cost of the subjective scores are leading hospitals to steer focus away from patient health, messing with the highest stakes possible: people’s lives…
[A] national study revealed that patients who reported being most satisfied with their doctors actually had higher healthcare and prescription costs and were more likely to be hospitalized than patients who were not as satisfied. Worse, the most satisfied patients were significantly more likely to die in the next four years.
Joshua Fenton, a University of California, Davis, professor who conducted the study, said these results could reflect that doctors who are reimbursed according to patient satisfaction scores may be less inclined to talk patients out of treatments they request or to raise concerns about smoking, substance abuse, or mental-health issues. By attempting to satisfy patients, healthcare providers unintentionally might not be looking out for their best interests. New York Times columnist Theresa Brown observed, “Focusing on what patients want—a certain test, a specific drug—may mean they get less of what they actually need. In other words, evaluating hospital care in terms of its ability to offer positive experiences could easily put pressure on the system to do things it can’t, at the expense of what it should.”
As a Missouri clinical instructor told me, “Patients can be very satisfied and dead an hour later. Sometimes hearing bad news is not going to result in a satisfied patient, yet the patient could be a well-informed, prepared patient.”…
And because almost every question on the survey involves nurses, some hospitals are forcing them to undergo unnecessary nonmedical training and spend extra time on superfluous steps. Perhaps hospitals’ most egregious way of skewing care to the survey is the widespread practice of scripting nurses’ patient interactions. Some administrators are ordering nurses to use particular phrases and to gush effusively to patients about both their hospital and their fellow nurses, and then evaluating them on how well they comply. An entire industry has sprouted, encouraging hospitals to waste precious dollars on expensive consultants claiming to provide scripts or other resources that boost satisfaction scores. Some institutions have even hired actors to rehearse the scripts with nurses.
In Massachusetts, a medical/surgical nurse told The Boston Globe that the scripting made her feel like a “Stepford nurse,” and wondered whether patients would notice that their nurses used identical phrasing. She’s right to be concerned. Great nurses are warm, funny, personal, or genuine—and requiring memorized scripts places a needless obstacle in their path.
The concept of “patient experience” has mischaracterized patients as customers and nurses as automatons. Some hospital job postings advertise that they are looking for nurses with “good customer-service skills” as their first qualification. University of Toledo Medical Center evaluates staff members on “customer satisfaction.”…
More disturbing, several health systems are now using patient satisfaction scores (likely from hospitals’ individual surveys) as a factor in calculating nurses’ and doctors’ pay or annual bonuses. These health systems are ignoring the possibility that health providers, like hospitals, could have fantastic patient satisfaction scores yet higher numbers of dead patients, or the opposite…
Many hospitals seem to be highly focused on pixie-dusted sleight of hand because they believe they can trick patients into thinking they got better care. The emphasis on these trappings can ultimately cost hospitals money and patients their health, because the smoke and mirrors serve to distract from the real problem, which CMS does not address: Patient surveys won’t drastically and directly improve healthcare.
But research has shown that hiring more nurses, and treating them well, can accomplish just that. It turns out that nurses are the key to patient satisfaction after all—but not in the way that hospitals have interpreted.
A Health Affairs study comparing patient-satisfaction scores with HCAHPS surveys of almost 100,000 nurses showed that a better nurse work environment was associated with higher scores on every patient-satisfaction survey question. And University of Pennsylvania professor Linda Aiken found that higher staffing of registered nurses has been linked to fewer patient deaths and improved quality of health. Failure-to-rescue rates drop. Patients are less likely to die or to get readmitted to the hospital. Their hospital stay is shorter and their likelihood of being the victim of a fatigue-related error is lower. When hospitals improve nurse working conditions, rather than tricking patients into believing they’re getting better care, the quality of care really does get better.
Instead, hospitals are responding to the current surveys and weighting system by focusing on smiles over substance, hiring actors instead of nurses, and catering to patients’ wishes rather than their needs. Then again, perhaps it’s no wonder that companies are airbrushing healthcare with a “Disney-like experience,” a glossy veneer. One of the leading consulting companies now advising hospitals on “building a culture of healthcare excellence” is, oddly enough, the Walt Disney Company.
__
http://www.theatlantic.com/health/archive/2015/04/the-problem-with-satisfied-patients/390684/