One of the most fun things about the United States Medical Licensing Examination (USMLE) pass/fail debate is that it’s accessible to everyone. Some controversies in medicine are discussed only by the initiated few – but if we’re talking USMLE, everyone can participate.
Simultaneously, one of the most frustrating things about the USMLE pass/fail debate is that everyone’s an expert. See, everyone in medicine has experience with the exam, and on the basis of that, we all think that we know everything there is to know about it.
Unfortunately, there’s a lot of misinformation out there – especially when we’re talking about Step 1 score interpretation. In fact, some of the loudest voices in this debate are the most likely to repeat misconceptions and outright untruths.
Hey, I’m not pointing fingers. Six months ago, I thought I knew all that I needed to know about the USMLE, too – just because I’d taken the exams in the past.
But I’ve learned a lot about the USMLE since then, and in the interest of helping you interpret Step 1 scores in an evidence-based manner, I’d like to share some of that with you here.
If you think I’m just going to freely give up this information, you’re sorely mistaken. Just as I’ve done in the past, I’m going to make you work for it, one USMLE-style multiple choice question at a time._
In the last fifteen years, we have witnessed dozens of natural disasters affecting our most vulnerable patients, from post-hurricane victims in Haiti to drought and famine refugees in Malawi. The vast majority of these patients suffered from acute on chronic disasters, culminating in life-threatening medical illnesses. Yet, during the course of providing clinical care and comfort, we rarely, if ever, pointed to climate change as the root cause of their conditions. The evidence for climate change is not new, but the movement for climate justice is now emerging on a large scale, and clinicians should play an active role.
Let’s be clear: there is no such thing as an “equal opportunity”
disaster. Yes, climate change poses an existential threat to us all, but not on
equal terms. When nature strikes, it has always been the poor and historically
underserved who are most vulnerable to its wrath. Hurricane Katrina provides an
example of how natural disasters target their victims along racial and
socioeconomic lines even in the wealthiest nations. Writes TalkPoverty.org, “A black homeowner in New Orleans was more than three times as
likely to have been flooded as a white homeowner. That wasn’t due to bad luck;
because of racially discriminatory housing practices, the high-ground was taken
by the time banks started loaning money to African Americans who wanted to buy
a home.” Throughout the world, historically marginalized communities have been
pushed to overcrowded, poorly-built, and unsanitary neighborhoods where natural
disasters invoke much greater harm.
So many primary care patients have several multifaceted problems these days, and the more or less unspoken expectation is that we must touch on everything in every visit. I often do the opposite.
It’s not that I don’t pack a lot into each visit. I do, but I tend to go deep on one topic, instead of just a few minutes or maybe even moments each on weight, blood sugar, blood pressure, lipids, symptoms and health maintenance.
When patients are doing well, that broad overview is perhaps all that needs to be done, but when the overview reveals several problem areas, I don’t try to cover them all. I “chunk it down”, and I work with my patient to set priorities.
What non-clinicians don’t seem to think of is that primary health care is a relationship based care delivery that takes place over a continuum that may span many years, or if we are fortunate enough, decades.
In a previous post, I described how some features of the Affordable
Care Act, despite the best intentions, have made it harder or even impossible
for many plans to compete against dominant players in the individual and small
employer markets. This has undermined aspects of the ACA designed to improve
competition, like the insurance exchanges, and exacerbated a long
term trend toward consolidation and reduced choice, and there is evidence it
is resulting in higher costs. I focused on the ACA’s risk adjustment program
and its impact on the small group market where the damage has been greatest.
The goal of risk adjustment is commendable: to create
stability and fairness by removing the ability of plans to profit by “cherry
picking” healthier enrollees, so that plans instead compete on innovative
services, disease management, administrative efficiency, and customer support.
But in the attempt to find stability, the playing field was tilted in favor of
plans with long-tenured enrollment and sophisticated operations to identify all
scorable health risks. The next generation of risk adjustment should truly even
out the playing field by retaining the current program’s elimination of an
incentive to avoid the sick, while also eliminating its bias towards incumbency
and other unintended effects.
One important distinction concerns when to use risk
adjustment to balance out differences that arise from consumer preferences. For
example, high deductible plans tend to attract healthier enrollees, and without
risk adjustment these plans would become even cheaper than they already are,
while more comprehensive plans that attract sicker members would get
disproportionately more expensive, setting off a race to the bottom that pushes
more and more people into the plans that have the least benefits, while the
sickest stay behind in more generous plans whose premium cost spirals upward. Using
risk adjustment to counteract this effect has been widely beneficial in the
individual market, along with other features like community rating and
However, in other cases where risk levels between plans differ
due to consumer preferences it may not be helpful. For example, it has been
documented that older and sicker members have a greater aversion to change (changing
plans to something less familiar) and to constraints intended to lower cost
even if they do not undermine benefit levels or quality of care, like narrow networks.
These aversions tend to make newer plans and small network plans score as
healthier. Risk adjustment would then force those plans to pay a penalty that in
turn forces enrollees in the plans to pay for the preferences of others.
It is not wise for Democrats to spend all their energy
debating Single Payer health care solutions.
None of their single player
plans has much chance to pass in 2020, especially under the limited
reconciliation process. In the words of Ezra Klein, “If Democrats don’t have a
plan for the filibuster, they don’t really have a plan for ambitious health
Yet while we debate Single Payer – or, even if it somehow
passed, wait for it to be installed — millions of persons are still hurting
under our current system.
We can help these people now!
Here are six practical programs to create a better ACA.
Taken all together they should not cost more than $50
billion a year. This is a tiny fraction of the new taxes that would be needed
for full single payer. This is at least negotiable, especially if Democrats can
take the White House and the Senate.
A few weeks ago, WTF Health took the show on the road to Australia’s digital health conference, HIC 2019. We captured more than 30 interviews (!) from the conference, which is run by the Health Informatics Society of Australia (hence the HISA Studio branding) and I had the opportunity to chat with most of the Australian Digital Health Agency’s leadership, many administrators from the country’s largest health systems, and a number of health informaticians, clinicians, and patients. I’ll be spotlighting a few of my favorites here in a four-part series to give you a flavor of what’s happening in health innovation ‘Down Under.’ For much more, check out all the videos on the playlist here.
This is the final post in our series, and in it I’m sharing four interviews on the theme of the future of the health tech workforce. This was a huge topic of conversation at HIC19 — dominating the discussion more than at any other conference I’ve been to in the US or Europe — and what struck me was all the different ways Aussies are looking at ‘workforce preparedness.’
There’s Kerryn Butler-Henderson, Associate Professor for Digital Health at the University of Tasmania, who is leading a Health Information Workforce Census that will take place in 2020. She’ll be “counting” the health data analysts, healthcare informaticians, health information managers, clinical coders and health librarians (more on what that job does in the interview) in not only Australia, New Zealand, and Tasmania, but also the US, UK, Canada, and Middle East to give us a larger look at the demographics of this part of the industry. A surprising take-away from her previous work in this space? More than 70% of health information workers are over the age of 45, signaling a shortage that could come up pretty quickly if we don’t start doing a better job of recruiting for the field.
With each passing year, the Affordable Care Act becomes
further entrenched in the American health care system. There are dreams on both
the far left and far right to repeal and replace it with something they see as
better, but the reality is that the ACA is a remarkable achievement which will
likely outlast the political lifetimes of those opposing it. Future
improvements are more likely to tweak the ACA than to start over from scratch.
A critical part of making the ACA work is for it to support
healthy, competitive and fair health insurance markets, since it relies on them
to provide health care benefits and improve access to care. This is
particularly true for insurance purchased by individuals and small employers,
where the ACA’s mandates on benefits, premiums and market structure have the
most impact. One policy affecting this dynamic that deserves closer attention
is risk adjustment, which made real improvements in the fairness of these
markets, but has come in for accusations that it has undermined competition.
Risk adjustment in the ACA works by compensating plans with
sicker than average members using payments from plans with healthier members.
The goal is to remove an insurer’s ability to gain an unfair advantage by
simply enrolling healthier people (who cost less). Risk adjustment leads insurers
to focus on managing their members’ health and appropriate services, rather
than on avoiding the unhealthy. The program has succeeded enormously in bringing
insurers to embrace enrolling and retaining those with serious health
This is something to celebrate, and we should not go back to
the old days in which individuals or small groups would be turned down for
health insurance or charged much higher prices because they had a history of
health issues. However, the program has also had an undesired effect in many states:
it further tilted the playing field in favor of market dominant incumbents.
Recently, I was on The Accad and Koka Report to share my opinions on USMLE Step 1 scoring policy. (If you’re interested, you can listen to the episode on the show website or iTunes.)
Most of the topics we discussed were ones I’ve already dissected on this site. But there was an interesting moment in the show, right around the 37:30 mark, that raises an important point that is worthy of further analysis.
ANISH: There’s also the fact that nobody is twisting the arms of program directors to use [USMLE Step 1] scores, correct? Even in an era when you had clinical grades reported, there’s still seems to be value that PDs attach to these scores. . . There’s no regulatory agency that’s forcing PDs to do that. So if PDs want to use, you know, a number on a test to determine who should best make up their class, why are you against that?
BRYAN: I’m not necessarily against that if you make that as a reasoned decision. I would challenge a few things about it, though. I guess the first question is, what do you think is on USMLE Step 1 that is meaningful?
ANISH: Well – um – yeah…
BRYAN: What do you think is on that test that makes it a meaningful metric?
ANISH: I – I don’t- I don’t think that – I don’t know that memorizing… I don’t even remember what was on the USMLE. Was the Krebs Cycle on the USMLE Step 1?
I highlight this snippet not to pick on Anish – who was a gracious host, and despite our back-and-forth on Twitter, we actually agreed much more than we disagreed. And as a practicing clinician who is 15 years removed from the exam, I’m not surprised in the least that he doesn’t recall exactly what was on the test.
I highlight this exchange because it illuminates one of the central truths in the #USMLEPassFail debate, and that is this:
Physicians who took Step 1 more than 5 years ago honestly don’t have a clue about what is tested on the exam.
That’s not because the content has changed. It’s because the memories of minutiae fade over time, leaving behind the false memory of a test that was more useful than it really was.
Today on Health in 2 Point 00, Jess and I are at Connected Health in Boston. On Episode 98, Jess asks me about Sandoz breaking up with Pear Therapeutics; Papa raising $10 million to send college students to keep seniors company (and I’m bitter); and the massive news of the Sutter settlement after they were sued for price fixing and monopoly behavior. Catch us at HLTH in Vegas in a couple of weeks, too. —Matthew Holt
Medical AI testing is unsafe, and that isn’t likely to change anytime soon.
No regulator is seriously considering implementing “pharmaceutical style” clinical trials for AI prior to marketing approval, and evidence strongly suggests that pre-clinical testing of medical AI systems is not enough to ensure that they are safe to use. As discussed in a previous post, factors ranging from the laboratory effect to automation bias can contribute to substantial disconnects between pre-clinical performance of AI systems and downstream medical outcomes. As a result, we urgently need mechanisms to detect and mitigate the dangers that under-tested medical AI systems may pose in the clinic.
In a recent preprint co-authored with Jared Dunnmon from Chris Ré’s group at Stanford, we offer a new explanation for the discrepancy between pre-clinical testing and downstream outcomes: hidden stratification. Before explaining what this means, we want to set the scene by saying that this effect appears to be pervasive, underappreciated, and could lead to serious patient harm even in AI systems that have been approved by regulators.
But there is an upside here as well. Looking at the failures of pre-clinical testing through the lens of hidden stratification may offer us a way to make regulation more effective, without overturning the entire system and without dramatically increasing the compliance burden on developers.