Tag: USMLE

USMLE Step 1 During COVID-19: A Fog of Uncertainty

Jun 19, 2020

By PRANAV PURI, PUNEET KAUR, and MARCUS WIGGINS, MBA

As current medical students, the ongoing COVID-19 pandemic represents the most significant healthcare crisis of our lifetimes. COVID-19 has upended nearly every element of healthcare in the United States, including medical education. The pandemic has exposed shortcomings in healthcare delivery ranging from the care of nursing home residents to the lack of interoperable health data. However, the pandemic has also exposed shortcomings in the residency match process.

Consider the United States Medical Licensing Examination (USMLE) Step 1. A 2018 survey of residency program directors cited USMLE Step 1 scores as the most important factor in selecting candidates to interview. Moreover, program directors frequently apply numerical Step 1 score cutoffs to screen applicants for interviews. As such, there are marked variations in mean Step 1 scores across clinical specialties. For example, in 2018, US medical graduates who matched into neurosurgery had a mean Step 1 scores of 245, while those matching into neurology had a mean Step 1 score of 231.

One would assume that, at a minimum, Step 1 scores are a standardized, objective measure to statistically distinguish applicants. Unfortunately, this does not hold true. In its score interpretation guidelines, the National Board of Medical Examiners (NBME) provides Step 1’s standard error of difference (SED) as an index to determine whether the difference between two scores is statistically meaningful. The NBME reports a SED of 8 for Step 1. Assuming Step 1 scores are normally distributed, the 95% confidence interval of a Step 1 score can thus be estimated as the score plus or minus 1.96 times the standard error (Figure 1). For example, consider Student A who is interested in pursuing neurosurgery and scores 231. The 95% confidence interval of this score would span from 215 to 247. Now consider Student B who is also interested in neurosurgery and scores 245. The 95% interval of this score would span from 229 to 261. The confidence intervals of these two scores clearly overlap, and therefore, there is no statistically significant difference between Student A and Student B’s exam performance. If these exam scores represented the results of a clinical trial, we would describe the results as null and dismiss the difference in scores as mere chance.

The Step 1 Score Reporting Change – A Step in the Right Direction for IMGs?

Feb 14, 2020• 1

By TALAL HILAL, MD

The United States Medical Licensing Examination (USMLE) Step 1, a test co-sponsored by the Federation of State Medical Boards (FSMB) and the National Board of Medical Examiners (NBME), has been the exam that people love to hate. For many years, blogs, Twitter feeds, and opinion pieces have been accumulating urging the presidents of the FSMB/NBME to stop reporting a 3-digit score and instead report a pass/fail score. This animosity towards the Step 1 exam originates from the reality that medical schools have increasingly focused their curriculum on teaching what the Step 1 wants you to learn – medical trivia that almost always has no bearing on how to approach a clinical problem.

This “Step 1 Madness” is unhealthy. The reasons for its existence are many: residency and fellowship programs allow it to exist by idolizing higher scores, some believe it is a metric that can predict future quality of care, board pass rates, etc. And some are naïve enough to think that what is tested on the Step 1 is actually useful medical knowledge! It may be due to a combination of the above that the Step 1 has found itself in such a peculiar spot. However, the emphasis on the Step 1 score means that medical students’ fate is being determined by a single test. Nobody wants their fate to be so unmalleable.

Those who were writing vehemently against a 3-digit score rejoiced when the FSMB/NBME announced on February 12 that the Step 1 will finally become a pass/fail test as early as January 2022!

#USMLEPassFail: A Brave New Day

Feb 13, 2020

By BRYAN CARMODY, MD

Well, it happened.

Beginning as soon as 2022, USMLE Step 1 scores will be reported pass/fail.

I’m shocked. Starting around two weeks ago, I began hearing rumors from some well-connected people that this might happen… but I still didn’t believe it.

I was wrong.

The response thus far has been enormous – I haven’t been able to clear my Twitter mentions since the news broke. And unsurprisingly, the reaction has been mixed.

In the future, I’ll post more detailed responses on where we go from here – but for now, I’d like to emphasize these five things.

Another MCQ Test on the USMLE

Oct 29, 2019• 1

By BRYAN CARMODY, MD

One of the most fun things about the United States Medical Licensing Examination (USMLE) pass/fail debate is that it’s accessible to everyone. Some controversies in medicine are discussed only by the initiated few – but if we’re talking USMLE, everyone can participate.

Simultaneously, one of the most frustrating things about the USMLE pass/fail debate is that everyone’s an expert. See, everyone in medicine has experience with the exam, and on the basis of that, we all think that we know everything there is to know about it.

Unfortunately, there’s a lot of misinformation out there – especially when we’re talking about Step 1 score interpretation. In fact, some of the loudest voices in this debate are the most likely to repeat misconceptions and outright untruths.

Hey, I’m not pointing fingers. Six months ago, I thought I knew all that I needed to know about the USMLE, too – just because I’d taken the exams in the past.

But I’ve learned a lot about the USMLE since then, and in the interest of helping you interpret Step 1 scores in an evidence-based manner, I’d like to share some of that with you here.

However…

If you think I’m just going to freely give up this information, you’re sorely mistaken. Just as I’ve done in the past, I’m going to make you work for it, one USMLE-style multiple choice question at a time._

USMLE Step 1: Leveling the Playing Field – or Perpetuating Disadvantage?

Mar 20, 2019• 2

By BRYAN CARMODY

Let me show you some data.

I’m going to show you the Match rate and mean Step 1 score for three groups of residency applicants. These are real data, compiled from the National Resident Matching Program’s (NRMP) Charting Outcomes in the Match reports.

Ready?

U.S. Allopathic Seniors: 92% match rate; Step 1 232.3
U.S. Osteopathic Seniors: 83% match rate; Step 1 225.8
International Medical Graduates, or IMGs (both U.S. and non-U.S. citizen: 53% match rate; Step 1 223.6

Now. What do you conclude when you look at these numbers?

In the debate over the U.S. Medical Licensing Examination’s (USMLE) score reporting policy, there’s one objection that comes up time and time again: that graduates from less-prestigious medical schools (especially IMGs) need a scored USMLE Step 1 to compete in the match with applicants from “top tier” medical schools.

In fact, this concern was recently expressed by the president of the National Board of Medical Examiners (NBME) in an article in Academic Medicine (quoted here, with my emphasis added).

“Students and U.S. medical graduates (USMGs) from elite medical schools may feel that their school’s reputation assures their successful competition in the residency application process, and thus may perceive no benefit from USMLE scores. However, USMGs from the newest medical schools or schools that do not rank highly across various indices may feel that they cannot rely upon their school’s reputation, and have expressed concern in various settings that they could be disadvantaged if forced to compete without a quantitative Step 1 score. This concern may apply even more for graduates of international medical schools (IMGs) that are lesser known, regardless of any quality indicator.”

The funny thing is, when I look at the data above, I’m not sure why we would conclude that IMGs are gaining advantage from a scored Step 1. In fact, we might conclude just the opposite – that a scored Step 1 is a key reason why IMGs have a lower match rate.

Should Medical Schools Teach to the Boards?

Apr 29, 2014• 13

By Abraar Karan

In the 2012 National Residency Match Program Survey, which is sent out to residency program directors around the country by the NRMP, the factor that was ranked highest with regards to criteria considered for receiving an interview—higher than honors in clinical clerkships, higher than extracurricular experiences or AOA election, and even higher than evidence of professionalism, interpersonal skills, and humanistic qualities—was the USMLE Step 1 score.

When considering where to rank an interviewed applicant, the Step 1 score took a backseat to some of the aforementioned criteria that are perhaps more telling of what kind of person the interviewee is, although it was still one of the highest considered criteria for ranking applicants as well.

When a single exam is given this level of importance in determining a future physician’s most critical period in career development—their residency—we have to look carefully at our system.

Two points of consideration come to mind. First, is it wise to weigh a test score so heavily? Many students and faculty could easily point out that student performance on exams by no means always reflects their clinical acumen and social skills when seeing patients.

Medicine is, after all, an art far more than a science.

Nonetheless, it would be foolish to assume that scores have no worth—a high score on an exam, particularly a behemoth such as the USMLE Step 1, points out many qualities in an individual: hard work, persistence, discipline, and frankly, an understanding of textbook medicine.

And thus, we are left somewhere in the middle—perhaps we should weigh scores less than we do, but when you have to sort through thousands of applications, the only standardized metric to quickly compare is, in the end, a number somewhere between 192 and 300.