By PRANAV PURI, PUNEET KAUR, and MARCUS WIGGINS, MBA
As current medical students, the ongoing COVID-19 pandemic represents the most significant healthcare crisis of our lifetimes. COVID-19 has upended nearly every element of healthcare in the United States, including medical education. The pandemic has exposed shortcomings in healthcare delivery ranging from the care of nursing home residents to the lack of interoperable health data. However, the pandemic has also exposed shortcomings in the residency match process.
Consider the United States Medical Licensing Examination (USMLE) Step 1. A 2018 survey of residency program directors cited USMLE Step 1 scores as the most important factor in selecting candidates to interview. Moreover, program directors frequently apply numerical Step 1 score cutoffs to screen applicants for interviews. As such, there are marked variations in mean Step 1 scores across clinical specialties. For example, in 2018, US medical graduates who matched into neurosurgery had a mean Step 1 scores of 245, while those matching into neurology had a mean Step 1 score of 231.
One would assume that, at a minimum, Step 1 scores are a standardized, objective measure to statistically distinguish applicants. Unfortunately, this does not hold true. In its score interpretation guidelines, the National Board of Medical Examiners (NBME) provides Step 1’s standard error of difference (SED) as an index to determine whether the difference between two scores is statistically meaningful. The NBME reports a SED of 8 for Step 1. Assuming Step 1 scores are normally distributed, the 95% confidence interval of a Step 1 score can thus be estimated as the score plus or minus 1.96 times the standard error (Figure 1). For example, consider Student A who is interested in pursuing neurosurgery and scores 231. The 95% confidence interval of this score would span from 215 to 247. Now consider Student B who is also interested in neurosurgery and scores 245. The 95% interval of this score would span from 229 to 261. The confidence intervals of these two scores clearly overlap, and therefore, there is no statistically significant difference between Student A and Student B’s exam performance. If these exam scores represented the results of a clinical trial, we would describe the results as null and dismiss the difference in scores as mere chance.
However, medical students and program directors know that this is not how USMLE Step 1 scores are interpreted in the residency match process. Practically speaking, program directors are inundated with applications and must filter applicants based on score cutoffs. For example, a neurosurgery program director faced with more than 100 applications for 2 spots may choose to filter out all applicants with Step 1 scores below 235. In this scenario, Student A will be filtered out of consideration and might be advised to pursue neurology instead of neurosurgery. Meanwhile, Student B will remain eligible for the position. Even though there is no statistically significant difference between their Step 1 scores, Student B is more likely to match into a neurosurgery position than Student A.
This example highlights the unyielding pressure medical students face to maximize USMLE Step 1 scores. USMLE Step 1 scores are certainly the most objective and standardized measure made available to program directors. However, these scores lack statistical precision, and therefore are effectively meaningless in distinguishing applicants. Recognizing the burdens imposed on medical students, the NBME and the Federation of State Medical Boards (FSMB) earlier this year announced that Step 1 will transition from a three digit numeric score to a pass/fail outcome, potentially as soon as 2022. Yet, until this change takes effect, program directors will likely continue to use USMLE Step 1 scores to filter applicants.
Unfortunately, this has placed current medical students in a precarious position. Due to the COVID-19 pandemic, Step 1 exams were not administered for nearly all of March and April. This created an enormous scheduling backlog. Medical students were forced to hastily reschedule exams and traveled across the country to limited open testing locations in May and June, often breaking state-issued social distancing guidelines in doing so. These chaotic changes have added further confusion and uncertainty to Step 1 scoring. In the absence of in-person clinical rotations, will program directors place even more weight on Step 1 scores? Will certain program directors make Step 1 score reporting optional? Are students taking the exam during the pandemic at a disadvantage in relation to students who took the exam before the pandemic? These are just a few of the countless questions medical students and their advisors are currently grappling with.
But times of crisis are also times of great opportunity. Now more than ever, the country is in need of a motivated and inspired cohort of medical graduates. However, from both a statistical and practical perspective, the uncertainty surrounding the residency match process and USMLE Step 1 score reporting is weighing heavily upon our next generation of physicians. Therefore, the Association of American Medical Colleges (AAMC) , NBME, and FSMB should harness the disruptions caused by the pandemic as an opportunity to implement more robust and equitable methods of evaluating medical students.
For one, the NBME and FSMB could accelerate their transition to pass/fail score reporting. Instead of waiting till 2022, pass/fail score reporting could be implemented immediately. However, this change by itself would not address the underlying problem. Residency program directors would continue to receive overwhelming volumes of applications. By adopting pass/fail score reporting, program directors would have even less objective, standardized information to distinguish applicants.
Therefore, a more prudent solution would combine pass/fail score reporting with a cap on the number of programs students could apply to. Program directors would no longer be overwhelmed by the sheer volume of applications they receive. Instead of superficially screening applicants based on Step 1 scores, program directors would be freed to more thoroughly and holistically evaluate applications.
Such changes would improve the transparency of the application cycle and allow both medical students and program directors to make more informed decisions in the Match. After all, as we emerge from the fog of the pandemic, wouldn’t we all benefit from some more clarity?
Pranav Puri (@pranavpuri17), Puneet Kaur, and Marcus Wiggins, MBA are third year medical students at Mayo Clinic Alix School of Medicine.