Pass/Fail for USMLE Step 1: Examining Program Director Values and Match Statistics to Imagine the Future of Candidate Selection

Introduction

The residency selection process poses complex challenges and pressures medical students considerably. Traditionally, the United States Medical Licensing Examination (USMLE) Step 1 has been a pivotal factor in the selection of candidates for interviews, as repeatedly highlighted by program directors.¹ Nevertheless, the dynamics of the selection process underwent a substantial transformation in 2022, when residency programs were no longer privy to the USMLE Step 1 three-digit score of applicants. This shift introduced uncertainty for students concerning interview selection and compelled residency programs to explore alternative assessment methods.²

Research suggests that there might be a shift in the assessment focus of applications following the transition of the USMLE score reporting to a pass/fail model. Investigations conducted by the National Resident Matching Program (NRMP) have shed light on the criteria deemed necessary by program directors in selecting candidates for interviews, particularly using USMLE Step 1 as a screening tool based on either pass/fail criterion or a three-digit score.³

In response to these developments, the primary objectives of this study are as follows:

1. To identify the characteristics of residency programs that correlate with the use of USMLE Step 1 as either a pass/fail indicator or a three-digit score.

2. To discern the factors that are considered vital in the selection of applicants for an interview, contingent on the program’s current implementation of USMLE Step 1.

3. To investigate trends in the reported importance of various components of the residency application during the sample period.

Through analyzing aggregate match statistics and program directors’ reports, our study attempts to illuminate the evolving dynamics of residency application evaluations. This provides clarity in an uncertain landscape.

Methods

Project design

We utilized publicly available data obtained from three years (2018, 2020, and 2021) of National Residency Matching Program-Program Director (NRMP-PD) survey results^4–6 and the corresponding three years of Electronic Residency Application Service (ERAS)⁷ and Association of American Medical College (AAMC)⁸ reported Match statistics.^9–11 In 2019, there was no NRMP-PD survey. The survey respondents from 19 residency specialties were included in the analysis. We excluded the Transitional Year due to it being a pre-residency preparatory experience and Radiation-Oncology, Thoracic Surgery, and Vascular Surgery due to missing data. This study was IRB exempt because of its minimal risk and publicly available data.

Variables

The primary outcome variable was the percentage of programs within a specialty using USMLE Step 1 as either pass/fail or 3-digit score when selecting applicants for interview. Secondary outcome measures were the reported percent importance of various components of the residency application when selecting candidates for interviews.

We evaluated the following predictor variables for the use of USMLE Step 1 (primary outcome): specialty type (procedural or cognitive), the average number of applications received, the number of interview invitations offered, and the number of positions offered.

Covariates considered were the following: the percent of applicants rejected on a standardized screen, the percent of applicants who were subjected to a holistic review, the percent of unfilled positions, and the percent of applicants who are graduating US allopathic seniors. All outcome, predictor, and covariate variables were obtained using the average 2018, 2020, and 2021 values from the NRMP-PD surveys and reported ERAS and AAMC match statistics.

Statistical analysis

SAS 9.4 was used for all data management and statistical analysis. Linear and Logistic regression were used as our primary means of statistical inference testing. The Chi-square test and Fisher’s exact test assessed statistical significance between categorical independent variables with a binary dependent variable. The student’s t-test assessed the association between 2 variables with continuous dependent variables. Statistical significance was defined as α<0.05, with all p-values reported as two-tailed. This study was 80% powered to detect a full model (3 variables) r2 = 0.62 and a reduced model (1 variable) r2 = 0.31.

Confounding was assessed through statistical analysis between the covariates and both the predictor and outcome variables based on the statistical tests described above. Multivariable regression and weighted analysis were performed for variables that were significantly associated with both the predictor and outcome variable(s).

Results

Demographics and exploratory analysis

In Table 1, we present demographic information related to the residency specialties included in this study, as well as classifications of specialty type (procedural or cognitive). USMLE Step 1 3-digit scores were used by 62.7% of programs when selecting candidates for interviews, whereas 35.5% of programs reported using USMLE Step 1 as a pass/fail measure. Orthopedic Surgery (91%) and Plastic Surgery (87%) were the specialties most likely to use USMLE Step 1 as a 3-digit score, whereas Family Medicine (69%) and Psychiatry (66%) were the specialties most likely to use USMLE Step 1 as pass/fail.

Use of 3-digit USMLE Step 1 score cutoff

Specialty type was found to be the strongest predictor of how USMLE Step 1 is used in selecting interview candidates (p < 0.001) (utilizing multivariable regression model building). In comparison to cognitive specialties, procedural specialties were significantly more likely to use the USMLE Step 1 3-digit score when selecting candidates for interviews (Figure 1).

Figure 1.The average residency program use of the 3-digit USMLE Step 1 score, stratified by specialty type (procedural vs cognitive), when selecting a candidate for an interview from years 2018-2021

Data was compiled from the average NRMP-Program Director survey responses for participating programs. Procedural specialties are significantly more likely to use the 3-digit USMLE Step 1 score when selecting candidates for interview (mean = 79.39%, SD 10.05) as opposed to cognitive specialties (mean = 55.05%, SD 14.02); **p <0.001.

We found no association between how a specialty uses USMLE Step 1 and the average number of applications received per program specialty (p = 0.52). Additionally, there was no association between use of USMLE Step 1 and the number of candidates interviewed (p = 0.37), number of programs (p = 0.16) or number of positions offered (p = 0.31), the average number of applicants per position offered (p = 0.35), or the applicants volunteer experience (p = 0.63).

Use of USMLE Step 1 and valued features of residency applications in interview selection

Figure 2a summarizes the combined reporting regarding what program directors consider to be the most and least important considerations when deciding who to interview. Figure 2b summarizes the most and least important aspects of the residency application based on whether a residency program uses USMLE Step 1 as a 3-digit score or as pass/fail when selecting a candidate for an interview. Specialties that use the 3-digit USMLE score (procedural specialty being the strongest predictor) were significantly more likely to report valuing USMLE Step 1 when selecting candidates for interview (p < 0.001). Additionally, these programs were significantly more likely to value the candidates research experience (p < 0.001), applicants who are apart of AOA (p < 0.001), perceived leadership abilities (p = 0.04) and receiving honors in required clerkships (p = 0.05).

Figure 2a.The average reported importance for various components of the residency application when selecting a candidate for interview.

Data was compiled from the average NRMP-Program Director survey responses for participating programs during the years 2018-2021.

Figure 2b.The average reported importance of various aspects of the residency application when selecting candidates for interviews stratified by program use of USMLE Step 1 as either a 3-digit score or pass/fail in selecting candidates for interviews.

Data was compiled from the average NRMP-Program Director survey responses for participating programs during the years 2018-2021. Black represents programs that are more likely to use USMLE Step 1 as a 3-digit score (>70%) when selecting candidates for interview, where gray represents programs that are less likely to use USMLE Step 1 3-digit score (<50%) and more likely to use it as pass/fail (>50%) when selecting a candidate for interview.

Specialties that use USMLE score as pass/fail (more likely to be cognitive specialties) were significantly more likely to report valuing a failed USMLE Step 1 attempt when selecting candidates for interview (p < 0.001). Additionally, specialties that use USMLE Step 1 as pass/fail were significantly more likely to value USMLE Step 2CK (p = 0.005), CS (p <0.001), and Step 3 (p < 0.001). These specialties are also significantly more likely to value the Medical Student Performance Evaluation (p <0.001), the language proficiency of the applicant (p < 0.001), no gaps during medical school training (p < 0.001), perceived applicant commitment to the specialty (p < 0.001), NRMP flag (p = 0.004), and the applicant’s personal statement (p = 0.05).

Figure 3 illustrates the percent change (2018 to 2021) in the reported importance of the various elements of an application when choosing a candidate for an interview, stratified by the use of USMLE Step 1. The percent change in the reported value of the USMLE Step 2CK differed significantly between the two groups (p=0.006).

Figure 3.Percentage change in the reported importance of various aspects of residency applications from 2018 to 2021 stratified by use of USMLE Step 1 as either a 3-digit score or pass/fail in selecting candidates for interviews.

Data was compiled from the average NRMP-Program Director survey responses for participating programs. Black represents programs that are more likely to use USMLE Step 1 as a 3-digit score (>70%) when selecting candidates for interview, where gray represents programs that are less likely to use USMLE Step 1 3-digit score (<50%) and more likely to use it as pass/fail (>50%) when selecting a candidate for interview.

Discussion

This study focused on the evaluation of USMLE Step 1 scores as a screening tool for residency interviews. Our findings suggest that the specialty type significantly influences the application of USMLE Step 1 scores in selecting interview candidates. Specifically, procedural specialties tend to use the 3-digit score, whereas cognitive specialties favor the pass/fail approach. Importantly, the use of USMLE Step 1 was found to be unaffected by the number of applicants.

Our data further suggests that the importance assigned to various aspects of the application varies across specialties during candidate selection for interviews. Specialties utilizing the 3-digit USMLE Step 1 score were observed to place a greater emphasis on USMLE Step 1, AOA, research experience, perceived leadership, and clerkship honors. In contrast, specialties that employ USMLE Step 1 as a pass/fail criterion tend to focus more on the results of additional USMLE exams in the selection process. This study thereby offers valuable insights into potential alternatives to USMLE Step 1 as a selection criterion for residency interviews.

This study explores the implications of the transition to pass/fail reporting for USMLE Step 1 scores, positing an increased importance of USMLE Step 2CK in the selection of future residents. In a 2021 webinar series by the International Association of Medical Science Educators (IAMSE), experts discussed the impact of this transition, with many audience members and experts indicating a shift towards USMLE Step 2CK as the primary focus of preclinical curricula.¹²

Our research supports the premise that USMLE Step 2CK will play a more prominent role in the selection of interview candidates once access to the USMLE Step 1 3-digit score is eliminated. This conclusion is based on two key findings: 1) programs using USMLE Step 1 as pass/fail (which will soon be all programs) greatly value USMLE Step 2CK scores during interview selection, and 2) programs traditionally employing the 3-digit USMLE Step 1 score report an increasing relevance of Step 2CK.¹² As such, prospective applicants should be encouraged to complete the USMLE Step 2CK exam prior to applying, and medical schools may want to mandate early USMLE Step 2CK testing to better prepare their students for upcoming interviews.

This study aims to provide clarity regarding the attributes that will be crucial for residency interview consideration in future cycles. Given the historical significance of USMLE Step 1 as a screening tool, this topic has been hotly debated. In 2022, Dr. Katherine Chretien and the AAMC provided guidance for navigating the post-USMLE Step 1 score education system, emphasizing the importance of students pursuing their genuine interests during medical school.¹³ However, most medical students still seek recommendations for extracurricular activities that are both realistic and relevant.¹⁴

Research is a well-documented and invaluable component of the medical student experience.¹⁵ Quality research experience can provide substantial benefits, such as meaningful letters of support, enhanced organizational and critical thinking skills, and crucial personal development.¹⁶ Our study suggests that research will become more crucial in residency interview selection following the transition of USMLE Step 1 to pass/fail. Programs that are more likely to use the 3-digit Step 1 score significantly value research experience when selecting candidates for interviews. Furthermore, these programs have shown an increasing value of research experience from 2018 to 2021.¹⁶ Despite this, barriers such as time constraints, inadequate mentorship, and uncertainty about opportunities can hinder student participation in research.¹⁷ We propose that medical school curriculums should place a greater emphasis on research education and training.

Our study also reveals that being a procedural specialty strongly predicts the use of USMLE Step 1 as a 3-digit score for interview selection. Furthermore, procedural specialties value leadership experience, research experience, and AOA membership more highly than cognitive specialties when selecting interview candidates. Previous research, such as that conducted by Dr. Steven Stain, has identified these factors as predictors of higher ranking in general surgery programs.¹⁸ Similar results have been found in other surgical specialties.^19,20 While these factors are likely to become increasingly important for all applicants, medical students applying to procedural specialties should receive explicit instruction on the importance of leadership, research, and honor societies.

This study does come with certain limitations that need to be acknowledged. Primarily, our research employs a cross-sectional design, which, while useful for identifying associations, is restricted in its predictive capabilities. Additionally, our reliance on public data sets, not originally intended for this study, may raise concerns about internal validity. However, the strong correlation found between the use of USMLE Step 1 as a 3-digit score or pass/fail and the reported importance of USMLE Step 1 and any failed USMLE attempt respectively, does suggest an adequate level of internal validity. A further limitation is the low response rate of 25% for the NRMP PD survey, which could introduce potential selection bias. However, by consolidating three years of survey data, we facilitated a longitudinal analysis and increased the possibility of a more diverse respondent set, thereby enhancing the study’s generalizability. Despite the low response rate, the data includes a significant number of programs, averaging 50 programs per specialty per year, which further bolsters our confidence in the study’s generalizability.

Conclusion

It is reasonable to expect that the new USMLE Step 1 score reporting policy will alter the resident selection process, as most program directors consider Step 1 the most critical factor when deciding whom to interview. This study provides insight into what aspects of the residency application may replace USMLE Step 1 as an interview selection criterion. Students should be encouraged to take Step 2CK earlier than in the past and advised concerning the likely increased importance of research, leadership qualities, and membership in AOA. Medical schools and students can use these results to assess the effects of the potential shift in values when program directors can no longer use the 3-digit USMLE Step 1 score during interview selection.

Corresponding author

Address correspondence to:
Stephen M. Kavic, MD
Professor of Surgery, University of Maryland School of Medicine
29 South Greene Street, GS 631
Baltimore, MD 21201
Tel: (410) 328-6187
Fax: (410) 328-5919
email: skavic@som.umaryland.edu

DECLARATIONS

Ethics approval and consent to participate: This study has been deemed exempt from Institutional Review Board (IRB) review as it exclusively utilizes publicly available data and does not involve human subjects. The exemption aligns with the ethical guidelines outlined in the Common Rule (45 CFR part 46), and the research has been conducted in strict adherence to all relevant ethical standards and regulations. Funding: No external funding was acquired for this study. Competing Interests: The authors affirm that they have no financial or non-financial competing interests pertaining to this research.