Medical licensing boards in the United States and its territories aim to protect public health by establishing and maintaining standards of training and competence for physicians. Regulating the physician workforce starts with medical licensure, which requires that candidates successfully complete educational requirements and a medical licensing examination series. The United States Medical Licensing Examination (USMLE) comprises a series of assessments required for the licensure of U.S. MD-trained graduates as well as those who complete their training internationally. Several prior studies have documented the relationship between licensing examinations and various indicators of competence, yet, to our knowledge, few have established a relationship with the outcomes of care.^1–9 Demonstration of such a link is desirable for a licensing process that seeks to protect the health of the public. Thus, the purpose of this study was to investigate whether physician USMLE performance was related to adjusted patient in-hospital mortality and length of stay (LOS) for 5 predetermined common medical conditions.

The USMLE is composed of 3 examinations. Performance above a minimum passing score is required for an MD-trained physician to be eligible for an unrestricted license to practice medicine in the United States and its territories. The Step 1 examination is an assessment of the candidate’s ability to apply foundational science knowledge to medical practice. The Step 2 Clinical Knowledge (CK) examination targets an individual’s ability to apply the medical knowledge and clinical science essential for providing supervised patient care. The Step 3 examination assesses a candidate’s ability to provide unsupervised care, especially in ambulatory settings. These 3 examinations have been part of the licensure sequence since 1992, with 56% of 1,018,776 actively licensed physicians in the United States having taken all or part of the USMLE sequence (the remainder being those who are licensed by presenting prior regulatory exams for licensure such as the National Board of Medical Examiners [NBME] “Parts” or FLEX [Federation Licensing Examination] exams, or osteopathic physicians who did not take any USMLE Step examinations).¹⁰

Prior studies have documented the relationship between the number of test attempts and scores on licensing examinations with other markers of physician competence.^2–6 These include demonstrated associations between USMLE performance and specialty board examination performance, clinical performance evaluations, and subsequent disciplinary actions in practice. Moreover, past studies have demonstrated a relationship between licensing examination scores and process of care measures including proper prescribing practices and adherence to preventive health screening guidelines.^7–9 Specific to the USMLE, Norcini and colleagues demonstrated that, among patients with congestive heart failure (HF) or acute myocardial infarction (AMI) who were cared for by international medical graduate (IMG) attending physicians, higher provider USMLE Step 2 CK scores were associated with lower patient mortality. Researchers determined that an increase of 1 standard deviation on the USMLE Step 2 CK score scale correlated with a 4% decrease in relative risk for patient mortality.¹⁰

Studies of this type have the potential to add to the overall validity argument for the use of high-stakes examinations in medical regulation. Furthermore, as part of the recent exploration of potential changes to USMLE Step 1 score reporting, a deficit has been acknowledged in the body of work seeking associations between licensing examination performance and residency performance and clinical practice outcomes. A recommendation stemming from the related 2019 Invitational Conference on USMLE Scoring (InCUS) was to “accelerate research on the correlation of USMLE performance to measures of residency performance and clinical practice.”¹¹

In response to the InCUS recommendation, the purpose of this study was to investigate whether physicians’ USMLE performance was related to adjusted in-hospital mortality and LOS for patients hospitalized in Pennsylvania with a primary diagnosis of 1 of 5 predetermined conditions: HF, AMI, stroke, pneumonia, or chronic obstructive pulmonary disease (COPD). Our investigation seeks to add to the body of evidence examining the validity of the USMLE program.

Method

This study followed the Strengthening the Reporting of Observational Studies in Epidemiology (STROBE) reporting guidelines.¹² The Institutional Review Board at the American Institute for Research (number EX00533) approved it as exempt.

Sources of data

Data for this study came from the Pennsylvania Health Care Cost Containment Council (PHC4), the American Medical Association (AMA) Physician Masterfile, and the examination score records of the USMLE program. These 3 data sources were linked through a series of matches based on the National Provider Identifier (NPI) number, name, self-reported sex, and birth year.

All hospitals in Pennsylvania must send data to the PHC4 each time a patient is discharged. Patient demographic characteristics, primary and secondary diagnoses, discharge status, LOS, and hospital where care was provided are included. The attending physician, defined by PHC4 as the individual who has overall responsibility for the medical care and treatment of the patient, is typically also identified by the hospital. PHC4 applies validation and editing procedures to patient data, which can be corrected by the hospitals. In this study, we analyzed hospitalizations for AMI, HF, pneumonia, COPD, and stroke from January 1, 2017 to December 31, 2019. We chose these conditions because they have been used previously as a means for judging the quality of patient care.¹³

The AMA Physician Masterfile was the primary source of information on practicing physicians. For this study, we limited analysis to self-identified family medicine and general internal medicine physicians (excluding self-identified hospitalists because their numbers were small and their training and certification backgrounds were heterogeneous) who were the attending physicians for patients with 1 or more of the 5 conditions in the PHC4 database. We did so because these physicians provide considerable care for these conditions and it prevented the results from being confounded by subspecialization. Providers’ initial board certification was also available, and we excluded subspecialists and physicians who held certification in anything other than internal medicine or family medicine.

We analyzed USMLE scores on physicians’ first attempt at Step 1, Step 2 CK, and Step 3 of the examination sequence (we excluded the Step 2 Clinical Skills examination since it is no longer required), but only for those physicians who had taken and passed all 3 steps. During the study period, USMLE Step 3 eligibility required passing Step 1 and Step 2 CK. In addition, physician sex and country of medical school were accessed.

Data elements

For patients, we extracted age, sex, race/ethnicity, principal and secondary diagnoses, discharge status, LOS, home county, and hospital. A modified version of the Charlson Comorbidity Index was developed as a measure of the sickness of patients.¹⁴ In addition, we combined race and ethnicity data to a single variable with White, non-Hispanic patients in 1 group and all remaining participants in another. This was done because some race or ethnicity groups were too small for analysis. For each hospitalization, we created an index of whether the patient’s home county was rural by reference to a list maintained by the state of Pennsylvania.¹⁵

We used discharge status to determine whether the patient died while in the hospital. For LOS analyses, we removed patients who died in hospital (n = 13,198), were transferred to another short-term inpatient facility, had missing LOS data, or had extreme values for LOS (0 days or more than 40 days). To reduce the effects of outliers, we calculated the logarithm of the LOS data.

For attending physicians, we knew their self-designated specialty, whether they were U.S. medical graduates (USMGs) or IMGs, their sex, and whether they had initial board certification in family medicine or internal medicine. In addition, we calculated each individual physician’s patient volume by counting the number of patients with any of the 5 conditions the physician treated during the study timeframe. In previous work, these characteristics have been found to be associated with patient outcomes, which we wanted to control for in this study.^16–18

Scores across Steps 1, 2, and 3 are very highly correlated. To avoid the problems of multicollinearity, we developed a composite measure of USMLE performance for use in the analyses. First-time scores on each of the 3 steps of USMLE were on a scale that had a mean of 200 (standard deviation [SD] = 20) in an archival reference group. We converted these to z scores (mean = 0; SD = 1) and averaged them to develop a composite measure of USMLE performance for each physician.¹⁹

We also calculated the number of patients with any of the 5 conditions treated at the hospital by the study physicians (i.e., facility patient volume).

Analysis

We began by calculating descriptive statistics for hospitalizations of the total patient population and for those patients with each condition. To determine the adjusted relationship between average USMLE score and patient mortality, we applied a multivariate logistic regression model adjusted for the following:

• comorbidity index;
• condition (reference: AMI);
• patient age;
• patient sex (reference: male);
• patient race/ethnicity (reference: White/non-Hispanic);
• patient location (reference: nonrural);
• whether the physician was an IMG (reference: USMG);
• self-designated specialty (reference: family medicine);
• physician sex (reference: female);
• specialty board certification (with not certified as reference);
• physician volume; and
• number of hospitalizations in the institution for the studied conditions.

We applied a similar model to determine the adjusted relationship between USMLE average score and log LOS with a reduced number of hospitalizations as described above. To adjust for the clustering of patients within physicians and physicians within hospitals, we used generalized estimating equations. Statistical analyses were conducted in SAS software, version 7.15 (SAS Institute, Cary, NC).

We undertook an analysis to explore the possibility that physicians with higher scores on USMLE also worked at hospitals with better patient outcomes. If these 2 variables were confounded, we might mistakenly attribute patient outcomes to USMLE performance rather than to hospitals. To address this issue, we correlated the mean USMLE scores for hospitals with their mortality rates. The correlation was −0.05 (P = .54) and not statistically significant.

We were also concerned that there might be interactions between patients’ age and race/ethnicity, as well as between physicians’ specialization and board certification. Running our models with these interactions included did not alter our findings.

Results

Table 1 presents data for the 196,881 hospitalizations in the study, stratified by primary diagnosis. Hospitalizations are further broken down by patient and physician characteristics as well as facility volume. Most of the hospitalizations were for females (n = 100,075, 50.8%), White, non-Hispanics (n = 158,936, 80.7%), and those living in nonrural locations (n = 137,709, 70.6%). Patients had a mean LOS of 4.86 days (SD = 5.16 days) and 1.7 (SD = 1.3) comorbid conditions; and 4,142 (2.1%) hospitalizations resulted in death. Of all hospitalizations, 166,357 (84.5%) were managed by internists, 187,336 (95.2%) were managed by board-certified doctors, 132,087 (67.1%) were managed by male physicians, and 136,620 (69.4%) were managed by USMGs.

There were 1,765 attending physicians for these hospitalizations, 1,316 (75%) were internists, 731 (41%) were female, 1,009 (57%) were USMGs, and 1,663 (94%) were board certified in their self-designated specialty. Mean physician volume through the period of study was 11.2 (SD = 12.8) hospitalizations, and mean converted USMLE score was 0.64 (SD = 1.1). The hospitalizations took place at 171 institutions, with a mean hospital volume of 2,379 (SD = 2,393).

Table 2 presents results of the multivariate analysis with in-hospital mortality as the dependent measure and characteristics of the patients, physicians, and facilities as the covariates. Age, race/ethnicity, and rural location were among the patient demographic characteristics that had statistically significant associations with mortality, as did physician sex and patient volume. After adjustment, higher scores on USMLE were also associated with lower patient mortality. The adjusted odds ratio was 0.94 (95% confidence interval [CI] = 0.90, 0.99; P < .02). Each standard deviation increase in the converted score was associated with a 5.51% reduction in the odds of in-hospital mortality.

Table 3 presents the results of the multivariate analysis with log LOS as the dependent measure and characteristics of the patients, physicians, and facilities as the covariates. Sex, age, and race/ethnicity were among the patient characteristics that had statistically significant associations with log LOS, as did physician specialty and certification status and facility volume. After adjustment, higher USMLE scores were associated with shorter LOS. The adjusted estimate was 0.99 (95% CI = 0.98, 0.99; P < .001). Each standard deviation increase in the converted score was associated with a 1.34% reduction in log LOS.

Discussion

The aim of this study was to investigate whether the USMLE performance of physicians was related to the adjusted in-hospital mortality and LOS of their patients with a primary diagnosis of HF, AMI, stroke, pneumonia, or COPD. In terms of mortality, each increase of 1 standard deviation in converted USMLE scores was associated with a 5.51% decline in the odds for mortality. This is comparable to findings from earlier research that focused only on IMGs.⁹ If replicated, our findings may be of clinical significance when compared to the magnitude of the effects of recommended medical treatments such as low-dose aspirin in the secondary prevention of cardiovascular and cerebrovascular events (18%).²⁰ In a normal distribution of scores, the worst performers are conservatively 4 standard deviations below the best performers, which would translate to a 22.04% difference in odds for the mortality of their patients (i.e., 4 × 5.51%).

Similarly, shorter lengths of stay were associated with higher USMLE scores. Each increase of 1 standard deviation in converted USMLE scores was associated with a 1.34% decrease in log LOS. This translates to a reduction of less than 2 hours for the mean LOS (4.86 days). Although it is statistically significant, even at the extremes, this is unlikely to be of practical importance.

It is noteworthy that our findings likely understate the true relationship between USMLE and practice performance. Those physicians for whom the USMLE is a licensure requirement must meet or exceed the passing scores for all 3 steps to practice (and therefore have patient outcomes); those who were ultimately unsuccessful were not included in the study. This restricts the range of scores we could analyze and attenuates the magnitude of the relationships we report.

This study has several limitations. It is important to note that in a retrospective observational study, we can establish an association, but not causality. However, we adjusted for the number of patient, physician, and hospital characteristics known to be related to outcomes, tested for interactions among several of them, and found that unmeasured hospital effects did not significantly influence our findings. Nonetheless, there may be factors that are not included in our study but that are correlated with both USMLE scores and practice performance that influenced the results. There may have also been variability across hospitals in terms of the process for identifying the attending physician. Finally, we confined our analyses to 5 common hospital conditions in 1 U.S. state. Broader sampling of conditions, sites of care, and locations are needed to increase confidence in our findings.

Despite these limitations, it is incumbent upon assessment organizations and licensing and certification bodies to generate evidence that their examinations are associated with relevant outcomes—optimally, those that relate to patients, such as process of care measures or patient health or morbidity and mortality. Doing so helps to build a broader validity argument for the information that results from a single assessment or system of assessment. In the specific case of this study, our results provide some support for the component of Kane’s validity framework known as extrapolation, or the extent to which scores reflect real-world performance.^21,22 Until recently, the ability to develop such a body of evidence for the USMLE program has been limited in part by the fact that practice data were generally unavailable in the quality and quantity needed for analysis. Such data are now accessible, and guidelines for analyzing and reporting observational research studies based on them have been strengthened. Future research is needed to address the limitations of this study noted above, as well as to look more closely at the separate steps in the USMLE process to further contribute to validity arguments.

This study demonstrates the ongoing and important role of standardized assessments for higher-stakes decision-making and medical regulation and, along with other evaluations, adds to the inferences we can make about trainees and practicing physicians based on program participation and performance. However, given the many factors that ultimately influence patient outcomes, we must design overall systems to include multiple types of content and assessment to make broader, more comprehensive inferences about trainees and examinees and to more fully support validity arguments. This would include other structured assessments of performance as well as authentic workplace-based assessment. An ideal assessment system would include an educational system focused on outcomes, leveraging assessment results for feedback and performance improvement, complemented by high-stakes examinations with validity evidence related to patient outcomes. This approach would benefit learners and would be optimal for patients, particularly since unrestricted licensure is one of the last stages of physician regulation prior to providing patient care without supervision.