How accurate are the Step 1 NBMEs for predicting your Score

In medical education in the United States, very few assessments hold as much weight and anticipation as the United States Medical Licensing Examination (USMLE) Steps 1 and 2. As a pivotal milestone for medical students on their journey to becoming licensed physicians, and probably the most challenging obstacle, the exam's outcome significantly influences residency placements and future career paths. In the journey of Step 1 preparation, students often rely on practice tests, particularly those offered by the National Board of Medical Examiners (NBME), to gauge their readiness and predict their actual Step 1 score. However, the question remains: How accurate are these NBME practice tests in predicting one's final Step 1 score?


Understanding the NBMEs and their Role:
The National Board of Medical Examiners (NBME) is a non-profit that designs and administers multiple standardized tests for medical students and medical professionals, which are widely recognized and used across various stages of medical education in the United States, from shelf exams, Steps 1 and 2, to residency board examinations.
Regarding Step 1 in specific, the NBME offers a set of practice exams designed to replicate the actual testing experience. These exams, known colloquially as NBMEs, are considered by most students as vital tools in their preparation arsenal, using anywhere between 1 to all available assessments. The self-assessments that the NBME offers have been updated in 2020, and the older versions of these assessments were phased out. According to their official announcement on their website: "Starting April 3, 2020, the NBME will offer a new web-based format for the Step 1 and Step 2 CK examinations."


Currently Available NBMEs for Step 1
The NBME offers several ‘forms’ of practice exams specifically tailored for Step 1 prep. Each form is structured to simulate the content and structure of the actual Step 1 exam, providing students with a realistic assessment of their knowledge and test-taking skills. The forms are periodically updated to reflect changes in exam content and to ensure relevance to current medical practice standards. These practice tests, called self-assessment forms, can be found at https://www.nbme.org/examinees/self-assessments.

Quantifying Predictive Accuracy
The most important question for students relying on NBME practice tests is ‘how well do these assessments predict their actual Step 1 score?’. Over the years, various studies and analyses have attempted to quantify this predictive accuracy. Predictmystepscore.com, a popular resource among medical students, provides valuable insights into which NBME forms correlate most closely with actual Step 1 scores. Their statistics page features a table that ranks the predictive value of each NBME form based on extensive data analysis from student submissions.

Data and Analysis
The accuracy of NBMEs in predicting Step 1 scores rests, rather hinges, on robust data collection and analysis methodologies. A site such as Predictmystepscore.com aggregates data from thousands of students who voluntarily share their practice test scores and actual Step 1 outcomes. By comparing these scores, the website generates statistical correlations that highlight which NBME forms tend to provide the most reliable predictions.


The Main Factors Influencing The Predictive Accuracy of NBMEs

1. The Assessment’s Difficulty and Content: The similarity in difficulty and content between NBME practice tests and the actual Step 1 exam plays a huge role. The forms that closely mimic the real exam will naturally provide more accurate predictions than ‘easier’ forms.

2. Data Sample Sizes Analyzed: The larger the dataset used for analysis, the more reliable the predictions become. Sites such as predictmystepscore.com benefit from a substantial volume of data submissions, enhancing the statistical robustness of their predictive models.

3. Timing of Assessment: When students take NBME practice tests relative to their actual Step 1 exam date can impact predictive accuracy. Testing closer to the exam date tends to provide a more accurate reflection of readiness. In my experience, a form every week in the month leading up to your exam is the most predictive.

4. Exam Testing Conditions: Tying in with the above point, taking your self-assessments under exam conditions will be more predictive of your performance. For example, taking 2 forms back to back, taking timed breaks and starting at the time you would start your real exam at, makes for a more predicting score as it mimics the conditions under which you will take the USMLE step 1.



To conclude, NBME practice tests are quite valuable tools in predicting Step 1 scores, albeit with some varying degrees of accuracy depending on several factors as we’ve discussed. The availability of data-driven insights from platforms such as predictmystepscore.com empowers students to make informed decisions regarding which NBMEs to prioritize based on their predictive performance and accuracy. With time, these methodologies for assessing and predicting student performance will evolve. But by leveraging the resources available and understanding the nuances of predictive accuracy, (keeping in mind that this is only one component of a comprehensive study plan) medical students can approach Step 1 with greater confidence and readiness.

In the landscape of medical education, especially in the US, the role of these predictive assessments remains integral, serving as benchmarks for students like us, striving towards excellence in their medical careers.

Sources:
- Source: NBME Announcement on Changes to Step 1 and Step 2 CK Examinations