부산대학교 도서관

Purpose: Assessments during core surgical clerkships (CSC) frequently include the National Board of Medical Examiners’ (NBME) Clinical Subject (“shelf”) Exams, which may not reflect curriculum and are associated with factors external to the CSC. NBME Customized Assessment Service (CAS) allows for the creation of assessments from an NBME question pool. The aim of this study is to evaluate our experience creating and implementing a CSC summative assessment using CAS with goals of curricular alignment and reducing disparities for URM students.Methods: We created a 100-question CAS exam from 10 topics covered in our curriculum. We retrospectively compared our students’ CAS item performance with the published p score (percent of examinees answering item correctly on parent exam) and calculated each item’s discrimination index. CAS scores were compared with prior years’ shelf scores based on term, Internal Medicine (IM) clerkship timing, and student characteristics via univariate analysis.Results: Records of 248 medical students were reviewed, 56 taking CAS (57% female, 25% underrepresented in medicine [URM], median age 25). The average CAS score was 76.9 ± 7.68. The average shelf score was 70.4 ± 8.2. Of 100 items, 41 had an observed p score that was more than 0.1 away from the parent exam p score. 39% of items had a discrimination index of 0.2 or above. Performance on the CAS was not associated with gender, race, URM, term taken, or timing of the IM clerkship. Performance on the shelf varied with race (p = 0.009), term taken (p < 0.001), and IM timing (p < 0.001). Given a cutoff of 75% correct, 76.6% of CAS students met exam honors criteria vs 36.1% of shelf students.Conclusion: The NBME CAS provides an opportunity to more closely align assessment and curriculum, allowing for targeted curriculum development and data-informed modifications of assessment items. CSC student performance cannot be assumed by published p scores and setting pass and honors thresholds requires careful consideration. Our experience suggests CAS may improve observed disparities in exam scores for URM students.