Measurement precision at the cut score in medical multiple choice exams: Theory matters

Perspectives on Medical Education

article

Lahner, Felicitas-Maria¹  Schauber, Stefan³  Lörwald, Andrea Carolin¹  Kropf, Roger⁴  Guttormsen, Sissel¹  Fischer, Martin R.⁵  Huwendiek, Sören¹ 

[1] Institute for Medical Education, University of Bern;Department of Health Professions, University of Applied Sciences;Centre for Educational Measurement at the University of Oslo (CEMO) and Centre for Health Sciences Education, University of Oslo;Faculty of Medicine, University of Zurich;Institute for Medical Education, University Hospital

关键词: Multiple choice exams;    Measurement precision;    Reliability;    Conditional reliability;   

DOI  :  10.1007/s40037-020-00586-0

学科分类：医学（综合）

来源: Bohn Stafleu van Loghum B.V.

【 摘 要 】

Introduction In high-stakes assessment, the measurement precision of pass-fail decisions is of great importance. A concept for analyzing the measurement precision at the cut score is conditional reliability, which describes measurement precision for every score achieved in an exam. We compared conditional reliabilities in Classical Test Theory (CTT) and Item Response Theory (IRT) with a special focus on the cut score and potential factors influencing conditional reliability at the cut score. Methods We analyzed 32 multiple-choice exams from three Swiss medical schools comparing conditional reliability at the cut score in IRT and CCT. Additionally, we analyzed potential influencing factors such as the range of examinees’ performance, year of study, and number of items using multiple regression. Results In CTT, conditional reliability was highest for very low and very high scores, whereas examinees with medium scores showed low conditional reliabilities. In IRT, the maximum conditional reliability was in the middle of the scale. Therefore, conditional reliability at the cut score was significantly higher in IRT compared with CTT. It was influenced by the range of examinees’ performance and number of items. This influence was more pronounced in CTT. Discussion We found that conditional reliability shows inverse distributions and conclusions regarding the measurement precision at the cut score depending on the theory used. As the use of IRT seems to be more appropriate for criterion-oriented standard setting in the framework of competency-based medical education, our findings might have practical implications for the design and quality assurance of medical education assessments.

【 授权许可】

CC BY