Comparison of formula and number-right scoring in undergraduate medical training: a Rasch model analysis

BMC Medical Education

Research Article

Dario Cecilio-Fernandes¹  Janke Cohen-Schotanus¹  René A. Tio¹  Harro Medema²  Carlos Fernando Collares³  Lambert Schuwirth⁴ 

[1] Center for Education Development and Research in Health Professions (CEDAR), University of Groningen and University Medical Center Groningen, Antonius Deusinglaan 1, FC40, 9713, Groningen, AV, The Netherlands;Department Business IT & Management, NHL University of Applied Sciences, Leeuwarden, Netherlands;Faculty of Health, Medicine and Life Sciences, Educational Development and Research, Maastricht University, Maastricht, Netherlands;Faculty of Health, Medicine and Life Sciences, Educational Development and Research, Maastricht University, Maastricht, Netherlands;Prideaux Centre for Research into Health Professions Education, Flinders University, Adelaide, Australia;

关键词: Assessment;    Multiple choice questions;    Formula scoring;    Number-right scoring;    Rasch model;    Reliability;    Validity;    Construct-irrelevant variance;   

DOI  :  10.1186/s12909-017-1051-8

 received in 2017-01-16, accepted in 2017-11-02,  发布年份 2017

来源: Springer

【 摘 要 】

BackgroundProgress testing is an assessment tool used to periodically assess all students at the end-of-curriculum level. Because students cannot know everything, it is important that they recognize their lack of knowledge. For that reason, the formula-scoring method has usually been used. However, where partial knowledge needs to be taken into account, the number-right scoring method is used. Research comparing both methods has yielded conflicting results. As far as we know, in all these studies, Classical Test Theory or Generalizability Theory was used to analyze the data. In contrast to these studies, we will explore the use of the Rasch model to compare both methods.MethodsA 2 × 2 crossover design was used in a study where 298 students from four medical schools participated. A sample of 200 previously used questions from the progress tests was selected. The data were analyzed using the Rasch model, which provides fit parameters, reliability coefficients, and response option analysis.ResultsThe fit parameters were in the optimal interval ranging from 0.50 to 1.50, and the means were around 1.00. The person and item reliability coefficients were higher in the number-right condition than in the formula-scoring condition. The response option analysis showed that the majority of dysfunctional items emerged in the formula-scoring condition.ConclusionsThe findings of this study support the use of number-right scoring over formula scoring. Rasch model analyses showed that tests with number-right scoring have better psychometric properties than formula scoring. However, choosing the appropriate scoring method should depend not only on psychometric properties but also on self-directed test-taking strategies and metacognitive skills.

【 授权许可】

CC BY   
© The Author(s). 2017