Toward a better judgment of item relevance in progress testing

BMC Medical Education

Research Article

Arno M. M. Muijtjens¹  Xandra M. C. Janssen-Brandt²  Dominique M. A. Sluijsmans² 

[1] Department of Educational Development and Research, Maastricht University, Universiteitssingel 60, 6229, Maastricht, ER, The Netherlands;Faculty of Health, Zuyd University of Applied Sciences, Nieuw Eyckholt 300, 6419, Heerlen, DJ, The Netherlands;

关键词: Item relevance;    Midwifery education;    Progress testing;    Rubric;   

DOI  :  10.1186/s12909-017-0989-x

 received in 2017-01-27, accepted in 2017-08-27,  发布年份 2017

来源: Springer

【 摘 要 】

BackgroundItems must be relevant to ensure item quality and test validity. Since “item relevance” has not been operationalized yet, we developed a rubric to define it. This study explores the influence of this rubric on the assessment of item relevance and on inter-rater agreement.MethodsMembers of the item review committee (RC) and students, teachers, and alumni (STA) reassessed the relevance of 50 previously used progress test (PT) items and decided about their inclusion using a 5-criteria rubric. Data were analyzed at item level using paired samples t-tests, Intraclass Correlation Coefficients (ICC), and linear regression analysis, and at rater level in a generalizability analysis per group.ResultsThe proportion of items that the RC judged relevant enough to be included decreased substantially from 1.00 to 0.72 (p < 0.001). Agreement between the RC and STA was high, with an ICC of >0.7 across items. The relation between inclusion and relevance was strong (correlation = 0.89, p < 0.001), and did not differ between RC and STA. To achieve an acceptable inter-rater reliability for relevance and inclusion, 6 members must serve on the RC.ConclusionsUse of the rubric results in a stricter evaluation of items’ appropriateness for inclusion in the PT and facilitates agreement between the RC and other stakeholders. Hence, it may help increase the acceptability and validity of the PT.

【 授权许可】

CC BY   
© The Author(s). 2017