Impact of differential item functioning on group score reporting in the context of large-scale assessments

Large-scale Assessments in Education

Research

Frederic Robin¹  Usama Ali²  Hyo Jeong Shin³  Sean Joo⁴ 

[1] Educational Testing Service, New Jersey, USA;Educational Testing Service, New Jersey, USA;South Valley University, Qena, Egypt;Sogang University, Seoul, South Korea;University of Kansas, Kansas, USA;

关键词: Large-scale assessment;    Programme for International Student Assessment;    Differential item functioning;    Group score reporting;    Jackknife sampling;   

DOI  :  10.1186/s40536-022-00135-7

 received in 2022-01-08, accepted in 2022-09-26,  发布年份 2022

来源: Springer

【 摘 要 】

We investigated the potential impact of differential item functioning (DIF) on group-level mean and standard deviation estimates using empirical and simulated data in the context of large-scale assessment. For the empirical investigation, PISA 2018 cognitive domains (Reading, Mathematics, and Science) data were analyzed using Jackknife sampling to explore the impact of DIF on the country scores and their standard errors. We found that the countries that have a large number of DIF items tend to increase the difference of the country scores computed with and without the DIF adjustment. In addition, standard errors of the country score differences also increased with the number of DIF items. For the simulation study, we evaluated bias and root mean squared error (RMSE) of the group mean and standard deviation estimates using the multigroup item response theory (IRT) model to explore the extent to which DIF items create a bias of the group mean scores and how effectively the DIF adjustment corrects the bias under various conditions. We found that the DIF adjustment reduced the bias by 50% on average. The implications and limitations of the study are further discussed.

【 授权许可】

CC BY   
© The Author(s) 2022. corrected publication 2022

【 图 表 】

Fig. 1

13690_2022_1011_Article_IEq4.gif

13690_2022_1011_Article_IEq3.gif

13690_2022_1011_Article_IEq2.gif

13690_2022_1011_Article_IEq1.gif

Fig. 7

Fig. 3

Fig. 2

Fig. 1

Fig. 1

Fig. 1

40644_2022_507_Article_IEq1.gif

Fig. 3

Fig. 4

12936_2022_4386_Article_IEq84.gif

Fig. 2

Fig. 2

Fig. 1

Fig. 6

Fig. 1

12902_2022_1244_Article_IEq30.gif

Fig. 5

12902_2022_1244_Article_IEq28.gif

12902_2022_1244_Article_IEq27.gif

Fig. 4

12902_2022_1244_Article_IEq24.gif

12902_2022_1244_Article_IEq23.gif

12902_2022_1244_Article_IEq21.gif

12902_2022_1244_Article_IEq17.gif

12982_2022_119_Article_IEq170.gif