Comparing different trend estimation approaches in country means and standard deviations in international large-scale assessment studies

Large-scale Assessments in Education

Methodology

Oliver Lüdtke¹  Alexander Robitzsch¹ 

[1] IPN − Leibniz Institute for Science and Mathematics Education, Olshausenstraße 62, 24118, Kiel, Germany;Centre for International Student Assessment (ZIB), Kiel, Germany;

关键词: Large-scale assessment;    Trend estimation;    Item response models;    Linking;    Differential item functioning;    Item parameter drift;    Original trend;    Marginal trend;   

DOI  :  10.1186/s40536-023-00176-6

 received in 2022-11-12, accepted in 2023-07-07,  发布年份 2023

来源: Springer

【 摘 要 】

One major aim of international large-scale assessments (ILSA) like PISA is to monitor changes in student performance over time. To accomplish this task, a set of common items (i.e., link items) is repeatedly administered in each assessment. Linking methods based on item response theory (IRT) models are used to align the results from the different assessments on a common scale. This work employs the one-parameter logistic (1PL) and the two-parameter logistic (2PL) IRT models as scaling models for dichotomous item response data. The present article discusses different types of trend estimates in country means and standard deviations for countries in ILSA. These types differ in three aspects. First, the trend can be assessed by an indirect or direct linking approach for linking a country’s performance at an international metric. Second, the linking for the trend estimation can rely on either all items or only the link items. Third, item parameters can be assumed to be invariant or noninvariant across countries. It is shown that the most often employed trend estimation methods of original trends and marginal trends can be conceived as particular cases of indirect and direct linking approaches, respectively. Through a simulation study and analytical derivations, it is demonstrated that trend estimates using a direct linking approach and those that rely on only link items outperformed alternatives for the 1PL model with uniform country differential item functioning (DIF) and the 2PL model with uniform and nonuniform country DIF. We also illustrated the performance of the different scaling models for assessing the PISA trend from PISA 2006 to PISA 2009 in the cognitive domains of reading, mathematics, and science. In this empirical application, linking errors based on jackknifing testlets were utilized that adequately quantify DIF effects in the uncertainty of trend estimates.

【 授权许可】

CC BY   
© The Author(s) 2023

【 图 表 】

Fig. 1

Fig. 11

Fig. 4

Fig. 1

Fig. 3

Fig. 2

Fig. 1

Fig. 4

Fig. 2

40798_2023_622_Article_IEq21.gif

40798_2023_622_Article_IEq20.gif

40798_2023_622_Article_IEq19.gif

Fig. 6

Fig. 9

Fig. 1

Fig. 5

40798_2023_622_Article_IEq15.gif

40517_2023_263_Article_IEq1.gif

Fig. 10

Fig. 5

Scheme 1

Fig. 3

40798_2023_622_Article_IEq9.gif

Fig. 2

Fig. 5

Fig. 4

12888_2023_4959_Article_IEq3.gif

Fig. 4

12888_2023_4959_Article_IEq1.gif

Fig. 8

Fig. 6

Fig. 4

Fig. 8

Fig. 11

Fig. 4

Fig. 12

Fig. 1

Fig. 3

Fig. 2

Fig. 11

Fig. 3

Fig. 1

Fig. 1

12938_2023_1137_Article_IEq68.gif

Fig. 2

40517_2023_267_Article_IEq8.gif

Fig. 10

Fig. 1

Fig. 1

Fig. 1

Fig. 7

Fig. 4

Fig. 3

Fig. 7

Fig. 3

Fig. 5

40798_2023_622_Article_IEq7.gif

40798_2023_622_Article_IEq6.gif

40798_2023_622_Article_IEq4.gif

Fig. 5

Fig. 9

40798_2023_622_Article_IEq1.gif

Fig. 2

41512_2023_153_Article_IEq116.gif

Fig. 2

Fig. 1

Fig. 3

Fig. 3

Fig. 4

Fig. 1

12938_2023_1137_Article_IEq32.gif

40517_2023_267_Article_IEq5.gif

40517_2023_267_Article_IEq4.gif

40517_2023_267_Article_IEq3.gif

Fig. 2

Fig. 3

Fig. 5

Fig. 8

Fig. 1

Fig. 1

Fig. 7

Fig. 3

Fig. 1

Fig. 3

41512_2023_153_Article_IEq101.gif

Fig. 2

Fig. 1

Fig. 2

Fig. 4

41512_2023_153_Article_IEq93.gif

Fig. 3

Fig. 1

Fig. 5

Fig. 5

Fig. 2

Fig. 4

Fig. 6

Fig. 3

41512_2023_153_Article_IEq73.gif

Fig. 2

Fig. 1

Fig. 4

Fig. 14

Fig. 2

41512_2023_153_Article_IEq65.gif

Fig. 3

Fig. 3

Fig. 1

Fig. 6

Fig. 1

Fig. 7.

Fig. 1

13750_2023_309_Article_IEq64.gif

Fig. 3

Fig. 6

Fig. 7

Fig. 4

Fig. 1

13570_2023_282_Article_IEq20.gif

13570_2023_282_Article_IEq19.gif

13570_2023_282_Article_IEq18.gif

13570_2023_282_Article_IEq15.gif

13570_2023_282_Article_IEq12.gif

13570_2023_282_Article_IEq8.gif