Impact of composite population priors on computer adaptive test proficiency estimates

Computer adaptive testing;Testing;Estimation;IRT;Collateral information

Morrison, Kristin M. ; Embretson, Susan E. Psychology Roberts, James S. Spieler, Daniel Templin, Jonathan Thomas, Rick ; Embretson, Susan E.

University:Georgia Institute of Technology

Department:Psychology

关键词: Computer adaptive testing;    Testing;    Estimation;    IRT;    Collateral information;   

Others  :  https://smartech.gatech.edu/bitstream/1853/58555/1/MORRISON-DISSERTATION-2017.pdf

美国|英语

来源: SMARTech Repository

【 摘 要 】

Testing has existed for thousands of years and has evolved from all examinees receiving the same test to adaptive testing, in which the test is tailored to the individual examinee. These adaptive testing designs have shown to be improvements over fixed-length, conventional tests in terms of proficiency measurement, reduced testing time, and faster scoring. A popular method of test administration is computer adaptive testing (CAT) using expected a posteriori (EAP) estimation. This Bayesian estimation approach utilizes previous information known about the examinee to obtain more precise estimates of the individual’s ability. An appropriate prior will generally increase estimation precision, decrease outlier influences, and provide an estimate for all possible response patterns. An inappropriate prior, however, may result in biased estimates (Embretson & Reise, 2000). Previous studies have used collateral information (i.e., additional information) concerning the examinee, such as demographic variables, age, grade, or previous test scores, to aid in estimation. Several studies have used previous test scores (Matteucci & Veldkamp, 2013; Veldkamp & Matteucci, 2013; van der Linden, 1999), but none have looked directly at priors based on group membership. This study examined the influence of various group priors, such as composite priors (i.e., priors created from combining groups) and individual priors (i.e., priors specific to the group), on estimation in CAT designs. Results of the study show group-specific priors perform best; however, it is impossible to know true group membership. Thus, results of the study support the use of priors based on the population because priors based on demographics may adversely impact some high ability groups.