Implementing a cognitive diagnostic assessment in an institutional test: a new networking model in language testing and experiment with a new psychometric model and task type

cognitive diagnostic assessment;cognitive diagnostic model comparison;general modeling framework;language test data;Test of English as a Foreign Language (TOEFL) Reading and Listening;university French placement test;elicited imitation task;resampling;polytomous;networking in language testing;Examination for the Certificate of Proficiency in English (ECPE) Grammar

Yi, Yeonsook

关键词: cognitive diagnostic assessment;    cognitive diagnostic model comparison;    general modeling framework;    language test data;    Test of English as a Foreign Language (TOEFL) Reading and Listening;    university French placement test;    elicited imitation task;    resampling;    polytomous;    networking in language testing;    Examination for the Certificate of Proficiency in English (ECPE) Grammar;   

Others  :  https://www.ideals.illinois.edu/bitstream/handle/2142/42124/Yeonsook_Yi.pdf?sequence=1&isAllowed=y

美国|英语

来源: The Illinois Digital Environment for Access to Learning and Scholarship

【 摘 要 】

This dissertation is based on two major projects, cognitive diagnostic model comparison andimplementing cognitive diagnostic assessment in an institutional test. In the first project, fivecognitive diagnostic models are empirically compared for language test data under a unifiedgeneral modeling framework. The models are applied to three sets of data, TOEFL Reading,TOEFL Listening and ECPE Grammar and examined in terms of their model fit to the data andfunctioning. The item-association root mean squared error values and multiple informationcriteria all indicate that the general model (LCDM) and the compensatory-RUM model are thebest fit to all three test data sets used. The functioning of the models examined through multipleindices also unanimously confirms these fit indices. Based on these results, a discussion followsto argue that the general modeling framework is optimal for language assessment data due to itsmuch greater flexibility. The behaviors of the compensatory RUM and non-compensatory RUM(Fusion) models are also compared and the differences are analyzed.Cognitive diagnostic assessment (CDA) has gained attention in language testing sincelate 90’s. A few models of this new assessment method have been applied to the response data ofdifferent language tests, yielding encouraging results in general. Most of these empirical studiesused large-scale, standardized tests and retrofitted to these existing tests.The second research study in this proposal was an effort to go beyond this limitedresearch context of previous studies of CDA in language testing. Using a new psychometriccognitive diagnostic model (Log-linear cognitive diagnostic model) and task type (elicitedimitation task) as well as constructing the EIT test with a CDA implementation in mind from theoutset (thus not retrofitting), this project tried to implement a cognitive diagnostic assessment inan institutional placement test. In doing so, the study employed a statistical method (resamplingtechnique) as a way to resolve the issue caused by a small size of a test-taker pool at aninstitution, which is usually smaller than the optimal size for CDA implementation. It also triedto analyze polytomously scored response data.The study achieved a success at estimating with polytomous response data that werescored with a three-point scale (i.e., zero to two points). Though it was a limited success (in thatmore complex rating scale could not be analyzed), it was the first success in estimating withpolytomous response data in the context of CDA research in language testing.The analysis results of the study also provide many insights and implications for theprocess of Q-matrix construction, grain size of attributes, appropriate task types and item typesfor cognitive diagnostic assessment as well as appropriate cognitive psychometric models fordiffering contexts of CDA implementation. The study also rediscovers or introduces theusefulness of bootstrap resampling method as an approach that is gaining popularity even inareas where only traditional quantitative methods are usually employed.Also, the networking occurred in this project between students in differentspecializations could be established as a new networking model in language testing. Consideringsuch a collaboration is very much needed for implementing a relatively new measurementmethod in a specific knowledge domain, the co-work attempted in this project could serve as amodel for implementing CDA in language testing.