Modeling items for text comprehension assessment using confirmatory factor analysis

Frontiers in Psychology

article

Monika Tschense¹  Sebastian Wallot¹ 

[1] Department of Language and Literature, Max Planck Institute for Empirical Aesthetics;Research Group for Research Methods and Evaluation, Institute of Psychology, Leuphana University Lüneburg;Research Group for Neurocognition of Music and Language, Planck Institute for Empirical Aesthetics

关键词: reading;    text comprehension;    reading comprehension;    Comprehension assessment;    Discourse representation;    mental model;   

DOI  :  10.3389/fpsyg.2022.966347

学科分类：社会科学、人文和艺术（综合）

来源: Frontiers

【 摘 要 】

Reading is a complex cognitive task with the ultimate goal of comprehending the written input. For longer, connected text, readers generate a mental representation that serves as its basis. Due to limited cognitive resources, common models of discourse representation assume distinct processing levels, each relying on different processing mechanisms. Since Kintsch and van Dijk (1978) proposed three distinct levels – a local micro structure, a more global macro structure, and a situation model considering readers’ prior knowledge and inferences – ample empirical evidence pointed towards their individual importance for comprehension. However, only little research addressed all of these levels simultaneously. Moreover, current studies that tried to relate process measures of reading (e.g., reading times, eye movements) to comprehension did not consider comprehension as a multi-faceted, but rather a uni-dimensional construct, usually assessed with one-shot items. Thus, the first aim of this paper is to use confirmatory factor analysis (CFA) to test whether comprehension can be modelled as a uni- or multi-dimensional concept. The second aim is to investigate how well widely used one-shot items can be used to capture comprehension. 400 participants read one of three short stories of comparable length, linguistic characteristics, and complexity. Based on the evaluation of three independent raters per story, 16 wh-questions and 60 yes/no-statements were compiled in order to retrieve information at micro and inference level, and 16 main contents were extracted to capture information at the macro level in participants’ summaries. Still, only a fraction of these items showed satisfactory psychometric properties and factor loadings – a blatant result considering the common practice for item selection. For CFA, two models were set up that address text comprehension as either a one-dimensional construct (a uni-factor model with a single comprehension factor), or a three-dimensional construct reflecting the three distinct representational levels (three correlated first-order factors). Across stories and item types, model fit was consistently better for the three-factor model providing evidence for a multi-dimensional construct of text comprehension. Our results provide concrete guidance for the preparation of comprehension measurements in studies investigating the reading process.

【 授权许可】

CC BY