Psychological assessment of AI-based decision support systems: tool development and expected benefits

Frontiers in Artificial Intelligence

Artificial Intelligence

Sarah Hatfield¹  Katharina Buschmeyer¹  Julie Zenner² 

[1] Faculty of Business, Augsburg Technical University of Applied Science, Augsburg, Germany;Faculty of Liberal Arts and Science, Augsburg Technical University of Applied Science, Augsburg, Germany;

关键词: AI-based decision support systems;    work;    human-centered evaluation;    survey inventory;    system properties;    characteristics of the supported task;    psychological load;   

DOI  :  10.3389/frai.2023.1249322

 received in 2023-06-28, accepted in 2023-09-05,  发布年份 2023

来源: Frontiers

【 摘 要 】

This study aimed to develop an evaluation tool that assesses the use of AI-based decision support systems (DSSs) in professional practice from a human-centered perspective. Following the International Organization for Standardization, this perspective aims to ensure that the use of interactive technologies improves users' psychological load experience and behavior, e.g., in the form of reduced stress experience or increased performance. Concomitantly, this perspective attempts to proactively prevent or detect and correct the potential negative effects of these technologies on user load, such as impaired satisfaction and engagement, as early as possible. Based on this perspective, we developed and validated a questionnaire instrument, the Psychological Assessment of AI-based DSSs (PAAI), for the user-centered evaluation of the use of AI-based DSSs in practice. In particular, the instrument considers central design characteristics of AI-based DSSs and the corresponding work situation, which have a significant impact on users' psychological load. The instrument was tested in two independent studies. In Study 1, N = 223 individuals were recruited. Based on the results of item and scale analyses and an exploratory factor analysis, the newly developed instrument was refined, and the final version was tested using a confirmatory factor analysis. Findings showed acceptable-to-good fit indices, confirming the factorial validity of the PAAI. This was confirmed in a second study, which had N = 471 participants. Again, the CFA yielded acceptable-to-good fit indices. The validity was further confirmed using convergent and criterion validity analyses.

【 授权许可】

Unknown   
Copyright © 2023 Buschmeyer, Hatfield and Zenner.