In models we trust: preregistration, large samples, and replication may not suffice

Frontiers in Psychology

Psychology

Pascal Jordan¹  Martin Spiess² 

[1] Institute of Psychology, Department of Psychology and Human Movement Science, University of Hamburg, Hamburg, Germany;null;

关键词: population;    sampling design;    non-response;    selectivity;    misspecification;    biased inference;    diagnostics;    robust methods;   

DOI  :  10.3389/fpsyg.2023.1266447

 received in 2023-07-25, accepted in 2023-09-04,  发布年份 2023

来源: Frontiers

【 摘 要 】

Despite discussions about the replicability of findings in psychological research, two issues have been largely ignored: selection mechanisms and model assumptions. Both topics address the same fundamental question: Does the chosen statistical analysis tool adequately model the data generation process? In this article, we address both issues and show, in a first step, that in the face of selective samples and contrary to common practice, the validity of inferences, even when based on experimental designs, can be claimed without further justification and adaptation of standard methods only in very specific situations. We then broaden our perspective to discuss consequences of violated assumptions in linear models in the context of psychological research in general and in generalized linear mixed models as used in item response theory. These types of misspecification are oftentimes ignored in the psychological research literature. It is emphasized that the above problems cannot be overcome by strategies such as preregistration, large samples, replications, or a ban on testing null hypotheses. To avoid biased conclusions, we briefly discuss tools such as model diagnostics, statistical methods to compensate for selectivity and semi- or non-parametric estimation. At a more fundamental level, however, a twofold strategy seems indispensable: (1) iterative, cumulative theory development based on statistical methods with theoretically justified assumptions, and (2) empirical research on variables that affect (self-) selection into the observed part of the sample and the use of this information to compensate for selectivity.

【 授权许可】

Unknown   
Copyright © 2023 Spiess and Jordan.