Statistical Significance Filtering Overestimates Effects and Impedes Falsification: A Critique of Endsley (2019)

Frontiers in Psychology

Elyssa Twedt¹  Jared B. Kenworthy²  Laura R. Marusich³  Jonathan Z. Bakdash⁴  Erin G. Zaroukian⁵ 

[1]Department of Psychology, St. Lawrence University, Canton, NY, United States

[2]Department of Psychology, University of Texas at Arlington, Arlington, TX, United States

[3]United States Army Combat Capabilities Development Command, Army Research Laboratory South at the University of Texas at Arlington, Arlington, TX, United States

[4]United States Army Combat Capabilities Development Command, Army Research Laboratory South at the University of Texas at Dallas, Richardson, TX, United States

[5]Department of Psychology and Special Education, Texas A&M University–Commerce, Commerce, TX, United States

[6]United States Army Combat Capabilities Development Command, Army Research Laboratory, Computational and Information Sciences Directorate, Aberdeen, MD, United States

关键词: significance filter;    selection bias;    p-hacking;    meta-analysis;    confirmation bias;    situation awareness;    performance;    falsification;   

DOI  :  10.3389/fpsyg.2020.609647

来源: Frontiers

【 摘 要 】

Whether in meta-analysis or single experiments, selecting results based on statistical significance leads to overestimated effect sizes, impeding falsification. We critique a quantitative synthesis that used significance to score and select previously published effects for situation awareness-performance associations (Endsley, 2019). How much does selection using statistical significance quantitatively impact results in a meta-analytic context? We evaluate and compare results using significance-filtered effects versus analyses with all effects as-reported. Endsley reported high predictiveness scores and large positive mean correlations but used atypical methods: the hypothesis was used to select papers and effects. Papers were assigned the maximum predictiveness scores if they contained at-least-one significant effect, yet most papers reported multiple effects, and the number of non-significant effects did not impact the score. Thus, the predictiveness score was rarely less than the maximum. In addition, only significant effects were included in Endsley’s quantitative synthesis. Filtering excluded half of all reported effects, with guaranteed minimum effect sizes based on sample size. Results for filtered compared to as-reported effects clearly diverged. Compared to the mean of as-reported effects, the filtered mean was overestimated by 56%. Furthermore, 92% (or 222 out of 241) of the as-reported effects were below the mean of filtered effects. We conclude that outcome-dependent selection of effects is circular, predetermining results and running contrary to the purpose of meta-analysis. Instead of using significance to score and filter effects, meta-analyses should follow established research practices.

【 授权许可】

CC BY