Consensus-based guidance for conducting and reporting multi-analyst studies

eLife

Alexandra Sarafoglou¹  Eric-Jan Wagenmakers¹  Dora Matzke¹  Brian A Nosek²  Rotem Botvinik-Nezer³  Balazs Aczel⁴  Barnabas Szaszi⁴  Martin Schweinsberg⁵  Eric Luis Uhlmann⁶  Gustav Nilsonne⁷  Sabine Hoffmann⁸  Kai J Jonas⁹  Andrea M Cataldo^1,10  Gary F Egan^1,11  Andrew Delios^1,12  Matthew J Salganik^1,13  Alexander T Kindel^1,13  Russell A Poldrack^1,14  Magnus Johannesson^1,15  Anna Dreber^1,16  Tom Schonberg^1,17  Jelte Wicherts^1,18  Olmo R van den Akker^1,18  Marcel ALM van Assen^1,19  David Shanks^2,20  Gilles Dutilh^2,21  Yoram K Kunkels^2,22  Jojanneke A Bastiaansen^2,23  Samuel St-Jean^2,24  Noah NN van Dongen^2,25  Udo Boehm^2,25  Johnny B van Doorn^2,25  Jörg Rieskamp^2,26  Marcus R Munafò^2,27  Daniel Benjamin^2,28  Rink Hoekstra^2,29  Don van Ravenzwaaij^2,29  Laura F Bringmann^2,29  Casper J Albers^2,29  Daniel J Simons^3,30  Michael Kirchler^3,31  Juergen Huber^3,31  Felix Holzmeister^3,31  Jeffrey J Starns^3,32  Nelson Cowan^3,33  Niko A Busch^3,34  Chris Donkin^3,35  Ben R Newell^3,35  Emmanuel Caruyer^3,36  Raphael Silberzahn^3,37  D Stephen Lindsay^3,38  Barbara A Spellman^3,39  Morton Ann Gernsbacher^4,40  Jean-Francois Mangin^4,41 

[1] Amsterdam University, Amsterdam, Netherlands;Center for Open Science, Charlottesville, United States;University of Virginia, Charlottesville, United States;Dartmouth College, Hanover, United States;ELTE Eotvos Lorand University, Budapest, Hungary;ESMT Berlin, Berlin, Germany;INSEAD, Singapore, Singapore;Karolinska Institutet, Stockholm, Sweden;Stockholm University, Stockholm, Sweden;Ludwig-Maximilians-University, Munich, Germany;Maastricht University, Maastricht, Netherlands;McLean Hospital, Belmont, United States;Harvard Medical School, Boston, United States;Monash University, Melbourne, Australia;National University of Singapore, Singapore, Singapore;Princeton University, Princeton, United States;Stanford University, Stanford, United States;Stockholm School of Economics, Stockholm, Sweden;Stockholm School of Economics, Stockholm, Sweden;University of Innsbruck, Innsbruck, Austria;Tel Aviv University, Tel Aviv, Israel;Tilburg University, Tilburg, Netherlands;Tilburg University, Tilburg, Netherlands;Utrecht University, Utrecht, Netherlands;University College London, London, United Kingdom;University Hospital Basel, Basel, Switzerland;University Medical Center Groningen, University of Groningen, Groningen, Netherlands;University Medical Center Groningen, University of Groningen, Groningen, Netherlands;Friesland Mental Health Care Services, Leeuwarden, Netherlands;University of Alberta, Edmonton, Canada;Lund University, Lund, United States;University of Amsterdam, Amsterdam, Netherlands;University of Basel, Basel, Switzerland;University of Bristol, Bristol, United Kingdom;University of California Los Angeles, Los Angeles, United States;National Bureau of Economic Research, Cambridge, United States;University of Groningen, Groningen, Netherlands;University of Illinois, Urbana-Champaign, United States;University of Innsbruck, Innsbruck, Austria;University of Massachusetts Amherst, Amherst, United States;University of Missouri, Columbia, United States;University of Münster, Muenster, Germany;University of New South Wales, Sydney, Australia;University of Rennes, CNRS, Inria and Inserm, Rennes, France;University of Sussex, Brighton, United Kingdom;University of Victoria, Victoria, Canada;University of Virginia, Charlottesville, United States;University of Wisconsin-Madison, Madison, United States;Université Paris-Saclay, Paris, France;Neurospin, CEA, Paris, France;

关键词: multi-analyst;    metascience;    statistical practice;    science forum;    expert consensus;    analytical variability;    None;   

DOI  :  10.7554/eLife.72185

来源: eLife Sciences Publications, Ltd

【 摘 要 】

Any large dataset can be analyzed in a number of ways, and it is possible that the use of different analysis strategies will lead to different results and conclusions. One way to assess whether the results obtained depend on the analysis strategy chosen is to employ multiple analysts and leave each of them free to follow their own approach. Here, we present consensus-based guidance for conducting and reporting such multi-analyst studies, and we discuss how broader adoption of the multi-analyst approach has the potential to strengthen the robustness of results and conclusions obtained from analyses of datasets in basic and applied research.

【 授权许可】

CC BY