The most important tasks for peer reviewers evaluating a randomized controlled trial are not congruent with the tasks most often requested by journal editors

BMC Medicine

Isabelle Boutron¹  Caroline Barnes¹  Gabriel Baron¹  Philippe Ravaud¹  Anthony Chauvin¹ 

[1] INSERM, UMR 1153, Centre of Research in Epidemiology and Statistics Sorbonne Paris Cité – (CRESS), METHODS team, Paris, France

关键词: Recommendations to reviewers;    Randomized controlled trials;    Q-sort;    Peer review;   

Others  :  1217704
DOI  :  10.1186/s12916-015-0395-3

 received in 2015-01-20, accepted in 2015-06-05,  发布年份 2015

【 摘 要 】

Background

The peer review process is a cornerstone of biomedical research publications. However, it may fail to allow the publication of high-quality articles. We aimed to identify and sort, according to their importance, all tasks that are expected from peer reviewers when evaluating a manuscript reporting the results of a randomized controlled trial (RCT) and to determine which of these tasks are clearly requested by editors in their recommendations to peer reviewers.

Methods

We identified the tasks expected of peer reviewers from 1) a systematic review of the published literature and 2) recommendations to peer reviewers for 171 journals (i.e., 10 journals with the highest impact factor for 14 different medical areas and all journals indexed in PubMed that published more than 15 RCTs over 3 months regardless of the medical area). Participants who had peer-reviewed at least one report of an RCT had to classify the importance of each task relative to other tasks using a Q-sort technique. Finally, we evaluated editors’ recommendations to authors to determine which tasks were clearly requested by editors in their recommendations to peer reviewers.

Results

The Q-sort survey was completed by 203 participants, 93 (46 %) with clinical expertise, 72 (36 %) with methodological/statistical expertise, 17 (8 %) with expertise in both areas, and 21 (10 %) with other expertise. The task rated most important by participants (evaluating the risk of bias) was clearly requested by only 5 % of editors. In contrast, the task most frequently requested by editors (provide recommendations for publication), was rated in the first tertile only by 21 % of all participants.

Conclusions

The most important tasks for peer reviewers were not congruent with the tasks most often requested by journal editors in their guidelines to reviewers.

【 授权许可】

   
2015 Chauvin et al.

【 图 表 】

Fig. 1.

Fig. 2.

【 参考文献 】

• [1]Kassirer JP, Campion EW. Peer review: crude and understudied, but indispensable. JAMA. 1994; 272:96-7.
• [2]Jefferson T, Rudin M, Folse S, Davidoff F. Editorial peer review for improving the quality of reports of biomedical studies. Cochrane Database Syst Rev. 2007; 18:MR000016.
• [3]Rennie R. Editorial peer review: its development and rationale. In: Peer review in health sciences. 2nd ed. Godlee F, Jefferson T, editors. BMJ Books, London; 2003: p.1-13.
• [4]Public Library of Science. Peer review—optimizing practices for online scholarly communication. In: House of Commons Science and Technology Committee, editor. Peer Review in Scientific Publications, Eighth Report of Session 2010–2012, Vol. I: Report, Together with Formal, Minutes, Oral and Written Evidence. London: The Stationery Office Limited; 2011. p. 21–2.
• [5]Public Library of Science. Peer review—optimizing practices for online scholarly communication. In: House of Commons Science and Technology Committee, editor. Peer Review in Scientific Publications, Eighth Report of Session 2010–2012, Vol. I: Report, Together with Formal, Minutes, Oral and Written Evidence. London: The Stationery Office Limited; 2011. p. 174–8.
• [6]Rubriq. How we found 15 million hours of lost time. In: Rubriq: independent peer review system. 2013. http://blog.rubriq.com/2013/06/03/how-we-found-15-million-hours-of-lost-time/. Accessed 27 Mar 2015.
• [7]Ghimire S, Kyung E, Kang W, Kin E. Assessment of adherence to the CONSORT statement for quality of reports on randomized controlled trial abstracts from four high-impact general medical journals. Trials. 2012; 13:77. BioMed Central Full Text
• [8]Boutron I, Dutton S, Ravaud P, Altman DG. Reporting and interpretation of randomized controlled trials with statistically nonsignificant results for primary outcomes. JAMA. 2010; 303:2058-64.
• [9]Hopewell S, Collins GS, Boutron I, Yu LM, Cook J, Shanyinde M et al.. Impact of peer review on reports of randomised trials published in open peer review journals: retrospective before and after study. BMJ. 2014; 349:g4145.
• [10]Turner EH, Matthew AM, Linardatos E, Tell RA, Rosenthal R. Selective publication of antidepressant trials and its confluence on apparent efficacy. N Engl J Med. 2008; 358:252-60.
• [11]Melander H, Ahlqvist-Rastad J, Meijer G, Beermann B. Evidence b(i)ased medicine–selective reporting from studies sponsored by pharmaceutical industry: review of studies in new drug applications. BMJ. 2003; 326:1171-3.
• [12]Rowe BH, Strome TL, Spooner C, Bilitz S, Grafstein E, Worster A. Reviewer agreement trends from four years of electronic submissions of conference abstract. BMC Med Res Methodol. 2006; 6:14. BioMed Central Full Text
• [13]Jackson JL, Srinivasan M, Rea J, Fletcher KE, Kravitz RL. The validity of peer review in a general medicine journal. PLoS ONE. 2011; 6:e22475.
• [14]Weiner BK, Weiner JP, Smith HE. Spine journals: is reviewer agreement on publication recommendations greater than would be expected by chance? Spine J. 2010; 10:209-11.
• [15]Bohannon J. Who’s afraid by peer review? Science. 2013; 342:60-5.
• [16]Schroter S, Black N, Evans S, Godlee F, Osorio L, Smith R. What errors do peer reviewers detect, and does training improve their ability to detect them? J R Soc Med. 2008; 101:507-14.
• [17]Hirst A, Altman DG. Are peer reviewers encouraged to use reporting guidelines? A survey of 116 health research journals. PlosOne. 2012; 7:e35621.
• [18]Van Exel J, de Graaf G. Q methodology: A sneak preview. http://qmethod.org/articles/vanExel.pdf. 2005. Accessed 28 Mar 2014.
• [19]Watts S, Stenner P. Doing Q methodological research: theory, method and interpretation. Sage Publications, London; 2012.
• [20]Brown S. Q-methodology and qualitative research. Qual Life Res. 1996; 6:561-7.
• [21]Dennis KE. Q methodology: relevance and application to nursing research. Adv Nurs Sci. 1986; 8:6-17.
• [22]Mathieu S, Chan AW, Ravaud P. Use of trial register information during the peer review process. PLoS One. 2013; 10:8(4).
• [23]Stephenson W. Correlating persons instead of tests. J Personality. 1935; 4:17-24.
• [24]Hackert C, Braehler G. FlashQ. 2007. http://www.hackert.biz/flashq/. Accessed 30 Mar 2014.
• [25]Baxt WG, Waeckerle JF, Berlin JA, Callaham ML. Who reviews the reviewers? Feasibility of using a fictitious manuscript to evaluate peer reviewer performance. Ann Emerg Med. 1998; 32:310-7.
• [26]Kravitz RL, Franks P, Feldman MD, Gerrity M, Byrne C, Tierney WM. Editorial peer reviewers' recommendations at a general medical journal: are they reliable and do editors care? PLoS One. 2010; 5:e10072.
• [27]Henderson M. Problems with peer review. BMJ. 2010;340:c1409. doi:http://dx.doi.org/10.1136/bmj.c1409.
• [28]Yaffe MB. Re-reviewing peer review. Sci Signal. 2009; 2:eg11.
• [29]Stahel PF, Moore EE. Peer review for biomedical publications: we can improve the system. BMC Med. 2014; 12:179. BioMed Central Full Text
• [30]Matthieu S, Boutron I, Moher D, Altman DG, Ravaud P. Comparison of registered and published primary outcomes in randomized controlled trials. JAMA. 2009; 302:977-84.
• [31]Cobo E, Cortés J, Ribera JM, Cardellach F, Selva-O'Callaghan A, Kostov B et al.. Effect of using reporting guidelines during peer review on quality of final manuscripts submitted to a biomedical journal: masked randomised trial. BMJ. 2011; 22:343.
• [32]Mills EJ, Wu P, Gagnier J, Devereaux PJ. The quality of randomized trial reporting in leading medical journals since the revised CONSORT statement. Contemp Clin Trials. 2005; 26:480-7.