Does mentoring new peer reviewers improve review quality? A randomized trial

BMC Medical Education

Michael Callaham³  Steven Green¹  Debra Houry² 

[1] Department of Emergency Medicine, Loma Linda School of Medicine, Los Angeles, CA, USA;Department of Emergency Medicine, Emory University School of Medicine, Atlanta, GA, USA;Department of Emergency Medicine, University of California, San Francisco, CA, USA

关键词: Journal peer reviewer;    Critical analysis;    Scientific publication;    Peer review;    Mentoring;   

Others  :  1153377
DOI  :  10.1186/1472-6920-12-83

 received in 2012-04-07, accepted in 2012-08-16,  发布年份 2012

【 摘 要 】

Background

Prior efforts to train medical journal peer reviewers have not improved subsequent review quality, although such interventions were general and brief. We hypothesized that a manuscript-specific and more extended intervention pairing new reviewers with high-quality senior reviewers as mentors would improve subsequent review quality.

Methods

Over a four-year period we randomly assigned all new reviewers for Annals of Emergency Medicine to receive our standard written informational materials alone, or these materials plus a new mentoring intervention. For this program we paired new reviewers with a high-quality senior reviewer for each of their first three manuscript reviews, and asked mentees to discuss their review with their mentor by email or phone. We then compared the quality of subsequent reviews between the control and intervention groups, using linear mixed effects models of the slopes of review quality scores over time.

Results

We studied 490 manuscript reviews, with similar baseline characteristics between the 24 mentees who completed the trial and the 22 control reviewers. Mean quality scores for the first 3 reviews on our 1 to 5 point scale were similar between control and mentee groups (3.4 versus 3.5), as were slopes of change of review scores (-0.229 versus -0.549) and all other secondary measures of reviewer performance.

Conclusions

A structured training intervention of pairing newly recruited medical journal peer reviewers with senior reviewer mentors did not improve the quality of their subsequent reviews.

【 授权许可】

   
2012 Houry et al.; licensee BioMed Central Ltd.

【 图 表 】

Figure 1.

【 参考文献 】

• [1]Peer review. http://en.wikipedia.org/wiki/Peer_review webcite
• [2]Callaham ML, Tercier J: The relationship of previous training and experience of journal peer reviewers to subsequent review quality. PLoS Med 2007, 4:e40.
• [3]Evans AT, McNutt RA, Fletcher SW, Fletcher RH: The characteristics of peer reviewers who produce good-quality reviews. J Gen Intern Med 1993, 8:422-428.
• [4]Callaham ML, Knopp RK, Gallagher EJ: Effect of written feedback by editors on quality of reviews: two randomized trials. JAMA 2002, 287:2781-2783.
• [5]Callaham ML, Schriger DL: Effect of structured workshop training on subsequent performance of journal reviewers. Ann Emerg Med 2002, 40:323-328.
• [6]Callaham ML, Wears RL, Waeckerle JF: Effect of attendance at a training session on peer reviewer quality and performance. Ann Emerg Med 1998, 32:318-322.
• [7]Schroter S, Black N, Evans S, Carpenter J, Godlee F, Smith R: Effects of training on quality of peer review: randomised controlled trial. BMJ 2004, 328:673.
• [8]Freda MC, Kearney MH, Baggs JG, Broome ME, Dougherty M: Peer reviewer training and editor support: results from an international survey of nursing peer reviewers. J Prof Nurs 2009, 25:101-108.
• [9]Tercier J, Callaham ML: A normative model of peer review: qualitative assessment of manuscript reviewers’ attitudes towards peer review. http://escholarship.org/uc/item/4p90p67x webcite
• [10]Callaham ML, Schriger D, Cooper RJ: Annals of Emergency Medicine. An instructional guide for peer reviewers of biomedical manuscripts. http://www.journals.elsevierhealth.com/webfiles/images/journals/ymem/index.html webcite
• [11]Callaham ML, Baxt WG, Waeckerle JF, Wears RL: Reliability of editors' subjective quality ratings of peer reviews of manuscripts. JAMA 1998, 280:229-231.
• [12]van Rooyen S, Black N, Godlee F: Development of the review quality instrument (RQI) for assessing peer reviews of manuscripts. J Clin Epidemiol 1999, 52:625-629.
• [13]Callaham ML, McCulloch C: Longitudinal trends in the performance of scientific peer reviewers. Ann Emerg Med 2011, 57:141-148.
• [14]Fitzmaurice GM, Laird NM, Ware JH: Applied longitudinal analysis. Hoboken (NJ): Wiley-Interscience; 2004.
• [15]Sullivan LM, Dukes KA, Losina E: Tutorial in biostatistics. An introduction to hierarchical linear modelling. Stat Med 1999, 18:855-888.
• [16]Bryson M, Bereiter C, Scardamalia M, Joram E: Going beyond the problem as given: Problem solving in expert and novice writers. In Complex problem solving: Principles and mechanisms. Edited by Sternberg RJ, Frensch PA. Hillsdale (NJ): Lawrence Erlbaum; 1991:61-84.
• [17]Snell L, Spencer J: Reviewers’ perceptions of the peer review process for a medical education journal. Med Educ 2005, 39:90-97.
• [18]Parkes J, Hyde C, Deeks J, Milne R: Teaching critical appraisal skills in health care settings. Cochrane Database Syst Rev 2001, 3:CD001270.
• [19]Taylor RS, Reeves BC, Ewings PE, Taylor RJ: Critical appraisal skills training for health care professionals: a randomized controlled trial. BMC Med Educ 2004, 4:30. BioMed Central Full Text
• [20]Ebbert JO, Montori VM, Schultz HJ: The journal club in postgraduate medical education: a systematic review. Med Teach 2001, 23:455-461.
• [21]Green SM, Callaham ML: Implementation of a journal peer reviewer stratification system based on quality and reliability. Ann Emerg Med 2011, 57:149-52.e4.
• [22]Shashok K: Content and communication: How can peer review provide helpful feedback about the writing? BMC Medical Research Methodology 2008, 8:3. BioMed Central Full Text