Individual versus general structured feedback to improve agreement in grant peer review: a randomized controlled trial

Research Integrity and Peer Review

Pål Ulleberg¹  Jan-Ole Hesselberg¹  Knut Inge Fostervold¹  Ida Svege² 

[1] Department of Psychology, University of Oslo, Oslo, Norway;Faculty of Health Sciences, Oslo Metropolitan University, Oslo, Norway;

关键词: Peer review;    Inter-rater agreement;    Funding;    Reliability;    Feedback;    Training;   

DOI  :  10.1186/s41073-021-00115-5

来源: Springer

【 摘 要 】

BackgroundVast sums are distributed based on grant peer review, but studies show that interrater reliability is often low. In this study, we tested the effect of receiving two short individual feedback reports compared to one short general feedback report on the agreement between reviewers.MethodsA total of 42 reviewers at the Norwegian Foundation Dam were randomly assigned to receive either a general feedback report or an individual feedback report. The general feedback group received one report before the start of the reviews that contained general information about the previous call in which the reviewers participated. In the individual feedback group, the reviewers received two reports, one before the review period (based on the previous call) and one during the period (based on the current call). In the individual feedback group, the reviewers were presented with detailed information on their scoring compared with the review committee as a whole, both before and during the review period. The main outcomes were the proportion of agreement in the eligibility assessment and the average difference in scores between pairs of reviewers assessing the same proposal. The outcomes were measured in 2017 and after the feedback was provided in 2018.ResultsA total of 2398 paired reviews were included in the analysis. There was a significant difference between the two groups in the proportion of absolute agreement on whether the proposal was eligible for the funding programme, with the general feedback group demonstrating a higher rate of agreement. There was no difference between the two groups in terms of the average score difference. However, the agreement regarding the proposal score remained critically low for both groups.ConclusionsWe did not observe changes in proposal score agreement between 2017 and 2018 in reviewers receiving different feedback. The low levels of agreement remain a major concern in grant peer review, and research to identify contributing factors as well as the development and testing of interventions to increase agreement rates are still needed.Trial registrationThe study was preregistered at OSF.io/n4fq3.

【 授权许可】

CC BY