Why do reviewers disagree? Evidence from four funders and 134,000 reviews
- 1Department of Psychology, University of Oslo, Oslo, Norway
- 2The Foundation Dam, Oslo, Norway
- 3NIFU Nordic Institute for Studies of innovation, research and education, Oslo, Norway
Abstract
Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study is a registered report that examines factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we explored the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our analyses revealed several statistically significant associations, but most effects were small in absolute magnitude. Reviewer similarity in gender was associated with lower disagreement. The use of in-depth review procedures, referring to review roles with greater responsibility for evaluating the application, was also associated with slightly lower disagreement. In contrast, expert reviews, defined as assessments conducted by reviewers selected for subject-matter expertise rather than general panel membership, and certain research areas showed modestly higher disagreement. Application amount, reviewer age composition, and temporal trends exhibited limited influence. Only a small share of disagreement stemmed from stable reviewer tendencies (3.1 percent), whereas most variability arose at the application level (24.1 percent) and the residual level (55.2 percent), indicating that disagreement is largely application-specific rather than reviewer-driven. Inter-rater reliability was moderate across funders: average ICCs ranged from 0.50 to 0.65. Our findings only partly support previous research and indicate that disagreement is driven primarily by application-specific ambiguity and inherent variability in human judgment rather than systematic reviewer characteristics or program features. Consequently, interventions targeting reviewer behaviour alone are unlikely to substantially improve reliability; gains are more likely to come from clearer criteria, more structured review formats, and reducing application-level ambiguity, though such measures are expected to yield only incremental improvements.
Related
- MAPEVAScR: a four-phase framework integrating transparent rule based text mining, evidence gap mapping and a mandatory human validation gate for scoping reviews framework development and empirical validation in occupational health.
- Validation of a two-stage automated screening pipeline for medical systematic reviews: a stratified concordance study with three independent human reviewers, using PABAK and Gwet's AC1
- Evidence quality assessment of traditional Chinese medicine and integrated traditional Chinese and Western medicine for acute upper respiratory infection in children: an umbrella review of systematic reviews and meta-analyses
- Surgical Management in Patients With Drug-Resistant Epilepsy: A Summary of Four Systematic Reviews and Meta-Analyses.
- Best evidence summary on non-pharmacological management of procedural pain in NICU preterm infants: an evidence synthesis and overview of reviews.
BibTeX
@article{hesselberg2026reviewers,
title = {{Why do reviewers disagree? Evidence from four funders and 134,000 reviews}},
author = {Jan-Ole Hesselberg and Pål Ulleberg and Øystein Sørensen and Knut Inge Fostervold and Sigrid Hegna Ingvaldsen and Ida Svege},
journal = {PLOS One},
year = {2026},
month = sep,
doi = {10.1371/journal.pone.0356108},
url = {https://doi.org/10.1371/journal.pone.0356108},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Why do reviewers disagree? Evidence from four funders and 134,000 reviews
AU - Jan-Ole Hesselberg
AU - Pål Ulleberg
AU - Øystein Sørensen
AU - Knut Inge Fostervold
AU - Sigrid Hegna Ingvaldsen
AU - Ida Svege
PY - 2026
DA - 2026/09/23
JO - PLOS One
DO - 10.1371/journal.pone.0356108
UR - https://doi.org/10.1371/journal.pone.0356108
AB - Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study is a registered report that examines factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we explored the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our analyses revealed several statistically significant associations, but most effects were small in absolute magnitude. Reviewer similarity in gender was associated with lower disagreement. The use of in-depth review procedures, referring to review roles with greater responsibility for evaluating the application, was also associated with slightly lower disagreement. In contrast, expert reviews, defined as assessments conducted by reviewers selected for subject-matter expertise rather than general panel membership, and certain research areas showed modestly higher disagreement. Application amount, reviewer age composition, and temporal trends exhibited limited influence. Only a small share of disagreement stemmed from stable reviewer tendencies (3.1 percent), whereas most variability arose at the application level (24.1 percent) and the residual level (55.2 percent), indicating that disagreement is largely application-specific rather than reviewer-driven. Inter-rater reliability was moderate across funders: average ICCs ranged from 0.50 to 0.65. Our findings only partly support previous research and indicate that disagreement is driven primarily by application-specific ambiguity and inherent variability in human judgment rather than systematic reviewer characteristics or program features. Consequently, interventions targeting reviewer behaviour alone are unlikely to substantially improve reliability; gains are more likely to come from clearer criteria, more structured review formats, and reducing application-level ambiguity, though such measures are expected to yield only incremental improvements.
SN - 1932-6203
ER -