{
  "abstract": "Grant peer review processes are pivotal in allocating substantial research funding, yet concerns about their reliability persist, primarily due to low inter-rater agreement. This study is a registered report that examines factors associated with agreement among peer reviewers in grant evaluations, leveraging data from 134,991 reviews across four Norwegian research funders. Using a cross-classified linear regression model, we explored the relationship between inter-rater agreement and multiple factors, including reviewer similarity, experience, expertise, research area, application characteristics, review depth, and temporal trends. Our analyses revealed several statistically significant associations, but most effects were small in absolute magnitude. Reviewer similarity in gender was associated with lower disagreement. The use of in-depth review procedures, referring to review roles with greater responsibility for evaluating the application, was also associated with slightly lower disagreement. In contrast, expert reviews, defined as assessments conducted by reviewers selected for subject-matter expertise rather than general panel membership, and certain research areas showed modestly higher disagreement. Application amount, reviewer age composition, and temporal trends exhibited limited influence. Only a small share of disagreement stemmed from stable reviewer tendencies (3.1 percent), whereas most variability arose at the application level (24.1 percent) and the residual level (55.2 percent), indicating that disagreement is largely application-specific rather than reviewer-driven. Inter-rater reliability was moderate across funders: average ICCs ranged from 0.50 to 0.65. Our findings only partly support previous research and indicate that disagreement is driven primarily by application-specific ambiguity and inherent variability in human judgment rather than systematic reviewer characteristics or program features. Consequently, interventions targeting reviewer behaviour alone are unlikely to substantially improve reliability; gains are more likely to come from clearer criteria, more structured review formats, and reducing application-level ambiguity, though such measures are expected to yield only incremental improvements.",
  "abstract_shown": true,
  "affiliations": [
    "Department of Psychology, University of Oslo, Oslo, Norway",
    "The Foundation Dam, Oslo, Norway",
    "NIFU Nordic Institute for Studies of innovation, research and education, Oslo, Norway"
  ],
  "article_type": "Research Article",
  "arxiv": null,
  "author_total": 6,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Jan-Ole Hesselberg"
    },
    {
      "affs": [
        1
      ],
      "name": "Pål Ulleberg"
    },
    {
      "affs": [
        2
      ],
      "name": "Øystein Sørensen"
    },
    {
      "name": "Knut Inge Fostervold"
    },
    {
      "name": "Sigrid Hegna Ingvaldsen"
    },
    {
      "name": "Ida Svege"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/9310r6p6xc2m6.bib",
  "cited_by": 0,
  "doi": "10.1371/journal.pone.0356108",
  "fulltext_url": null,
  "id": "9310r6p6xc2m6",
  "issns": [
    "1932-6203"
  ],
  "landing_url": "https://doi.org/10.1371/journal.pone.0356108",
  "language": null,
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "journal",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "openalex",
      "1802|Research, Science, and Academia"
    ],
    [
      "plos",
      "/Biology and life sciences/Agriculture"
    ],
    [
      "plos",
      "/Medicine and health sciences"
    ],
    [
      "plos",
      "/Medicine and health sciences/Medical humanities"
    ],
    [
      "plos",
      "/Medicine and health sciences/Oncology/Cancers and neoplasms"
    ],
    [
      "plos",
      "/Research and analysis methods/Research assessment/Peer review"
    ],
    [
      "plos",
      "/Research and analysis methods/Research assessment/Research validity"
    ],
    [
      "plos",
      "/Science policy/Research funding"
    ],
    [
      "plos",
      "/Social sciences"
    ]
  ],
  "openalex": "W7214067911",
  "pdf_url": "https://journals.plos.org/plosone/article/file?id=10.1371/journal.pone.0356108&type=printable",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-23",
  "publisher": "Public Library of Science",
  "ris": "https://research.draconitic.com/paper/9310r6p6xc2m6.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:17:43Z",
      "native_id": "10.1371/journal.pone.0356108",
      "source": "plos",
      "url": "https://doi.org/10.1371/journal.pone.0356108"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "plos:cancers and neoplasms",
      "subfield": "oncology"
    }
  ],
  "title": "Why do reviewers disagree? Evidence from four funders and 134,000 reviews",
  "title_text": "Why do reviewers disagree? Evidence from four funders and 134,000 reviews",
  "type": "peer-reviewed",
  "updated_at": "2026-10-30T12:17:43Z",
  "url": "https://research.draconitic.com/paper/9310r6p6xc2m6",
  "venue": "PLOS One",
  "versions": []
}