Learning Checklists to Reduce Lottery Effects in Scientific Peer Review
Abstract
Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.
Related
- A partial post-review by a peer of the peer-reviewed article [1] "Modeling COVID-19 transmission: effects of age structure and vaccination, published in 2025 in the Springer Nature journal \it{\bf Scientific Reports}
- A distance-based theory of lottery complexity
- Generative Refinement Learning for Temporal Interpolation in Scientific Machine Learning
- PTED: A multi-dimensional two-sample test for scientific inference and generative machine learning
- A Method for Exact Enforcement of Boundary Conditions on General Curved Triangular Domains: Algorithm and Application to Scientific Machine Learning
BibTeX
@misc{liu2026learning,
title = {{Learning Checklists to Reduce Lottery Effects in Scientific Peer Review}},
author = {Xuanyou Liu and Ziyang Guo and Jason Hartline and Jessica Hullman},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.28666},
eprint = {2610.28666},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.28666},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Learning Checklists to Reduce Lottery Effects in Scientific Peer Review
AU - Xuanyou Liu
AU - Ziyang Guo
AU - Jason Hartline
AU - Jessica Hullman
PY - 2026
DA - 2026/10/14
PB - arXiv
DO - 10.48550/arxiv.2610.28666
UR - https://arxiv.org/abs/2610.28666
AB - Reviewer lottery error describes error due to a small set of peer reviews attending to only some of the issues with a paper that a larger pool of eligible reviewers would raise. We show this error can be reduced by surfacing a checklist of criticisms learned from a historical review corpus. Our approach generates a tree of checklists representing coarsenings of the atomic review claim space, then selects the coarsening that maximizes a theoretically-motivated measure of correlated agreement among reviewers. We realize this approach with an LLM-assisted pipeline and test it on NeurIPS 2021 and ICLR 2022 review data. In simulated review with LLM reviewers, the learned checklist reduces the mean squared error (MSE) of a three reviewer committee by 58% on NeurIPS and 21% on ICLR, and on both venues the checklist with the highest correlated agreement has the lowest error for committees of three or more. In a human study with 20 participants, the learned NeurIPS checklist lowers the MSE of a paper ranking, measured against the NeurIPS 2021 order, from 0.48 to 0.22.
ER -