Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis
Abstract
Despite their growing use in academic writing and statistical analysis, the performance of artificial intelligence (AI) tools in scientific peer review remains a largely unexplored area. A key challenge is jagged AI, a phenomenon where AI exhibits strong ability spikes in some domains while remaining deficient in others. To study this jaggedness in a practical data science context, we considered the task of reviewing partially observed Markov process (POMP) data analyses. POMP models, a generalization of state-space models or hidden Markov models, are used to fit mechanistic dynamic models to time series data in diverse applications including disease transmission, ecological dynamics, and financial risk assessment. High-quality peer review in this area entails assessment of scientific context, identification of errors in implementing complex algorithms, and decisions concerning methodological best practices. We studied 72 POMP projects from four semesters of a University of Michigan graduate time series course for which the project reports, the source code, and student peer reviews are anonymized and open access. We compared the human reviews with four AI review agents, using Claude Code with differing instructions implemented as skill files. We found that the AI review agents exhibited a jagged capability profile, identifying plausible technical errors and instances of invalid inference methodology overlooked by humans, while showing lower capability at finding issues involving interpretive errors, narrative coherence, and domain-informed model critique. The jaggedness was similar for all agents, consistent with other research finding that the specific instructions can have limited effect on the capability of AI models for some tasks. Skill file configuration shifted which weaknesses agents emphasized, without removing the jaggedness.
Related
- A partial post-review by a peer of the peer-reviewed article [1] "Modeling COVID-19 transmission: effects of age structure and vaccination, published in 2025 in the Springer Nature journal \it{\bf Scientific Reports}
- Learning Checklists to Reduce Lottery Effects in Scientific Peer Review
- Fear Moves Markets: Sentiment-Augmented POMP for Volatility Modeling of Bitcoin Returns
- Perceived factors associated with self-reported AI-assisted academic cheating among undergraduate students: Evidence from ethical ambiguity, technological affordances, institutional gaps, academic pressure, and peer comparison
- Y-Learn: Design and Initial Implementation of an OBE-Based Learning Platform for Group Work Monitoring and Peer Review Assessment
BibTeX
@misc{lee2026jagged,
title = {{Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis}},
author = {Jin Wook Lee and William Szegda and Zhisheng Song and Edward L. Ionides},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2605.07855},
eprint = {2605.07855},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.07855},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Jagged AI in Scientific Peer Review: Evidence from POMP Data Analysis
AU - Jin Wook Lee
AU - William Szegda
AU - Zhisheng Song
AU - Edward L. Ionides
PY - 2026
DA - 2026/10/02
PB - arXiv
DO - 10.48550/arxiv.2605.07855
UR - https://arxiv.org/abs/2605.07855
AB - Despite their growing use in academic writing and statistical analysis, the performance of artificial intelligence (AI) tools in scientific peer review remains a largely unexplored area. A key challenge is jagged AI, a phenomenon where AI exhibits strong ability spikes in some domains while remaining deficient in others. To study this jaggedness in a practical data science context, we considered the task of reviewing partially observed Markov process (POMP) data analyses. POMP models, a generalization of state-space models or hidden Markov models, are used to fit mechanistic dynamic models to time series data in diverse applications including disease transmission, ecological dynamics, and financial risk assessment. High-quality peer review in this area entails assessment of scientific context, identification of errors in implementing complex algorithms, and decisions concerning methodological best practices. We studied 72 POMP projects from four semesters of a University of Michigan graduate time series course for which the project reports, the source code, and student peer reviews are anonymized and open access. We compared the human reviews with four AI review agents, using Claude Code with differing instructions implemented as skill files. We found that the AI review agents exhibited a jagged capability profile, identifying plausible technical errors and instances of invalid inference methodology overlooked by humans, while showing lower capability at finding issues involving interpretive errors, narrative coherence, and domain-informed model critique. The jaggedness was similar for all agents, consistent with other research finding that the specific instructions can have limited effect on the capability of AI models for some tasks. Skill file configuration shifted which weaknesses agents emphasized, without removing the jaggedness.
ER -