Automatic Construction of Strong Simple Baselines
Abstract
To rigorously establish the value of a novel deep learning method, it must be compared against appropriate baselines. As deep learning has become increasingly common in the physical and life sciences, there have been numerous cases in which apparent progress has turned out to be an artifact of omitted or missing baselines. In this work, we investigate whether this problem can be mitigated by using large language models to automate the construction of baselines. We assemble 36 supervised classification and regression tasks from peer-reviewed studies in Nature, Science, and Cell-family journals that investigate novel deep learning methods for scientific applications. We then use a language model to construct simple baselines for each task. These automatically constructed baselines are strong, recovering a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the score of the deep learning model. The median simple baseline matches or exceeds the performance of the deep learning model in 12 studies. We further examine the variation across scientific domains and publication venues and we characterize the effects of test-time compute and access to different levels of study context. As human peer review has struggled to keep up with the rapidly increasing volume of research articles in recent years, our results suggest that automated baseline construction is a practical and scalable check on deep learning contributions in the physical and life sciences.
Related
- Where single-cell foundation models help: a benchmark of annotation, perturbation and regulatory inference against simple baselines
- Paleo- and historical ecology contributions to baselines for restoration and reintroductions in central Chile
- Baselines Matter: Cardiovascular Baseline Procedures in Affective Psychophysiology
- Context-aware quantile baselines for disturbance detection in non-stationary short-rotation forest plantations
- Classical baselines outperform released deep-learning ITS classifiers, which collapse on ITS2 where predictions follow the flanking regions
BibTeX
@misc{ellington2026automatic,
title = {{Automatic Construction of Strong Simple Baselines}},
author = {Caleb N. Ellington and Jiaqi Wang and Sohan Addagudi and Elijah Cole},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.40700},
eprint = {2610.40700},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.40700},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Automatic Construction of Strong Simple Baselines
AU - Caleb N. Ellington
AU - Jiaqi Wang
AU - Sohan Addagudi
AU - Elijah Cole
PY - 2026
DA - 2026/10/28
PB - arXiv
DO - 10.48550/arxiv.2610.40700
UR - https://arxiv.org/abs/2610.40700
AB - To rigorously establish the value of a novel deep learning method, it must be compared against appropriate baselines. As deep learning has become increasingly common in the physical and life sciences, there have been numerous cases in which apparent progress has turned out to be an artifact of omitted or missing baselines. In this work, we investigate whether this problem can be mitigated by using large language models to automate the construction of baselines. We assemble 36 supervised classification and regression tasks from peer-reviewed studies in Nature, Science, and Cell-family journals that investigate novel deep learning methods for scientific applications. We then use a language model to construct simple baselines for each task. These automatically constructed baselines are strong, recovering a median of 95.5% (mean 85.7%) of the gap between a trivial predictor and the score of the deep learning model. The median simple baseline matches or exceeds the performance of the deep learning model in 12 studies. We further examine the variation across scientific domains and publication venues and we characterize the effects of test-time compute and access to different levels of study context. As human peer review has struggled to keep up with the rapidly increasing volume of research articles in recent years, our results suggest that automated baseline construction is a practical and scalable check on deep learning contributions in the physical and life sciences.
ER -