Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation
Abstract
Estimating neural scaling laws involves an end-to-end pipeline: designing a scaling recipe, training models across scales, evaluating models on a target metric, and deploying an algorithm to fit a scaling law to the resulting data. This final fitting stage is often treated as routine: standard procedures minimize a non-convex fitting loss using local optimization from a grid of initializations, and return the best solution found. Yet such heuristic searches can fail in two distinct ways: they offer no guarantee of returning a nearly optimal solution, and they obscure whether fundamentally different parameter choices fit the data nearly as well. Indeed, we show that standard scaling law fitting is subject to a Rashomon effect (Breiman, 2001): the sublevel sets of near-minimizers correspond to broad parameter regions whose extrapolations diverge substantially at scale. We resolve these issues with a deterministic branch-and-bound algorithm that guarantees nearly optimal fitting loss and exhaustively maps the near-minimizers to the implied range of scaling curves.
Related
- ArborEnum: Decision Tree Rashomon Sets over Continuous Features
- Largest Rashomon sets of decision trees for robust contextual optimization
- The Challenges of Hyperparameter Tuning for Accurate Causal Effect Estimation.
- Learning Latent Algebraic Structure from Ambiguous Set Observations
- Infinite chains of perfect fits for expanding Thurston maps
BibTeX
@misc{mlodozeniec2026ambiguous,
title = {{Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation}},
author = {Bruno Mlodozeniec and Spencer Frei},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.34514},
eprint = {2610.34514},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.34514},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Ambiguous Fits: The Rashomon Effect and Challenges in Scaling Law Estimation
AU - Bruno Mlodozeniec
AU - Spencer Frei
PY - 2026
DA - 2026/10/25
PB - arXiv
DO - 10.48550/arxiv.2610.34514
UR - https://arxiv.org/abs/2610.34514
AB - Estimating neural scaling laws involves an end-to-end pipeline: designing a scaling recipe, training models across scales, evaluating models on a target metric, and deploying an algorithm to fit a scaling law to the resulting data. This final fitting stage is often treated as routine: standard procedures minimize a non-convex fitting loss using local optimization from a grid of initializations, and return the best solution found. Yet such heuristic searches can fail in two distinct ways: they offer no guarantee of returning a nearly optimal solution, and they obscure whether fundamentally different parameter choices fit the data nearly as well. Indeed, we show that standard scaling law fitting is subject to a Rashomon effect (Breiman, 2001): the sublevel sets of near-minimizers correspond to broad parameter regions whose extrapolations diverge substantially at scale. We resolve these issues with a deterministic branch-and-bound algorithm that guarantees nearly optimal fitting loss and exhaustively maps the near-minimizers to the implied range of scaling curves.
ER -