On the estimation and validity of AI time horizons---a statistical look at the METR plot
Abstract
METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \emph{converts} human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of 10 ×. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.
Related
- A Critical Look at Power Sampling when Test Time Sharpening LLMs
- Time-invariant linear quadratic regulators: A new look
- The Hidden Look-Ahead: Point-in-Time Leakage in Electricity Market Data
- A statistical theory for blind denoising: minimax estimation of the noise level from a single sample
- Long-time validity of the Korteweg-de Vries approximation for the Boussinesq equation
BibTeX
@misc{nguyen2026estimation,
title = {{On the estimation and validity of AI time horizons---a statistical look at the METR plot}},
author = {Drew T. Nguyen and William Fithian},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.12466},
eprint = {2610.12466},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.12466},
}
RIS
TY - UNPB
TI - On the estimation and validity of AI time horizons---a statistical look at the METR plot
AU - Drew T. Nguyen
AU - William Fithian
PY - 2026
DA - 2026/10/08
PB - arXiv
DO - 10.48550/arxiv.2610.12466
UR - https://arxiv.org/abs/2610.12466
AB - METR's 50\% time horizon measures the human completion time of software tasks that an AI solves with 50\% probability, allowing AI capabilities to be expressed in interpretable units. On 228 tasks and 26 AIs, we recompute the time horizons using splines and item-response theory to relax the assumption that the AI difficulty of a task depends linearly on the log of human time. Our fitted spline can be interpreted as a function that \emph{converts} human time to AI difficulty; it is nearly flat in a region from 2--30 min but close to linear elsewhere. Hence, a time-horizon jump from 3 min to 30 min is much easier than one from 30 min to 5 hours despite the same multiplier of 10 ×. Overall, we contribute time-horizon point estimates that perform better under a cross-validated suite of proper scoring rules, as well as diagnostic plots for assessing time horizons' construct validity. We suggest that time horizons be interpreted together with the diagnostic plots, especially as new time-horizon-based benchmarks are proposed or existing ones grow to include longer tasks.
ER -