The geometry of AI validation: From structural blindness to reusable audits
Abstract
AI systems increasingly search among candidate answers and deploy the highest-scoring one. Increasing search changes which errors matter, so a precise evaluation at one computation budget can leave another budget unresolved. We connect this information gap to the cost of closing it. For independent best-of-n search, aggregate reliability measurements identify deployment only through the directions they observe; we derive an exact ambiguity frontier when only small search widths are audited. Retaining candidate ranks and truth labels enables a constructive alternative: one audit can estimate reliability across all widths up to N. With known score percentiles, the minimax worst-coordinate mean squared error scales as (1 + log N)/T + N/M, capped at a constant, for expected budgets of T truth labels and M candidate observations. Matching lower bounds allow adaptive label acquisition, establishing that the distinct label and candidate costs are intrinsic to this experiment. An explicit design attains this order; a complementary record-based procedure supplies simultaneous guarantees without a known score distribution. Retrospective mathematical-reasoning and code-generation analyses show why search-dependent validation matters. In held-out CodeRM pools, a shared audit reduces the 95th-percentile maximum error across 100 widths by 58% and 40% relative to uniform labeling. These results turn structural ambiguity into a quantitative prescription for reusable validation.
Related
- Geometry Parameterisation via a Structural Modal Basis for Aerodynamic Shape Optimisation
- Fuzzy–BKM Spectral Geometry: Axial Blocks, Positivity, and Reproducible Finite-Dimensional Validation
- Elimination Geometry II: Integrability, Representation, and Structural Limits
- Deterministic Leave-One-Cluster-Out Cross-Validation for Multilevel Bayesian Structural Equation Models
- Risk-Limiting Audits for Parliamentary Majorities
BibTeX
@misc{fitas2026geometry,
title = {{The geometry of AI validation: From structural blindness to reusable audits}},
author = {Ricardo Fitas},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2608.21496},
eprint = {2608.21496},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2608.21496},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - The geometry of AI validation: From structural blindness to reusable audits
AU - Ricardo Fitas
PY - 2026
DA - 2026/09/11
PB - arXiv
DO - 10.48550/arxiv.2608.21496
UR - https://arxiv.org/abs/2608.21496
AB - AI systems increasingly search among candidate answers and deploy the highest-scoring one. Increasing search changes which errors matter, so a precise evaluation at one computation budget can leave another budget unresolved. We connect this information gap to the cost of closing it. For independent best-of-n search, aggregate reliability measurements identify deployment only through the directions they observe; we derive an exact ambiguity frontier when only small search widths are audited. Retaining candidate ranks and truth labels enables a constructive alternative: one audit can estimate reliability across all widths up to N. With known score percentiles, the minimax worst-coordinate mean squared error scales as (1 + log N)/T + N/M, capped at a constant, for expected budgets of T truth labels and M candidate observations. Matching lower bounds allow adaptive label acquisition, establishing that the distinct label and candidate costs are intrinsic to this experiment. An explicit design attains this order; a complementary record-based procedure supplies simultaneous guarantees without a known score distribution. Retrospective mathematical-reasoning and code-generation analyses show why search-dependent validation matters. In held-out CodeRM pools, a shared audit reduces the 95th-percentile maximum error across 100 widths by 58% and 40% relative to uniform labeling. These results turn structural ambiguity into a quantitative prescription for reusable validation.
ER -