{
  "abstract": "Repeated evaluation can estimate a benchmark score accurately while still requiring replication to certify narrow uncertainty. We characterize that requirement on a fixed grid of $M$ tasks with $L$ binary paths per task under the hard budget $(M+t)K$, where each path costs at most $K$ responses or episodes. For fixed $L \\ge 3$ and $0 < α\\le 1/12$, the optimal expected width on the worst pure cohort is $Θ_{α,L}([M(t+1)]^{-1/2})$ when every task is observed and $Θ_{α,L}([M(t+\\sqrt{M})]^{-1/2})$ when omission is allowed. The lower bounds cover adaptive hard-budget policies, and fixed random-subset designs attain both rates through disagreement certificates. A joint mean/disagreement interval turns the task-covering law into practical finite-budget inference. In an equal-budget LiveCodeBench replay with 16 models, 880 tasks, and five outputs per task, the task-covering design reduces median point-estimation MSE by 87.0\\% relative to pooled uniform sampling, while the Joint certificate produces narrower confidence intervals in 15/16 panels and reduces median interval width by 30.6\\%. Finite-regime analyses identify task coverage as the effective choice at the evaluated scale and characterize how cohort size and within-task agreement determine the useful operating region. Together, the sharp laws and fixed-budget evidence make replication and task coverage explicit design variables for information-efficient repeated evaluation.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2609.29140",
  "author_total": 8,
  "authors": [
    {
      "name": "Yezhou Cheng"
    },
    {
      "name": "Runjia Du"
    },
    {
      "name": "Zeming Liu"
    },
    {
      "name": "Qibai Chen"
    },
    {
      "name": "Hang Lyu"
    },
    {
      "name": "Yilan Wei"
    },
    {
      "name": "Yankai Zeng"
    },
    {
      "name": "Bojun Lin"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/dab1rfxqs22pv.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2609.29140",
  "fulltext_url": null,
  "id": "dab1rfxqs22pv",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2609.29140",
  "language": null,
  "licence": "LicenseRef-arXiv-nonexclusive-1.0",
  "licence_label": "arXiv licence",
  "licence_source": "article",
  "licence_url": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.IT"
    ],
    [
      "arxiv",
      "math.IT"
    ],
    [
      "openalex",
      "1702|Machine Learning and Algorithms"
    ]
  ],
  "openalex": "W7214322886",
  "pdf_url": "https://arxiv.org/pdf/2609.29140",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-24",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/dab1rfxqs22pv.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:24:15Z",
      "native_id": "2609.29140",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2609.29140"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:math.IT",
      "subfield": "general"
    }
  ],
  "title": "Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation",
  "title_text": "Sharp Limits for Honest Uncertainty in Hard-Budget Repeated Evaluation",
  "type": "preprint",
  "updated_at": "2026-10-30T12:24:15Z",
  "url": "https://research.draconitic.com/paper/dab1rfxqs22pv",
  "venue": "arXiv",
  "versions": []
}