{
  "abstract": "Predicting how many tokens a large language model (LLM) will decode, using only prefill-phase hidden states, is a key primitive for inference scheduling. We show that current benchmarks conflate predictor quality with task hardness: when the target model, dataset, or sampling configuration changes, reported accuracy collapses even though the predictor is unchanged. We trace this instability to an overlooked error source---the inherent stochasticity of autoregressive decoding---and derive bounds that every prefill-side method must obey. Our analysis treats decoding as a stopped stochastic process. The per-step stopping probability of a response is not constant but ramps up over hundreds of steps, so the decode length inherits a central limit theorem from the many local decisions that determine where a response ends; this yields per-prompt normality with an explicit error term, together with an affine variance law $σ=αμ+b$ whose slope is the coefficient of variation of the number of response segments. Empirically, the normal law reproduces the per-prompt MAE and the central mass of the decode-length distribution to within a few percent, and the residual departures from normality (mild right skew, slight excess kurtosis) are the ones predicted by the sampling-noise correction. Under fixed model, data, and sampling conditions, a broad design space of predictors (MLP, EGTP, FiLM, soft-label, and Mixture-of-Experts variants) converges to similar performance close to the bound, confirming that \\emph{system error dominates model error}. We argue for a benchmarking protocol that fixes the model, data, and sampling parameters and that reports relative-error metrics alongside absolute ones. Code and analysis tools are available at https://github.com/ShwStone/DecodingPredict.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2610.40541",
  "author_total": 4,
  "authors": [
    {
      "name": "Haowen Shi"
    },
    {
      "name": "Xuzhe Yan"
    },
    {
      "name": "Fahao Chen"
    },
    {
      "name": "Peng Li"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/9xna9gwxjvnx0.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2610.40541",
  "fulltext_url": null,
  "id": "9xna9gwxjvnx0",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2610.40541",
  "language": null,
  "licence": "CC-BY-SA-4.0",
  "licence_label": "CC BY-SA 4.0",
  "licence_source": "article",
  "licence_url": "http://creativecommons.org/licenses/by-sa/4.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.CL"
    ],
    [
      "arxiv",
      "cs.IT"
    ],
    [
      "arxiv",
      "math.IT"
    ]
  ],
  "openalex": null,
  "pdf_url": "https://arxiv.org/pdf/2610.40541",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-29",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/9xna9gwxjvnx0.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:36:35Z",
      "native_id": "2610.40541",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2610.40541"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:math.IT",
      "subfield": "general"
    }
  ],
  "title": "On the Fundamental Limits of Prefill-Based Decoding Length Prediction",
  "title_text": "On the Fundamental Limits of Prefill-Based Decoding Length Prediction",
  "type": "preprint",
  "updated_at": "2026-11-01T12:20:53Z",
  "url": "https://research.draconitic.com/paper/9xna9gwxjvnx0",
  "venue": "arXiv",
  "versions": []
}