{
  "abstract": "Best-of-N (BoN) is a common inference-time alignment method that selects the highest-scoring response among N samples from a reference model. Evaluating BoN policies from logged data is challenging under sample-only access because standard off-policy estimators require density ratios that depend on unavailable response likelihoods. In this paper, we propose a sample-only framework for evaluating and selecting BoN policies without access to these likelihoods. We show that the order-statistic structure of BoN allows the required density ratios to be expressed through score-rank probabilities that are estimable from samples alone. We then develop a doubly robust estimator of the BoN policy value (BoN-DR) that efficiently reuses a shared auxiliary sample pool across candidate budgets. We establish valid asymptotic inference even under reward estimator misspecification and prove the efficiency of our BoN-DR estimator. Since larger budgets can amplify errors in the score function and lead to reward overoptimization, we derive two selection rules: (i) maximizing the estimated policy value and (ii) maximizing a lower confidence bound on the improvement over the reference policy, which accounts for estimation uncertainty and provides a no-harm guarantee. Across synthetic experiments and GSM8K with multiple reference and reward models, our framework accurately estimates BoN policy values and selects effective sampling budgets.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2610.09250",
  "author_total": 5,
  "authors": [
    {
      "name": "Jonas Schweisthal"
    },
    {
      "name": "Yuxin Wang"
    },
    {
      "name": "Athiya Deviyani"
    },
    {
      "name": "Stefan Feuerriegel"
    },
    {
      "name": "Dennis Frauen"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/628vfj6sth795.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2610.09250",
  "fulltext_url": null,
  "id": "628vfj6sth795",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2610.09250",
  "language": null,
  "licence": "LicenseRef-arXiv-nonexclusive-1.0",
  "licence_label": "arXiv licence",
  "licence_source": "article",
  "licence_url": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.LG"
    ],
    [
      "arxiv",
      "stat.ML"
    ],
    [
      "openalex",
      "1702|Reinforcement Learning in Robotics"
    ]
  ],
  "openalex": "W7221025178",
  "pdf_url": "https://arxiv.org/pdf/2610.09250",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-07",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/628vfj6sth795.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:27:48Z",
      "native_id": "2610.09250",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2610.09250"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ML",
      "subfield": "statistics"
    }
  ],
  "title": "Efficient Best-of-N policy evaluation for inference-time alignment",
  "title_text": "Efficient Best-of-N policy evaluation for inference-time alignment",
  "type": "preprint",
  "updated_at": "2026-10-30T12:27:48Z",
  "url": "https://research.draconitic.com/paper/628vfj6sth795",
  "venue": "arXiv",
  "versions": []
}