{
  "abstract": "Researchers across academia increasingly base significance claims on LLM judge scores and small-sample AI evaluations. Yet without well-calibrated confidence intervals (CIs), hypothesis tests, and judge-bias corrections, such claims are unreliable. We address these issues in several contributions. First, we find that running statistics over raw LLM judge scores leads to inflated false positives: counterintuitively, for many inter-rater agreement metrics, false positive risk peaks at \"almost perfect\" human-LLM agreement. To help researchers understand how to run statistics over LLM judges responsibly, we present guidance and tooling for the statistical analysis of mixed human-AI judge designs, and implement nine hypothesis tests via prediction-powered inference (PPI), including the first known PPI corrections for four rank-based tests (Wilcoxon signed-rank, Mann-Whitney U, and omnibus variants). To keep PPI++ stable with small human-labeled calibration sets, we introduce bootstrap-adaptive power tuning, which shrinks the estimated weight toward a target estimated from the labeled data, and accounts for that weight's own sampling variance. Second, through Monte Carlo simulations, we derive recommendations for what CI, p-value, and FWER correction methods to use for small-sample AI evaluations (N<100), and warn researchers against bootstrap CIs. We package these recommendations into evalstats, an open-source Python package that selects calibrated methods automatically, and demonstrate it in three scenarios, including one where a real LLM judge validated at \"substantial agreement\" would have led a researcher to publish a spurious finding. evalstats is publicly available at https://github.com/ianarawjo/evalstats.",
  "abstract_shown": true,
  "affiliations": [],
  "article_type": null,
  "arxiv": "2609.35815",
  "author_total": 1,
  "authors": [
    {
      "name": "Ian Arawjo"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/1nhcxggv0q5z0.bib",
  "cited_by": 0,
  "doi": "10.48550/arxiv.2609.35815",
  "fulltext_url": null,
  "id": "1nhcxggv0q5z0",
  "issns": [],
  "landing_url": "https://arxiv.org/abs/2609.35815",
  "language": null,
  "licence": "LicenseRef-arXiv-nonexclusive-1.0",
  "licence_label": "arXiv licence",
  "licence_source": "article",
  "licence_url": "http://arxiv.org/licenses/nonexclusive-distrib/1.0/",
  "metadata_licence": "CC0-1.0",
  "native_subjects": [
    [
      "arxiv",
      "cs.AI"
    ],
    [
      "arxiv",
      "cs.CL"
    ],
    [
      "arxiv",
      "cs.HC"
    ],
    [
      "arxiv",
      "stat.ME"
    ],
    [
      "openalex",
      "1804|Reliability and Agreement in Measurement"
    ]
  ],
  "openalex": "W7214966144",
  "pdf_url": "https://arxiv.org/pdf/2609.35815",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-20",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/1nhcxggv0q5z0.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:25:41Z",
      "native_id": "2609.35815",
      "source": "arxiv",
      "url": "https://arxiv.org/abs/2609.35815"
    }
  ],
  "subjects": [
    {
      "field": "mathematics",
      "origin": "arxiv:stat.ME",
      "subfield": "statistics"
    }
  ],
  "title": "How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats",
  "title_text": "How to Run Statistics over LLM Judges and Trust the Results: Calibrated Inference for Small-Sample AI Evaluation with evalstats",
  "type": "preprint",
  "updated_at": "2026-10-30T12:25:41Z",
  "url": "https://research.draconitic.com/paper/1nhcxggv0q5z0",
  "venue": "arXiv",
  "versions": []
}