Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
Abstract
Equivalent probability requests can lead to different decisions even when both reports are valid. We introduce probability contracts, a benchmark that connects exact finite-world posteriors, validated event alignment, interface coherence, and failure-aware decision evaluation. Across four model-interface configurations on 1,000 worlds, Kev has lower aggregate canonical posterior error than Jev but larger complement and coarsening residuals; accuracy ordering varies by stratum. Jev's Event and Choice interfaces change the binary action on 32.8% of valid pairs at defer cost 0.10. Post-hoc analyses show that disagreement certifies only 11-52% of mean binary pair error and does not consistently outperform confidence for selection. An action-region characterization and a standard scoring-rule identity explain averaging's expected Brier guarantee relative to random interface selection, but not a decision-loss guarantee at each cost. The loss contrast takes both signs on a 99-cost grid for every configuration; small penalties where both policies beat deferral have pointwise intervals containing zero. Secondary checks specified before collection include a separate 400-root cohort, where Event/Choice effects remain configuration-dependent. A joint surface-order and answer-ID intervention shifts posttrained probabilities. Both bounded reasoning arms yield no valid probability reports, leaving their probability accuracy undefined. Probability contracts make these distinctions measurable by evaluating event semantics, posterior error, coverage, and decision cost together.
Related
- Coherence is not enough: Aggregation constraints across predictive distributions, forecasts and decisions
- Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
- One Probability, Two Roles: The Separation of Coherence and Frequency in Adaptive Regimes
- American condor contracts with a cash component: A variational inequality approach
- Distributed Model Predictive Control with Connectivity-based Contracts
BibTeX
@misc{chen2026probability,
title = {{Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces}},
author = {Han Chen and Yingrui Li},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.37470},
eprint = {2609.37470},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.37470},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
AU - Han Chen
AU - Yingrui Li
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2609.37470
UR - https://arxiv.org/abs/2609.37470
AB - Equivalent probability requests can lead to different decisions even when both reports are valid. We introduce probability contracts, a benchmark that connects exact finite-world posteriors, validated event alignment, interface coherence, and failure-aware decision evaluation. Across four model-interface configurations on 1,000 worlds, Kev has lower aggregate canonical posterior error than Jev but larger complement and coarsening residuals; accuracy ordering varies by stratum. Jev's Event and Choice interfaces change the binary action on 32.8% of valid pairs at defer cost 0.10. Post-hoc analyses show that disagreement certifies only 11-52% of mean binary pair error and does not consistently outperform confidence for selection. An action-region characterization and a standard scoring-rule identity explain averaging's expected Brier guarantee relative to random interface selection, but not a decision-loss guarantee at each cost. The loss contrast takes both signs on a 99-cost grid for every configuration; small penalties where both policies beat deferral have pointwise intervals containing zero. Secondary checks specified before collection include a separate 400-root cohort, where Event/Choice effects remain configuration-dependent. A joint surface-order and answer-ID intervention shifts posttrained probabilities. Both bounded reasoning arms yield no valid probability reports, leaving their probability accuracy undefined. Probability contracts make these distinctions measurable by evaluating event semantics, posterior error, coverage, and decision cost together.
ER -