TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
Abstract
System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface. Software can act on these probabilities through thresholds, cost-weighted choices, and escalation rules. Consequently, if these probabilities are miscalibrated or wording-sensitive, the software ma take unintended actions leaving human operators with no textual rationale to inspect. Current evaluations largely report accuracy and calibration on public classification datasets without a clear reference. We present TypedBench, a benchmark for typed decision models like Jev built from seven policy-labelled generators and nine evaluation suites. We report accuracy as median and range across paraphrases, and calibration error relative to the finite-sample noise floor of a matched, perfectly calibrated predictor. We assess probability quality through selective prediction, ordinal proper scoring rules, and realised cost under asymmetric cost matrices. We evaluate a hosted model, an open encoder, and a family of open decoders spanning 0.8B-9B parameters on identical items. The hosted model follows the stated policy but is wording-sensitive and systematically underconfident; under asymmetric costs, using its probabilities can be worse than taking its top answer. Decoders route exactly and are slow as options or questions are added. The decoder is least accurate on policy questions and degrades with more options. Overall, typed decision models must be evaluated jointly on policy adherence, wording robustness, probability quality, and induced decision outcomes.
Related
- CalArena: A Large-Scale Post-Hoc Calibration Benchmark
- A Path-Dependent Agent-Based Microsimulation of Crime and Violence Reduction Policy Portfolios in Bolivia: Survey Calibration, Adaptive Emulator Ensembles, Global Sensitivity, Tempered MCMC, and Multi-Objective Policy Analysis
- CleanScore: Black-Box Benchmark Audits with Negative Controls and Sensitivity Bounds
- Why decision curves cross treat-all and treat-none: A PPV and calibration perspective.
- Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
BibTeX
@misc{sharma2026typedbench,
title = {{TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models}},
author = {Rahul Sharma and Andrew B. Duncan and Gaétan Marceau Caron and Sebastian J. Vollmer},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.11392},
eprint = {2610.11392},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.11392},
}
RIS
TY - UNPB
TI - TypedBench: A Benchmark for Calibration, Framing Sensitivity, and Cost in System One Decision Models
AU - Rahul Sharma
AU - Andrew B. Duncan
AU - Gaétan Marceau Caron
AU - Sebastian J. Vollmer
PY - 2026
DA - 2026/10/09
PB - arXiv
DO - 10.48550/arxiv.2610.11392
UR - https://arxiv.org/abs/2610.11392
AB - System One models output calibrated probabilities over typed answers such as categorical choices, ordinal levels, or binary outcomes, via a non-generative interface. Software can act on these probabilities through thresholds, cost-weighted choices, and escalation rules. Consequently, if these probabilities are miscalibrated or wording-sensitive, the software ma take unintended actions leaving human operators with no textual rationale to inspect. Current evaluations largely report accuracy and calibration on public classification datasets without a clear reference. We present TypedBench, a benchmark for typed decision models like Jev built from seven policy-labelled generators and nine evaluation suites. We report accuracy as median and range across paraphrases, and calibration error relative to the finite-sample noise floor of a matched, perfectly calibrated predictor. We assess probability quality through selective prediction, ordinal proper scoring rules, and realised cost under asymmetric cost matrices. We evaluate a hosted model, an open encoder, and a family of open decoders spanning 0.8B-9B parameters on identical items. The hosted model follows the stated policy but is wording-sensitive and systematically underconfident; under asymmetric costs, using its probabilities can be worse than taking its top answer. Decoders route exactly and are slow as options or questions are added. The decoder is least accurate on policy questions and degrades with more options. Overall, typed decision models must be evaluated jointly on policy adherence, wording robustness, probability quality, and induced decision outcomes.
ER -