Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
Abstract
Patient-facing triage requires an appropriate care recommendation and recognition of when information is insufficient. We evaluate whether Jev, a decision model that returns probabilities over predefined actions, supports both judgments. Using CARE-Bench, we compare Jev with four general-purpose language models and three input-independent baselines across 925 decision points from 439 public cases. On the public test subset, Jev ranks fourth of five models by macro-F1 (0.461) but has the lowest observed multiclass Brier score (0.610), an 18.7\% improvement over uniform prediction (case-bootstrap 95\% interval, 4.9--31.1\%). Noul has lower binary Brier than Choice (0.215 versus 0.276). A evaluation of 453 HealthBench conversations again gives Jev the lowest observed Brier score (0.299), but measures a distinct emergency-referral judgment. Jev combines lower probability error with missed clarification needs in the four-action task. Asking about information sufficiency separately increases recall but also produces frequent unnecessary clarification assignments.
Related
- Calibrating Machine Learning Approaches for Probability Estimation Without Calibration Data.
- Probability Contracts: Accuracy, Coherence, and Decisions Across LLM Interfaces
- INLA for Adaptive UAV Sampling of Radio Environment Maps in IoT Networks: Prior Width Governs Accuracy and Calibration
- Why decision curves cross treat-all and treat-none: A PPV and calibration perspective.
- Precision follows the decision: how well an acceptance probability is known for mean-spread release rules
BibTeX
@misc{you2026native,
title = {{Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy}},
author = {Liang You and Zhiyin Zhou and Dawei Tang and Chuchu Wu and Jingzhuo Hu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.24692},
eprint = {2610.24692},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.24692},
}
RIS
TY - UNPB
TI - Native Probability Outputs for Clinical Triage: An Evaluation of Calibration and Decision Accuracy
AU - Liang You
AU - Zhiyin Zhou
AU - Dawei Tang
AU - Chuchu Wu
AU - Jingzhuo Hu
PY - 2026
DA - 2026/10/17
PB - arXiv
DO - 10.48550/arxiv.2610.24692
UR - https://arxiv.org/abs/2610.24692
AB - Patient-facing triage requires an appropriate care recommendation and recognition of when information is insufficient. We evaluate whether Jev, a decision model that returns probabilities over predefined actions, supports both judgments. Using CARE-Bench, we compare Jev with four general-purpose language models and three input-independent baselines across 925 decision points from 439 public cases. On the public test subset, Jev ranks fourth of five models by macro-F1 (0.461) but has the lowest observed multiclass Brier score (0.610), an 18.7\% improvement over uniform prediction (case-bootstrap 95\% interval, 4.9--31.1\%). Noul has lower binary Brier than Choice (0.215 versus 0.276). A evaluation of 453 HealthBench conversations again gives Jev the lowest observed Brier score (0.299), but measures a distinct emergency-referral judgment. Jev combines lower probability error with missed clarification needs in the four-action task. Asking about information sufficiency separately increases recall but also produces frequent unnecessary clarification assignments.
ER -