A decision-only model as a low-cost judge of medical AI answers
- 1BRIDGE GenAI Lab, Beth Israel Deaconess Medical Center, Harvard Medical School, Boston, MA, USA
Abstract
Background. Testing and deploying medical AI systems at scale already results in more outputs than clinicians can review, and the use of large language models (LLMs) as judges is growing rapidly. Decision-only models, cheap and fast models that return structured probabilities rather than text, offer a potentially scalable option for that goal. Methods. In an OSF pre-registered evaluation, we tested TypeSafe Jev 1.13 as a judge on tasks from five validated clinical benchmarks: rubric grading (HealthBench), planted clinical errors (When Rubrics Fail), risk in AI-written clinical documents (MedVAL), counseling ratings and failures (CounselBench) and physicians' preferences between answers (Real-POCQi). Comparators were expert raters' agreement with each other, OpenAI Decisions, GPT-6 Luna, a proposed hybrid team combining decision models and a cheap LLM and, as frontier comparators, GPT-6.1 Sol and Claude Opus 5.5. The main outcome was plain agreement with the expert label. Results. The test comprised 25,091 units and 213,913 model calls. Jev's agreement with expert labels averaged 64.7% over six tasks. On 3,614 matched units it averaged 66.1%, against 65.0% for Opus and 60.7% for Sol, and it was non-inferior to both on four of six tasks. Jev judged the whole pool in 17 minutes for US$1.46; per judgment, it cost 1/34 of Sol's price and 1/152 of Opus's. A preset cutoff held its false-alarm rate on fresh planted-error answers, and in simulation, reviewing Jev's least-sure answers first found 1.4 times the errors of a random audit. Still, omissions were hard for both: at the same false-alarm rate, Jev missed 55.6% of planted omissions and Opus 48.1%, and Opus caught 8.8 points more planted errors overall. Conclusions. Across more than 200,000 model calls, a decision-only model was non-inferior to two frontier judges on most tasks at a small fraction of their cost and time, while reporting how sure it was of every call. With local cutoffs and expert oversight of omissions and clinical documents, judging every output of a deployed medical AI system becomes more affordable.
Related
- Decision-only medical AI: early evidence and pairing with language models
- Medical students' appraisal of curated AI answers in a cardiovascular system course
- Medical and non-medical out-of-pocket cost and time burden of COVID-19 illness in U.S. children and adults: a cross-sectional survey study
- Total cost of care for long-acting injectable antipsychotics for schizophrenia: Pharmacy vs medical benefit
- Human–artificial intelligence collaboration in emergency medical services: from diagnostic algorithms to shared decision support
BibTeX
@misc{omar2026decisiononly,
title = {{A decision-only model as a low-cost judge of medical AI answers}},
author = {M. Omar and B. S. Glicksberg and A. Sakhuja and Y. Barash and A. W. Charney and E. Klang and G. N. Nadkarni},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.30.26366939},
url = {https://www.medrxiv.org/content/10.64898/2026.10.30.26366939v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - A decision-only model as a low-cost judge of medical AI answers
AU - M. Omar
AU - B. S. Glicksberg
AU - A. Sakhuja
AU - Y. Barash
AU - A. W. Charney
AU - E. Klang
AU - G. N. Nadkarni
PY - 2026
DA - 2026/10/30
PB - medRxiv
DO - 10.64898/2026.10.30.26366939
UR - https://www.medrxiv.org/content/10.64898/2026.10.30.26366939v1
AB - Background. Testing and deploying medical AI systems at scale already results in more outputs than clinicians can review, and the use of large language models (LLMs) as judges is growing rapidly. Decision-only models, cheap and fast models that return structured probabilities rather than text, offer a potentially scalable option for that goal. Methods. In an OSF pre-registered evaluation, we tested TypeSafe Jev 1.13 as a judge on tasks from five validated clinical benchmarks: rubric grading (HealthBench), planted clinical errors (When Rubrics Fail), risk in AI-written clinical documents (MedVAL), counseling ratings and failures (CounselBench) and physicians' preferences between answers (Real-POCQi). Comparators were expert raters' agreement with each other, OpenAI Decisions, GPT-6 Luna, a proposed hybrid team combining decision models and a cheap LLM and, as frontier comparators, GPT-6.1 Sol and Claude Opus 5.5. The main outcome was plain agreement with the expert label. Results. The test comprised 25,091 units and 213,913 model calls. Jev's agreement with expert labels averaged 64.7% over six tasks. On 3,614 matched units it averaged 66.1%, against 65.0% for Opus and 60.7% for Sol, and it was non-inferior to both on four of six tasks. Jev judged the whole pool in 17 minutes for US$1.46; per judgment, it cost 1/34 of Sol's price and 1/152 of Opus's. A preset cutoff held its false-alarm rate on fresh planted-error answers, and in simulation, reviewing Jev's least-sure answers first found 1.4 times the errors of a random audit. Still, omissions were hard for both: at the same false-alarm rate, Jev missed 55.6% of planted omissions and Opus 48.1%, and Opus caught 8.8 points more planted errors overall. Conclusions. Across more than 200,000 model calls, a decision-only model was non-inferior to two frontier judges on most tasks at a small fraction of their cost and time, while reporting how sure it was of every call. With local cutoffs and expert oversight of omissions and clinical documents, judging every output of a deployed medical AI system becomes more affordable.
ER -