Jev in Medicine: A Benchmark Evaluation
- 1Independent researcher Madrid , Spain
Abstract
Background: Jev is a non-generative System One model that assigns probabilities to predefined answer options and cannot answer outside them. Its calibration and its performance on complex diagnostic cases remain insufficiently characterised. Methods: We evaluated Jev 1.13 on four medical benchmarks: MetaMedQA (1,373 examination questions), PubMedQA (500 research questions), DiagnosisArena-MCQ (915 cases) and the NEJM Case Challenges (34 cases). GPT-6 Sol, with and without reasoning, was the reference. Outcomes were top-1 accuracy (primary), calibration, selective prediction and recognition of unanswerable questions Results: Jev's accuracy was similar to that of GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%), lower on MetaMedQA (74.8% vs 82.7%) and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%) and the NEJM cases (61.8% vs 82.4%, adjusted P = 0.078). On MetaMedQA, Jev's probabilities were the best calibrated (expected calibration error 0.063 vs 0.146), and its answers with a probability ≥0.9 (52.9% of questions) were 93.4% accurate, but GPT-6 Sol was as accurate when it accepted a similar proportion of questions. On DiagnosisArena-MCQ, Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768). Of the 162 questions whose correct answer was "I don't know or cannot answer", Jev chose that option for 10.5% (GPT-6 Sol, 8.6%). Median latency was 0.27–0.31 s; all 2,823 items cost US$0.08. Conclusions: Jev was fast and inexpensive, and its accuracy was similar to that of a frontier LLM on research abstracts but lower on examination questions and much lower on complex diagnostic cases. Its better calibration on examination questions gave no selective-prediction advantage, and it seldom chose "I don't know" when that was the correct answer. Task-specific validation is required before clinical use.
Related
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- From examinations to consultations: a physician-reviewed Japanese benchmark for large language models in traditional Kampo medicine
- Evaluation of treatment preferences among orthopedic surgeons and physical medicine and rehabilitation specialists for lateral epicondylitis : A cross-sectional survey study
- PsyEval: a comprehensive large language model evaluation benchmark for mental health
- Safety boundary maintenance in consumer AI systems responding to pediatric health queries: a cross-platform benchmark evaluation under naturalistic and adversarially pressured conditions
BibTeX
@misc{madridgarcia2026medicine,
title = {{Jev in Medicine: A Benchmark Evaluation}},
author = {Alfredo Madrid-García and Beatriz Merino Barbancho},
howpublished = {arXiv (Cornell University)},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.34024},
url = {https://arxiv.org/abs/2609.34024},
note = {CC BY},
}
RIS
TY - UNPB
TI - Jev in Medicine: A Benchmark Evaluation
AU - Alfredo Madrid-García
AU - Beatriz Merino Barbancho
PY - 2026
DA - 2026/09/27
PB - arXiv (Cornell University)
DO - 10.48550/arxiv.2609.34024
UR - https://arxiv.org/abs/2609.34024
AB - Background: Jev is a non-generative System One model that assigns probabilities to predefined answer options and cannot answer outside them. Its calibration and its performance on complex diagnostic cases remain insufficiently characterised. Methods: We evaluated Jev 1.13 on four medical benchmarks: MetaMedQA (1,373 examination questions), PubMedQA (500 research questions), DiagnosisArena-MCQ (915 cases) and the NEJM Case Challenges (34 cases). GPT-6 Sol, with and without reasoning, was the reference. Outcomes were top-1 accuracy (primary), calibration, selective prediction and recognition of unanswerable questions Results: Jev's accuracy was similar to that of GPT-6 Sol with medium reasoning on PubMedQA (78.4% vs 78.2%), lower on MetaMedQA (74.8% vs 82.7%) and much lower on DiagnosisArena-MCQ (59.8% vs 82.4%) and the NEJM cases (61.8% vs 82.4%, adjusted P = 0.078). On MetaMedQA, Jev's probabilities were the best calibrated (expected calibration error 0.063 vs 0.146), and its answers with a probability ≥0.9 (52.9% of questions) were 93.4% accurate, but GPT-6 Sol was as accurate when it accepted a similar proportion of questions. On DiagnosisArena-MCQ, Jev's probabilities discriminated poorly (AUROC 0.645 vs 0.768). Of the 162 questions whose correct answer was "I don't know or cannot answer", Jev chose that option for 10.5% (GPT-6 Sol, 8.6%). Median latency was 0.27–0.31 s; all 2,823 items cost US$0.08. Conclusions: Jev was fast and inexpensive, and its accuracy was similar to that of a frontier LLM on research abstracts but lower on examination questions and much lower on complex diagnostic cases. Its better calibration on examination questions gave no selective-prediction advantage, and it seldom chose "I don't know" when that was the correct answer. Task-specific validation is required before clinical use.
SN - 2331-8422
ER -