Language models reflect clinical evidence but fail to adapt it to patients

Abstract

BackgroundClinical language models must use evidence to produce the number and action required by a particular case. Whether numerical knowledge reliably becomes a correct clinical response is unclear.

MethodsNUMBERS evaluated 16 model configurations on 1,300 questions linked to public clinical evidence. Linked experiments tested prevalence updating, patient-specific estimates and clinical actions. The direct-action experiment compared cutoff recall plus action selection with a supplied complete rule across 50 rules, five models and 7,500 calls.

ResultsAmong diagnostic estimates outside the source-result tolerance, 77.1% remained within the evidences central 80% predictive range. Models updated prevalence-dependent quantities correctly in 78.5% of comparisons with inputs and a calculation request, versus 33.7% with clinical wording. Supplying inputs and requesting calculation raised patient-level near-target answers from 29.7% to 78.8% across 2,340 pairs. Models selected an incorrect action despite stating a cutoff that implied the correct action in 413 of 3,750 recall-arm calls (11.01%; 95% confidence interval, 8.93 to 13.17). Supplying the complete rule raised action accuracy from 85.63% to 99.41%, an improvement of 13.79 percentage points (95% confidence interval, 11.65 to 15.95).

ConclusionsModels often produced evidence-consistent numbers but failed to adapt them to a case or act consistently with their own stated cutoff. Explicit inputs, calculations and complete rules substantially improved performance in controlled prompts.

FundingNational Academy of Medicine, Agreement No. 2026A008797

Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
Masaki Sakamoto, Shintaro Honda, Bungo Otsuki +9
Spine surgery and related research8 Jul 2026Peer-reviewedCC BY-NC-NDFull textEpidemiology & public healthEurope PMC
Evidence-based clinical question answering using retrieval-augmented generation (RAG) and open-source large language models
Ayesha Qasim, Jesus Jaime Moreno Escobar, Christian E. Maldonado-Sifuentes +3
Scientific Reports13 Oct 2026Peer-reviewedCC BY 4.0Full textEpidemiology & public healthCrossref
A scoping review of community engagement and implementation outcomes for approaches to adapt mental and behavioral health evidence-based practices (EBPs) for autistic youth
Jessica Tschida
PsyArXiv17 Sep 2026PreprintCC0Full text1 citedPsychiatry & mental healthOSF
No evidence of rapid antiviral clearance or clinical benefit with the combination of hydroxychloroquine and azithromycin in patients with severe COVID-19 infection
J.M. Molina, Constance Delaugerre, J. Le Goff +4
Médecine et Maladies Infectieuses Formation7 Oct 2026Peer-reviewedLicence not statedFull text747 citedInfectious diseaseHAL
Clinical referral text overrides MRI evidence in multimodal large language models: glioma versus brain abscess
N. C. Beste, F. Bauer, N. Lopez +11
medRxiv17 Oct 2026PreprintCC BY-NC-NDFull textMedical imaging & radiologymedRxiv

Cite

BibTeX RIS

BibTeX

@misc{he2026language,
  title = {{Language models reflect clinical evidence but fail to adapt it to patients}},
  author = {S. He and J. W. Joseph and P. Safari and A. Goff and P. J. Slusarz and A. Mohamed and S. Lord and J. N. Goldstein and A. S. Raja},
  howpublished = {medRxiv},
  year = {2026},
  month = sep,
  doi = {10.64898/2026.09.14.26363047},
  url = {https://www.medrxiv.org/content/10.64898/2026.09.14.26363047v1},
  note = {CC BY},
}

RIS

TY  - UNPB
TI  - Language models reflect clinical evidence but fail to adapt it to patients
AU  - S. He
AU  - J. W. Joseph
AU  - P. Safari
AU  - A. Goff
AU  - P. J. Slusarz
AU  - A. Mohamed
AU  - S. Lord
AU  - J. N. Goldstein
AU  - A. S. Raja
PY  - 2026
DA  - 2026/09/14
PB  - medRxiv
DO  - 10.64898/2026.09.14.26363047
UR  - https://www.medrxiv.org/content/10.64898/2026.09.14.26363047v1
AB  - BackgroundClinical language models must use evidence to produce the number and action required by a particular case. Whether numerical knowledge reliably becomes a correct clinical response is unclear.  MethodsNUMBERS evaluated 16 model configurations on 1,300 questions linked to public clinical evidence. Linked experiments tested prevalence updating, patient-specific estimates and clinical actions. The direct-action experiment compared cutoff recall plus action selection with a supplied complete rule across 50 rules, five models and 7,500 calls.  ResultsAmong diagnostic estimates outside the source-result tolerance, 77.1% remained within the evidences central 80% predictive range. Models updated prevalence-dependent quantities correctly in 78.5% of comparisons with inputs and a calculation request, versus 33.7% with clinical wording. Supplying inputs and requesting calculation raised patient-level near-target answers from 29.7% to 78.8% across 2,340 pairs. Models selected an incorrect action despite stating a cutoff that implied the correct action in 413 of 3,750 recall-arm calls (11.01%; 95% confidence interval, 8.93 to 13.17). Supplying the complete rule raised action accuracy from 85.63% to 99.41%, an improvement of 13.79 percentage points (95% confidence interval, 11.65 to 15.95).  ConclusionsModels often produced evidence-consistent numbers but failed to adapt them to a case or act consistently with their own stated cutoff. Explicit inputs, calculations and complete rules substantially improved performance in controlled prompts.  FundingNational Academy of Medicine, Agreement No. 2026A008797
ER  -

Abstract

Related

Cite