{
  "abstract": "BackgroundClinical language models must use evidence to produce the number and action required by a particular case. Whether numerical knowledge reliably becomes a correct clinical response is unclear.\n\nMethodsNUMBERS evaluated 16 model configurations on 1,300 questions linked to public clinical evidence. Linked experiments tested prevalence updating, patient-specific estimates and clinical actions. The direct-action experiment compared cutoff recall plus action selection with a supplied complete rule across 50 rules, five models and 7,500 calls.\n\nResultsAmong diagnostic estimates outside the source-result tolerance, 77.1% remained within the evidences central 80% predictive range. Models updated prevalence-dependent quantities correctly in 78.5% of comparisons with inputs and a calculation request, versus 33.7% with clinical wording. Supplying inputs and requesting calculation raised patient-level near-target answers from 29.7% to 78.8% across 2,340 pairs. Models selected an incorrect action despite stating a cutoff that implied the correct action in 413 of 3,750 recall-arm calls (11.01%; 95% confidence interval, 8.93 to 13.17). Supplying the complete rule raised action accuracy from 85.63% to 99.41%, an improvement of 13.79 percentage points (95% confidence interval, 11.65 to 15.95).\n\nConclusionsModels often produced evidence-consistent numbers but failed to adapt them to a case or act consistently with their own stated cutoff. Explicit inputs, calculations and complete rules substantially improved performance in controlled prompts.\n\nFundingNational Academy of Medicine, Agreement No. 2026A008797",
  "abstract_shown": true,
  "affiliations": [
    "Division of AI, Department of Emergency Medicine, Mass General Brigham"
  ],
  "article_type": "PUBLISHAHEADOFPRINT",
  "arxiv": null,
  "author_total": 9,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "S. He"
    },
    {
      "name": "J. W. Joseph"
    },
    {
      "name": "P. Safari"
    },
    {
      "name": "A. Goff"
    },
    {
      "name": "P. J. Slusarz"
    },
    {
      "name": "A. Mohamed"
    },
    {
      "name": "S. Lord"
    },
    {
      "name": "J. N. Goldstein"
    },
    {
      "name": "A. S. Raja"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/3aqjmey3s3jgx.bib",
  "cited_by": 0,
  "doi": "10.64898/2026.09.14.26363047",
  "fulltext_url": "https://www.medrxiv.org/content/10.64898/2026.09.14.26363047v1.full",
  "id": "3aqjmey3s3jgx",
  "issns": [],
  "landing_url": "https://www.medrxiv.org/content/10.64898/2026.09.14.26363047v1",
  "language": null,
  "licence": "CC-BY",
  "licence_label": "CC BY",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [
    [
      "medrxiv",
      "health informatics"
    ],
    [
      "openalex",
      "2718|Artificial Intelligence in Healthcare and Education"
    ]
  ],
  "openalex": "W7213341994",
  "pdf_url": "https://www.medrxiv.org/content/10.64898/2026.09.14.26363047v1.full.pdf",
  "pmcid": null,
  "pmid": null,
  "published": "2026-09-14",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/3aqjmey3s3jgx.ris",
  "sources": [
    {
      "first_seen": "2026-10-30T12:18:39Z",
      "native_id": "10.64898/2026.09.14.26363047",
      "source": "medrxiv",
      "url": "https://www.medrxiv.org/content/10.64898/2026.09.14.26363047v1"
    }
  ],
  "subjects": [
    {
      "field": "medicine",
      "origin": "medrxiv:health informatics",
      "subfield": "epidemiology-public-health"
    }
  ],
  "title": "Language models reflect clinical evidence but fail to adapt it to patients",
  "title_text": "Language models reflect clinical evidence but fail to adapt it to patients",
  "type": "preprint",
  "updated_at": "2026-10-30T12:18:39Z",
  "url": "https://research.draconitic.com/paper/3aqjmey3s3jgx",
  "venue": "medRxiv",
  "versions": []
}