Skip to content
DraconiticResearch
Search Sources API
197,768 studies31 sourcesUpdated 14 min ago
Peer-reviewedCC BYNature Medicine

General-purpose large language models outperform specialized clinical AI tools on medical benchmarks

  • Krithik Vishwanath1,2,3
  • Anton Alyakin4,5,6
  • Mrigayu Ghosh7,8
  • Ali Hage9
  • Sean N. Neifert4
  • Cordelia Marcela Orillac4
  • Nataniel J. Mandelberg4
  • Hammad A. Khan4
  • Jin Vivian Lee4,5,6
  • Jie J. Yao10
  • William Robert Small11,12,13
  • Aakaash Varma14,15
4 more authors
  • Daniel Brock Hewitt16
  • Yindalon Aphinyanaphongs11,12,17
  • Daniel Alexander Alber13
  • Eric Karl Oermann18,19,20,21,22
22 affiliations
  1. 1Department of Aerospace Engineering & Engineering Mechanics, The University of Texas at Austin, Austin, TX, USA. [email protected]
  2. 2Department of Mathematics, The University of Texas at Austin, Austin, TX, USA. [email protected]
  3. 3Department of Neurological Surgery, NYU Langone Health, New York, NY, USA. [email protected]
  4. 4Department of Neurological Surgery, NYU Langone Health, New York, NY, USA
  5. 5Department of Neurosurgery, Washington University School of Medicine, St. Louis, MO, USA
  6. 6Global AI Frontier Lab, New York University, New York, NY, USA
  7. 7Department of Biomedical Engineering, The University of Texas at Austin, Austin, TX, USA
  8. 8Department of Molecular Biosciences, The University of Texas at Austin, Austin, TX, USA
  9. 9Department of Cardiothoracic Surgery, NYU Langone Health, New York, NY, USA
  10. 10Department of Orthopedic Surgery, NYU Langone Health, New York, NY, USA
  11. 11Department of MCIT Health Informatics, NYU Langone Health, New York, NY, USA
  12. 12Department of Medicine, NYU Langone Health, New York, NY, USA
  13. 13New York University (NYU) Grossman School of Medicine, New York, NY, USA
  14. 14Department of Medicine, NYU Langone Long Island, Mineola, NY, USA
  15. 15Division of Dermatology, NYU Langone Long Island, Mineola, NY, USA
  16. 16Department of Surgery, NYU Langone Health, New York, NY, USA
  17. 17Department of Population Health, NYU Langone Health, New York, NY, USA
  18. 18Center for Data Science, New York University, New York, NY, USA. [email protected]
  19. 19Department of Neurological Surgery, NYU Langone Health, New York, NY, USA. [email protected]
  20. 20Department of Radiology, NYU Langone Health, New York, NY, USA. [email protected]
  21. 21Global AI Frontier Lab, New York University, New York, NY, USA. [email protected]
  22. 22Neuroscience Institute, NYU Langone Health, New York, NY, USA. [email protected]
Open at sourcePDFDOICite

Abstract

Specialized clinical artificial intelligence (AI) tools are entering medical practice despite scarce independent evaluation. We quantitatively evaluate two clinical AI tools, OpenEvidence and UpToDate Expert AI, built on large language models (LLMs) against three frontier LLMs: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Our evaluation has three stages: (1) 500 MedQA questions testing medical knowledge, (2) 500 HealthBench items measuring alignment with clinicians and (3) the real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment. For the RCQ benchmark, 12 US clinicians performed randomized, blinded review of model outputs, producing 1,800 model-question annotations. Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ. These findings highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.

Related

  1. ClinStigma: A Clinician-Guided Framework for Developing Synthetic Benchmarks of Contextual Stigmatizing Language in Clinical Documentation
    J. A. Izzo, A. M. McIntyre, J. Nguyen +3
    medRxivPreprintCC BY-NCFull textEpidemiology & public healthmedRxiv
  2. Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
    A. Wright, S. Liu, A. P. Wright
    medRxivPreprintAll rights reservedFull textEpidemiology & public healthmedRxiv
  3. Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
    Masaki Sakamoto, Shintaro Honda, Bungo Otsuki +9
    Spine surgery and related researchPeer-reviewedCC BY-NC-NDFull textEpidemiology & public healthEurope PMC
  4. Errors, Hallucinations, and Clinical Impact of General-Purpose Multimodal Large Language Models in Histopathology
    K. Lami, S. Agarwal, A. Asaturova +20
    medRxivPreprintCC BY-NDFull textEpidemiology & public healthmedRxiv
  5. Magnitude and associated factors of delayed clinical stability among adult patients with severe community-acquired pneumonia admitted to the University of Gondar Comprehensive Specialized Hospital, Northwest Ethiopia in 2024/2025: a prospective observational study
    Getasew Muluken Abebe, Tadilo Biset Melese, Bethel Gezahegn Bogale +9
    BMJ OpenPeer-reviewedLicence not statedGeneral medicineCrossref

Cite

BibTeXRIS
BibTeX
@article{vishwanath2026generalpurpose,
  title = {{General-purpose large language models outperform specialized clinical AI tools on medical benchmarks}},
  author = {Krithik Vishwanath and Anton Alyakin and Mrigayu Ghosh and Ali Hage and Sean N. Neifert and Cordelia Marcela Orillac and Nataniel J. Mandelberg and Hammad A. Khan and Jin Vivian Lee and Jie J. Yao and William Robert Small and Aakaash Varma and Daniel Brock Hewitt and Yindalon Aphinyanaphongs and Daniel Alexander Alber and Eric Karl Oermann},
  journal = {Nature Medicine},
  year = {2026},
  month = jun,
  doi = {10.1038/s41591-026-04431-5},
  url = {https://doi.org/10.1038/s41591-026-04431-5},
  note = {CC BY},
}
RIS
TY  - JOUR
TI  - General-purpose large language models outperform specialized clinical AI tools on medical benchmarks
AU  - Krithik Vishwanath
AU  - Anton Alyakin
AU  - Mrigayu Ghosh
AU  - Ali Hage
AU  - Sean N. Neifert
AU  - Cordelia Marcela Orillac
AU  - Nataniel J. Mandelberg
AU  - Hammad A. Khan
AU  - Jin Vivian Lee
AU  - Jie J. Yao
AU  - William Robert Small
AU  - Aakaash Varma
AU  - Daniel Brock Hewitt
AU  - Yindalon Aphinyanaphongs
AU  - Daniel Alexander Alber
AU  - Eric Karl Oermann
PY  - 2026
DA  - 2026/06/12
JO  - Nature Medicine
DO  - 10.1038/s41591-026-04431-5
UR  - https://doi.org/10.1038/s41591-026-04431-5
AB  - Specialized clinical artificial intelligence (AI) tools are entering medical practice despite scarce independent evaluation. We quantitatively evaluate two clinical AI tools, OpenEvidence and UpToDate Expert AI, built on large language models (LLMs) against three frontier LLMs: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Our evaluation has three stages: (1) 500 MedQA questions testing medical knowledge, (2) 500 HealthBench items measuring alignment with clinicians and (3) the real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment. For the RCQ benchmark, 12 US clinicians performed randomized, blinded review of model outputs, producing 1,800 model-question annotations. Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ. These findings highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.
SN  - 1078-8956
SN  - 1546-170X
SN  - 1744-7933
ER  -