{ "abstract": "Specialized clinical artificial intelligence (AI) tools are entering medical practice despite scarce independent evaluation. We quantitatively evaluate two clinical AI tools, OpenEvidence and UpToDate Expert AI, built on large language models (LLMs) against three frontier LLMs: GPT-5.2, Gemini 3.1 Pro and Claude Opus 4.6. Our evaluation has three stages: (1) 500 MedQA questions testing medical knowledge, (2) 500 HealthBench items measuring alignment with clinicians and (3) the real clinical queries (RCQ) benchmark, built from 100 de-identified queries from physicians to a general-purpose language model in a live clinical environment. For the RCQ benchmark, 12 US clinicians performed randomized, blinded review of model outputs, producing 1,800 model-question annotations. Frontier LLMs outperformed clinical AI tools in all three evaluations. Clinical AI tools performed comparably to auto-enabled Google Search AI Overview on the RCQ. These findings highlight the need for independent, real-world evaluation of AI tools before they enter clinical settings.", "abstract_shown": true, "affiliations": [ "Department of Aerospace Engineering & Engineering Mechanics, The University of Texas at Austin, Austin, TX, USA. krithik.vish@utexas.edu", "Department of Mathematics, The University of Texas at Austin, Austin, TX, USA. krithik.vish@utexas.edu", "Department of Neurological Surgery, NYU Langone Health, New York, NY, USA. krithik.vish@utexas.edu", "Department of Neurological Surgery, NYU Langone Health, New York, NY, USA", "Department of Neurosurgery, Washington University School of Medicine, St. Louis, MO, USA", "Global AI Frontier Lab, New York University, New York, NY, USA", "Department of Biomedical Engineering, The University of Texas at Austin, Austin, TX, USA", "Department of Molecular Biosciences, The University of Texas at Austin, Austin, TX, USA", "Department of Cardiothoracic Surgery, NYU Langone Health, New York, NY, USA", "Department of Orthopedic Surgery, NYU Langone Health, New York, NY, USA", "Department of MCIT Health Informatics, NYU Langone Health, New York, NY, USA", "Department of Medicine, NYU Langone Health, New York, NY, USA", "New York University (NYU) Grossman School of Medicine, New York, NY, USA", "Department of Medicine, NYU Langone Long Island, Mineola, NY, USA", "Division of Dermatology, NYU Langone Long Island, Mineola, NY, USA", "Department of Surgery, NYU Langone Health, New York, NY, USA", "Department of Population Health, NYU Langone Health, New York, NY, USA", "Center for Data Science, New York University, New York, NY, USA. eric.oermann@nyulangone.org", "Department of Neurological Surgery, NYU Langone Health, New York, NY, USA. eric.oermann@nyulangone.org", "Department of Radiology, NYU Langone Health, New York, NY, USA. eric.oermann@nyulangone.org", "Global AI Frontier Lab, New York University, New York, NY, USA. eric.oermann@nyulangone.org", "Neuroscience Institute, NYU Langone Health, New York, NY, USA. eric.oermann@nyulangone.org" ], "article_type": null, "arxiv": null, "author_total": 16, "authors": [ { "affs": [ 0, 1, 2 ], "name": "Krithik Vishwanath", "orcid": "0009-0000-6894-1796" }, { "affs": [ 3, 4, 5 ], "name": "Anton Alyakin" }, { "affs": [ 6, 7 ], "name": "Mrigayu Ghosh", "orcid": "0009-0001-2310-3411" }, { "affs": [ 8 ], "name": "Ali Hage", "orcid": "0000-0002-9575-5566" }, { "affs": [ 3 ], "name": "Sean N. Neifert", "orcid": "0000-0001-9298-8295" }, { "affs": [ 3 ], "name": "Cordelia Marcela Orillac" }, { "affs": [ 3 ], "name": "Nataniel J. Mandelberg", "orcid": "0000-0002-9583-8556" }, { "affs": [ 3 ], "name": "Hammad A. Khan", "orcid": "0000-0001-5098-0463" }, { "affs": [ 3, 4, 5 ], "name": "Jin Vivian Lee", "orcid": "0000-0002-8961-0125" }, { "affs": [ 9 ], "name": "Jie J. Yao", "orcid": "0000-0002-9950-5273" }, { "affs": [ 10, 11, 12 ], "name": "William Robert Small", "orcid": "0009-0007-3105-6495" }, { "affs": [ 13, 14 ], "name": "Aakaash Varma", "orcid": "0009-0001-0016-7965" }, { "affs": [ 15 ], "name": "Daniel Brock Hewitt", "orcid": "0000-0001-6556-3830" }, { "affs": [ 10, 11, 16 ], "name": "Yindalon Aphinyanaphongs", "orcid": "0000-0001-8605-5392" }, { "affs": [ 12 ], "name": "Daniel Alexander Alber", "orcid": "0000-0001-7957-5170" }, { "affs": [ 17, 18, 19, 20, 21 ], "name": "Eric Karl Oermann", "orcid": "0000-0002-1876-5963" } ], "bibtex": "https://research.draconitic.com/paper/65cvnf8jpvjyf.bib", "cited_by": 81, "doi": "10.1038/s41591-026-04431-5", "fulltext_url": null, "id": "65cvnf8jpvjyf", "issns": [ "1078-8956", "1546-170X", "1744-7933" ], "landing_url": "https://doi.org/10.1038/s41591-026-04431-5", "language": "en", "licence": "CC-BY", "licence_label": "CC BY", "licence_source": "journal", "licence_url": "https://creativecommons.org/licenses/by/4.0/", "metadata_licence": null, "native_subjects": [ [ "openalex", "2718|Artificial Intelligence in Healthcare and Education" ] ], "openalex": "W7164566809", "pdf_url": "https://www.nature.com/articles/s41591-026-04431-5.pdf", "pmcid": "PMC13375586", "pmid": "42286322", "published": "2026-06-12", "publisher": "Nature Portfolio", "ris": "https://research.draconitic.com/paper/65cvnf8jpvjyf.ris", "sources": [ { "first_seen": "2026-10-30T13:26:54Z", "native_id": "W7164566809", "source": "openalex", "url": "https://doi.org/10.1038/s41591-026-04431-5" } ], "subjects": [ { "field": "medicine", "origin": "openalex:2718|Artificial Intelligence in Healthcare and Education", "subfield": "epidemiology-public-health" } ], "title": "General-purpose large language models outperform specialized clinical AI tools on medical benchmarks", "title_text": "General-purpose large language models outperform specialized clinical AI tools on medical benchmarks", "type": "peer-reviewed", "updated_at": "2026-11-08T13:20:50Z", "url": "https://research.draconitic.com/paper/65cvnf8jpvjyf", "venue": "Nature Medicine", "versions": [] }