Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries
4 affiliations
- 1Department of Epidemiology and Biostatistics , University of California , San Francisco , San Francisco , California , USA
- 2Brigham and Women's Hospital , Boston , Massachusetts , USA
- 3Harvard Medical School , Boston , Massachusetts , USA
- 4Department of Biostatistics , University of Washington , Seattle , Washington , USA
Abstract
Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice. We report a blinded evaluation built on 620 Real-world Point-Of-Care Queries (Real-POCQi) submitted to the OpenEvidence (OE) platform by physicians spanning 30 specialties, as well as 187 questions from HealthBench. 149 practicing physicians across 36 states made head-to-head comparisons between answers from three frontier general-purpose models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) and a specialized clinical tool (OE), with graders matched to each question's specialty. When comparing answers along five dimensions relevant to clinical decision support -- accuracy, clinical utility, source quality, verifiability, & completeness -- physicians scored the specialized tool highest on all axes; in the primary analysis on Real-POCQi, win differences (margins between win and loss rates) ranged from 25 to 39 percentage points (p<0.001). Results remained consistent in sensitivity analyses stratifying by citation display, answer length, OE-user status, and Real-POCQi versus HealthBench. In parallel, LLM judges were found to systematically differ from expert judges, though both generally agreed on the best model. These findings underscore two conclusions: (i) AI tool evaluations should reflect real-world query distributions and use expert judges that mirror the specialization defining modern medicine and (ii) the consistent advantage of the specialized tool over general-purpose models does not necessarily mean that the latter cannot serve similar purposes, but that targeted engineering and customization can yield meaningful gains in performance for its users. We release Real-POCQi as a public benchmark, as well as the prespecified statistical analysis for reproducing results of this study.
Related
- "I Know Where to Look," But Does the LLM? Charting the Gaps Between Clinical Expert Needs and Unstructured Data Abstraction Tools
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
- Development and expert evaluation of a culturally adapted large language model for Malay clinical history-taking simulation: mixed methods study
- Evaluating large language models as clinical decision support tools in primary healthcare settings: Protocol for a multi-country comparative validation study on expert-adjudicated hypothetical vignettes (hypMOOVE-PHC)
- Clinical trials and evaluation of AI tools in solid organ transplantation: implications for clinical care, regulatory science, and rare diseases
BibTeX
@misc{feng2026expert,
title = {{Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries}},
author = {Jean Feng and Vishal Patel and PJ Heagerty and Yifan Mai and Venkatesh Sivaraman and Patrick Vossler and Jialin Ouyang and Anupam Bapu Jena},
howpublished = {arXiv (Cornell University)},
year = {2026},
month = jun,
doi = {10.48550/arxiv.2606.28960},
url = {https://arxiv.org/abs/2606.28960},
note = {CC BY},
}
RIS
TY - UNPB
TI - Expert Evaluation of Clinical AI Tools on Real Point-of-Care Clinical Queries
AU - Jean Feng
AU - Vishal Patel
AU - PJ Heagerty
AU - Yifan Mai
AU - Venkatesh Sivaraman
AU - Patrick Vossler
AU - Jialin Ouyang
AU - Anupam Bapu Jena
PY - 2026
DA - 2026/06/27
PB - arXiv (Cornell University)
DO - 10.48550/arxiv.2606.28960
UR - https://arxiv.org/abs/2606.28960
AB - Physicians now pose millions of clinical questions to AI tools each week, yet these tools are evaluated largely on hypothetical or exam-style questions, not those actually asked in practice. We report a blinded evaluation built on 620 Real-world Point-Of-Care Queries (Real-POCQi) submitted to the OpenEvidence (OE) platform by physicians spanning 30 specialties, as well as 187 questions from HealthBench. 149 practicing physicians across 36 states made head-to-head comparisons between answers from three frontier general-purpose models (Claude Opus 4.8, Gemini 3.1 Pro, and GPT-5.5) and a specialized clinical tool (OE), with graders matched to each question's specialty. When comparing answers along five dimensions relevant to clinical decision support -- accuracy, clinical utility, source quality, verifiability, & completeness -- physicians scored the specialized tool highest on all axes; in the primary analysis on Real-POCQi, win differences (margins between win and loss rates) ranged from 25 to 39 percentage points (p<0.001). Results remained consistent in sensitivity analyses stratifying by citation display, answer length, OE-user status, and Real-POCQi versus HealthBench. In parallel, LLM judges were found to systematically differ from expert judges, though both generally agreed on the best model. These findings underscore two conclusions: (i) AI tool evaluations should reflect real-world query distributions and use expert judges that mirror the specialization defining modern medicine and (ii) the consistent advantage of the specialized tool over general-purpose models does not necessarily mean that the latter cannot serve similar purposes, but that targeted engineering and customization can yield meaningful gains in performance for its users. We release Real-POCQi as a public benchmark, as well as the prespecified statistical analysis for reproducing results of this study.
SN - 2331-8422
ER -