Precision Evidence Bench: Assessing PICO alignment and faithful reporting in clinical AI
- 1Stanford University
Abstract
Clinicians routinely make decisions for patients whose comorbidities, prior therapies, laboratory abnormalities, or demographic characteristics are not reflected in the supporting evidence, even though these factors can influence treatment selection. Given this gap in the evidence base, we sought to evaluate how AI models respond to clinical questions in which patient characteristics or the treatment comparison and outcome of interest may affect the applicability of available evidence. We developed and publicly released Precision Evidence Bench, a benchmark of 209 synthetic clinical questions with specified population, intervention, comparator, and outcome (PICO) elements. Each answer is assessed using a five-item rubric that evaluates whether a single cited source matches the PICO and whether the answer faithfully reports its sources. The item verdicts determine one of five ratings, ranging from Precision match to Unsupported claim, without requiring a reference answer. We evaluated four frontier large language models (LLMs) using web search across all 209 questions, then repeated the evaluation with additional access to a corpus of precision evidence-based findings (pEBFs) designed to address these evidence gaps, using two corpus sizes. For every model, Precision match rates increased with the amount of precision evidence available, from 13 to 18% with web search alone to 17 to 28% with 100 million pEBFs and 41 to 56% with 500 million pEBFs. These findings suggest that access to matching evidence is an important determinant of LLM performance on precision medicine questions.
Related
- Assessing selective outcome reporting in sleep disorder trials: A registry-publication comparison study
- OncoGen.AI: an integrated platform for automated genomic analysis and reporting in precision oncology
- Definitions and reporting of 'failure' after rotator cuff repair: A systematic review of level I and level II evidence and proposal for standardisation.
- Adverse event reporting in healthcare-Regulatory frameworks, institutional governance, and economic implications with evidence from Poland.
- A Pragmatic Study of Brief, Low-Effort Implementation Strategies to Increase Reporting of Evidence-Based Practice Use in a Community Mental Health Program Implementing Multiple EBPs
BibTeX
@misc{mukerji2026precision,
title = {{Precision Evidence Bench: Assessing PICO alignment and faithful reporting in clinical AI}},
author = {A. Mukerji and N. Sanghavi and T. Lauritzen and J. Muralidharan and S. Gombar},
howpublished = {medRxiv},
year = {2026},
month = oct,
doi = {10.64898/2026.10.07.26364976},
url = {https://www.medrxiv.org/content/10.64898/2026.10.07.26364976v1},
note = {CC BY-NC},
}
RIS
TY - UNPB
TI - Precision Evidence Bench: Assessing PICO alignment and faithful reporting in clinical AI
AU - A. Mukerji
AU - N. Sanghavi
AU - T. Lauritzen
AU - J. Muralidharan
AU - S. Gombar
PY - 2026
DA - 2026/10/07
PB - medRxiv
DO - 10.64898/2026.10.07.26364976
UR - https://www.medrxiv.org/content/10.64898/2026.10.07.26364976v1
AB - Clinicians routinely make decisions for patients whose comorbidities, prior therapies, laboratory abnormalities, or demographic characteristics are not reflected in the supporting evidence, even though these factors can influence treatment selection. Given this gap in the evidence base, we sought to evaluate how AI models respond to clinical questions in which patient characteristics or the treatment comparison and outcome of interest may affect the applicability of available evidence. We developed and publicly released Precision Evidence Bench, a benchmark of 209 synthetic clinical questions with specified population, intervention, comparator, and outcome (PICO) elements. Each answer is assessed using a five-item rubric that evaluates whether a single cited source matches the PICO and whether the answer faithfully reports its sources. The item verdicts determine one of five ratings, ranging from Precision match to Unsupported claim, without requiring a reference answer. We evaluated four frontier large language models (LLMs) using web search across all 209 questions, then repeated the evaluation with additional access to a corpus of precision evidence-based findings (pEBFs) designed to address these evidence gaps, using two corpus sizes. For every model, Precision match rates increased with the amount of precision evidence available, from 13 to 18% with web search alone to 17 to 28% with 100 million pEBFs and 41 to 56% with 500 million pEBFs. These findings suggest that access to matching evidence is an important determinant of LLM performance on precision medicine questions.
ER -