Can a General-Purpose Coding Agent Analyze a Production Hospital Data Warehouse?
- 1Stanford University - School of Medicine
Abstract
BackgroundHealth systems answer most questions by having expert analysts hand-write queries against a complex electronic health record data warehouse, a slow, resource-intensive process. Whether an autonomous coding agent can do this accurately is unknown.
MethodsIn a single-center quality-improvement evaluation, we posed ten questions about a common pediatric infection, acute otitis media. The questions went to analysts, whose adjudicated answers were the reference, and to an autonomous coding agent (OpenAI Codex), which wrote and ran read-only queries on a full copy of the production data warehouse (Epic Caboodle). We ran the agent under four conditions: an autonomous baseline (each question answered three times at two reasoning-effort settings), a variant in which it listed its assumptions, interactive analyst feedback, and reuse of a corrected definition across related questions. Outcomes were accuracy, patient-level agreement (F1), reproducibility, and efficiency (time and tokens).
ResultsWorking autonomously, the agent wrote valid queries and never fabricated data, but rarely produced the exact answer. At medium effort, it came within 5% of the reference on 27 of 30 runs but exact on only 10. Higher effort produced no improvement. Reproducibility was the greater weakness, with the three runs returning an identical answer on only 3 of 10 questions. When a run matched the reference, it had found the same patients (F1 = 1.00); the exception was an over-counted procedure (F1 = 0.72). With analyst feedback on four questions, it answered two exactly, and reusing the corrected definition on related questions restored reproducibility and accuracy.
ConclusionsA general-purpose coding agent matched an adjudicated analyst reference on most routine questions but not reproducibly, because key definitions depended on warehouse knowledge that the data dictionary omits. Supplying that knowledge, by prompt or feedback, restored reproducibility. Under expert analyst supervision, the agent is already a capable drafting aid and a promising step toward broader hospital analytic support.
Related
- Extracting smoking history from clinical notes for lung cancer screening decision support: comparing a structured-judgment model with general-purpose large language models
- A focused PubMed artificial intelligence search program creates better literature reviews than general purpose artificial intelligence and is comparable to trainees in shoulder and elbow surgery
- Reply to: "Evaluating the Clinical Readiness of General-Purpose Large Language Models in Evidence Synthesis".
- General-purpose large language models outperform specialized clinical AI tools on medical benchmarks
- Comparative Evaluation of a System One Model and a General-Purpose Large Language Model on the Korean Physical Therapist Licensing Examination
BibTeX
@misc{marshall2026generalpurpose,
title = {{Can a General-Purpose Coding Agent Analyze a Production Hospital Data Warehouse?}},
author = {N. P. Marshall and W. Haberkorn and J. Faulkenberry and G. Palakkattil and F. Nateghi Haredasht and H. T. Schwenk and J. H. Chen and K. Morse},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.02.26362008},
url = {https://www.medrxiv.org/content/10.64898/2026.09.02.26362008v1},
note = {CC BY},
}
RIS
TY - UNPB
TI - Can a General-Purpose Coding Agent Analyze a Production Hospital Data Warehouse?
AU - N. P. Marshall
AU - W. Haberkorn
AU - J. Faulkenberry
AU - G. Palakkattil
AU - F. Nateghi Haredasht
AU - H. T. Schwenk
AU - J. H. Chen
AU - K. Morse
PY - 2026
DA - 2026/09/02
PB - medRxiv
DO - 10.64898/2026.09.02.26362008
UR - https://www.medrxiv.org/content/10.64898/2026.09.02.26362008v1
AB - BackgroundHealth systems answer most questions by having expert analysts hand-write queries against a complex electronic health record data warehouse, a slow, resource-intensive process. Whether an autonomous coding agent can do this accurately is unknown. MethodsIn a single-center quality-improvement evaluation, we posed ten questions about a common pediatric infection, acute otitis media. The questions went to analysts, whose adjudicated answers were the reference, and to an autonomous coding agent (OpenAI Codex), which wrote and ran read-only queries on a full copy of the production data warehouse (Epic Caboodle). We ran the agent under four conditions: an autonomous baseline (each question answered three times at two reasoning-effort settings), a variant in which it listed its assumptions, interactive analyst feedback, and reuse of a corrected definition across related questions. Outcomes were accuracy, patient-level agreement (F1), reproducibility, and efficiency (time and tokens). ResultsWorking autonomously, the agent wrote valid queries and never fabricated data, but rarely produced the exact answer. At medium effort, it came within 5% of the reference on 27 of 30 runs but exact on only 10. Higher effort produced no improvement. Reproducibility was the greater weakness, with the three runs returning an identical answer on only 3 of 10 questions. When a run matched the reference, it had found the same patients (F1 = 1.00); the exception was an over-counted procedure (F1 = 0.72). With analyst feedback on four questions, it answered two exactly, and reusing the corrected definition on related questions restored reproducibility and accuracy. ConclusionsA general-purpose coding agent matched an adjudicated analyst reference on most routine questions but not reproducibly, because key definitions depended on warehouse knowledge that the data dictionary omits. Supplying that knowledge, by prompt or feedback, restored reproducibility. Under expert analyst supervision, the agent is already a capable drafting aid and a promising step toward broader hospital analytic support.
ER -