Development and Clinical Validation of an Automated LLM Judge for Evaluating Perioperative Patient Questions
7 affiliations
- 1Division of Plastic Surgery, Mayo Clinic, 4500 San Pablo Rd S, Jacksonville, FL 32224, USA
- 2Department of Radiology AI IT, Mayo Clinic, Rochester, MN 55905, USA
- 3Center for Digital Health, Mayo Clinic, Rochester, MN 55905, USA
- 4Department of Administration, Mayo Clinic, Rochester, MN 55905, USA
- 5Division of Clinical Trials and Biostatistics, Mayo Clinic, Jacksonville, FL 32224, USA;
- 6Department of Medicine, Mayo Clinic, Rochester, MN 55905, USA
- 7Department of Urology, Mayo Clinic, Rochester, MN 55905, USA
Abstract
Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.
Related
- Development and validation of machine learning models for preoperative prediction of perioperative outcomes in patients with thymic epithelial tumors
- Development and Validation of a Perioperative Risk Update Model for Postoperative Delirium in Patients Aged ≥ 80 Years Undergoing Hip Fracture Surgery: A Retrospective Cohort Study
- Development and Internal Validation of a Perioperative Model for Predicting Early Postoperative Nausea and Vomiting After Gynecologic Laparoscopic Surgery.
- Development and internal validation of a perioperative nomogram for predicting moderate-to-severe acute postoperative pain after hepatectomy for hepatocellular carcinoma.
- Risk prediction for 12-month post-prostatectomy urinary incontinence to support survivorship care: development and same-center temporal validation of a perioperative model
BibTeX
@article{collaco2026development,
title = {{Development and Clinical Validation of an Automated LLM Judge for Evaluating Perioperative Patient Questions}},
author = {Bernardo G. Collaco and Nadia G. Wood and Yunguo Yu and Carina Rosa Malena and Srinivasagam Prabha and Ashton L. Boon and Zhihui Fang and Anjali Bhagra and Bradley C. Leibovich and Antonio Jorge Forte},
journal = {Bioengineering (Basel, Switzerland)},
year = {2026},
month = sep,
doi = {10.3390/bioengineering13101115},
url = {https://europepmc.org/article/PMC/PMC13701884},
note = {CC BY},
}
RIS
TY - JOUR
TI - Development and Clinical Validation of an Automated LLM Judge for Evaluating Perioperative Patient Questions
AU - Bernardo G. Collaco
AU - Nadia G. Wood
AU - Yunguo Yu
AU - Carina Rosa Malena
AU - Srinivasagam Prabha
AU - Ashton L. Boon
AU - Zhihui Fang
AU - Anjali Bhagra
AU - Bradley C. Leibovich
AU - Antonio Jorge Forte
PY - 2026
DA - 2026/09/01
JO - Bioengineering (Basel, Switzerland)
DO - 10.3390/bioengineering13101115
UR - https://europepmc.org/article/PMC/PMC13701884
AB - Background: Large language models (LLMs) are increasingly used in patient-facing clinical applications, creating a need for scalable methods to evaluate the accuracy, safety, and appropriateness of their responses. Although physician review remains the reference standard, it is resource-intensive and difficult to scale. Objective: To develop and clinically validate an LLM-as-a-judge framework for evaluating responses to patients’ periprocedural questions. Methods: We retrospectively evaluated 1285 patient–AIVA interactions from 112 patients across two Mayo Clinic sites and multiple surgical specialties. Physician reviewers classified interactions using a predefined true-positive (TP), false-negative (FN), true-negative (TN), and false-positive (FP) framework. The finalized LLM judge independently evaluated the same interactions. Agreement was assessed using four-class and category-specific agreement, Cohen’s kappa, and a secondary binary analysis of response correctness. Results: Overall, four-class agreement was 93.1% (95% CI, 91.7–94.5%), with Cohen’s κ = 0.852 (95% CI, 0.823–0.882). Category-specific agreement was 94.1% for TP, 90.7% for FN, 95.5% for TN, and 84.9% for FP classifications. In the binary correctness analysis, accuracy was 93.4% (95% CI, 92.0–94.7%), sensitivity 94.6%, specificity 90.2%, precision 96.2%, and F1-score 0.954. McNemar’s test demonstrated no significant asymmetry between paired classifications (p = 0.11). Conclusions: A clinically grounded LLM judge closely approximated physician evaluation of patient-facing AI responses. These findings support its potential as a scalable assistive monitoring tool while preserving physician oversight for uncertain or safety-sensitive interactions.
SN - 2306-5354
ER -