Cross-paradigm audit of large language model judges for clinical text evaluation
10 affiliations
- 1Institute for Transfusion Medicine, University Hospital Essen, Essen, Germany
- 2Department of Computer Science, University of Applied Sciences and Arts Dortmund (FHDO), Dortmund, Germany
- 3Institute for Artificial Intelligence in Medicine (IKIM), University Hospital Essen, Essen, Germany
- 4Institute of Diagnostic and Interventional Radiology and Neuroradiology, University Hospital Essen, Essen, Germany
- 5Central IT Department, Data Integration Center, University Hospital Essen, Essen, Germany
- 6West German Cancer Center (WTZ), University Hospital Essen, Essen, Germany
- 7Department of Medical Oncology, University Hospital Essen, Essen, Germany
- 8Department Nursing Development and Nursing Research, University Hospital Essen, Essen, Germany
- 9Institute for Medical Informatics, Biometry and Epidemiology (IMIBE), University Hospital Essen, Essen, Germany
- 10National Center for Tumor Diseases (NCT), NCT West, Essen, Germany
Abstract
Large language models are increasingly being used to evaluate clinical texts, a concept known as LLM-as-a-judge. However, most of the existing evidence focuses on English data and uses pooled agreement metrics that obscure clinically relevant failure modes. This study audited six open-weight models against expert judges on four German clinical datasets covering report and nursing documentation summarization, discharge summary generation, and patient letter generation. The models were evaluated across three paradigms and four prompt configurations. Alignment with experts varied by criterion and corpus, and no configuration proved reliable across paradigms. All judges exhibited position and identity bias. Substituting LLM-as-a-judge for the expert panel shifted 57.2% of pointwise categories, swapped 41.7% of pairwise winners, missed 10.2% of clinical errors, and inverted clinical priorities by overrating the criteria succinct and accurate. Therefore, LLM-as-a-judge is not a safe substitute for experts in German clinical text evaluations, and deployment thresholds should follow decision-impact analysis for LLM-as-a-judge in clinical contexts.
Related
- Comparative Evaluation of Large Language Model Interfaces in Third Molar Surgery Complication Scenarios: Response Quality, Clinical Content, Potential Clinical Risk, Readability, and Externally Observable Response Latency—A Cross-Sectional Comparative Benchmark Study
- Cross-Cultural Adaptation and Initial Psychometric Evaluation of the Kazakh-Language Sino-Nasal Outcome Test-22 (SNOT-22) in Patients with Chronic Rhinosinusitis with Nasal Polyps
- Evaluation of Large Language Model Concordance With AAOS Clinical Practice Guidelines for Carpal Tunnel Syndrome
- Nutrient Calculation Accuracy, Expert-Rated Clinical Appropriateness and Safety Indicators, and Reproducibility of Large Language Model-Generated Diet Plans for Polyendocrine Metabolic Ovarian Syndrome: A Real-Case-Based Evaluation
- Development and expert evaluation of a culturally adapted large language model for Malay clinical history-taking simulation: mixed methods study
BibTeX
@article{pakull2026crossparadigm,
title = {{Cross-paradigm audit of large language model judges for clinical text evaluation}},
author = {Tabea M. G. Pakull and Sina Warmer and Kamyar Arzideh and Thomas Hilser and Bernadette Hosters and Hendrik Damm and Felix Nensa and Martin Schuler and Katarzyna Borys and Peter A. Horn and René Hosch and Christoph M. Friedrich},
journal = {NPJ Digital Medicine},
year = {2026},
month = nov,
doi = {10.1038/s41746-026-03443-2},
url = {https://doi.org/10.1038/s41746-026-03443-2},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Cross-paradigm audit of large language model judges for clinical text evaluation
AU - Tabea M. G. Pakull
AU - Sina Warmer
AU - Kamyar Arzideh
AU - Thomas Hilser
AU - Bernadette Hosters
AU - Hendrik Damm
AU - Felix Nensa
AU - Martin Schuler
AU - Katarzyna Borys
AU - Peter A. Horn
AU - René Hosch
AU - Christoph M. Friedrich
PY - 2026
DA - 2026/11/04
JO - NPJ Digital Medicine
DO - 10.1038/s41746-026-03443-2
UR - https://doi.org/10.1038/s41746-026-03443-2
AB - Large language models are increasingly being used to evaluate clinical texts, a concept known as LLM-as-a-judge. However, most of the existing evidence focuses on English data and uses pooled agreement metrics that obscure clinically relevant failure modes. This study audited six open-weight models against expert judges on four German clinical datasets covering report and nursing documentation summarization, discharge summary generation, and patient letter generation. The models were evaluated across three paradigms and four prompt configurations. Alignment with experts varied by criterion and corpus, and no configuration proved reliable across paradigms. All judges exhibited position and identity bias. Substituting LLM-as-a-judge for the expert panel shifted 57.2% of pointwise categories, swapped 41.7% of pairwise winners, missed 10.2% of clinical errors, and inverted clinical priorities by overrating the criteria succinct and accurate. Therefore, LLM-as-a-judge is not a safe substitute for experts in German clinical text evaluations, and deployment thresholds should follow decision-impact analysis for LLM-as-a-judge in clinical contexts.
SN - 2398-6352
ER -