Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study.
- 1Charité - University Medical Center Berlin, Dieter Scheffner Center for Medical Education and Educational Research, Dean's Office for Study Affairs, Charitéplatz 1, Berlin, 10117, Germany, 49 30450576207.
- 2Faculty of Health Sciences Brandenburg, Institute of Research in Health Sciences Education, Brandenburg Medical School, Neuruppin, Germany.
Abstract
Background
Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners' summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback. Still, their test-retest reliability and concordance with expert ratings in this context remain unclear.Objective
This study aims to evaluate the test-retest reliability-specifically, whether the models produce stable ratings across repetitions-of LLM-generated medical summary statement ratings, and to measure their concordance with human expert assessments across different prompting strategies.Methods
We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from 6 internal medicine VP cases previously rated by 2 expert raters. Statements were evaluated along 4 ternary components (semantic qualifiers, transformation, narrowing, and global rating), and 2 binary components (factual accuracy and person). Four LLMs (Anthropic's Claude, OpenAI's GPT-4o, Mistral AI's Mistral Large, and Mistral AI's Mistral 7B) were each run 5 times under 4 prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, and few-shot+CoT). The test-retest reliability was assessed with intraclass correlation coefficients (ICCs), Fleiss kappa (κ), and concordance with human ratings with accuracy (%) and chance-corrected interrater agreement (Gwet AC1 and AC2).Results
Large closed-weights models (Claude, GPT-4o, and Mistral Large) showed high test-retest reliability (ICC≥0.92; Fleiss κ=0.59-1.00). Claude was essentially deterministic (ICC and Fleiss κ≥0.99); GPT-4o and Mistral Large reached ICC≥0.97 and Fleiss κ≥0.80 except under zero-shot. Mistral 7B, a small open-weights model, had high ICCs (≥0.88) but low Fleiss κ (0.32-0.52). Few-shot+CoT prompting achieved the best concordance with human ratings for all the models (means across runs and rubric components), peaking for Mistral Large (accuracy 77%; Gwet AC1 and AC2 0.85, 95% CI 0.83-0.87), followed by Claude (71%; 0.77, 0.74-0.81), GPT-4o (67%; 0.72, 0.68-0.77), and Mistral 7B (60%; 0.64, 0.56-0.73). Concordance was higher for the binary rubric components (person and factual accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model.Conclusions
Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria in binary rubric components and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.
Related
- Comparative evaluation of large language models for esophageal cancer patient education: quality, reliability, readability, understandability, and actionability.
- Performance of large language models in postoperative hip fracture rehabilitation counseling for older adults: a comparative evaluation of safety, accuracy, reliability, readability, and empathy
- Evaluating the reliability, quality, and readability of AI-generated patient education on hallux valgus: a comparative study of large language models
- Reliability and readability of five large language models in answering patient-oriented questions about rectal prolapse: a comparative study
- Comparative Analysis of Large Language Model Outputs for Pre-Hypnotherapy Support in Irritable Bowel Syndrome: An Experimental Feasibility Study
BibTeX
@article{stadler2026testretest,
title = {{Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study.}},
author = {Patrick Stadler and Piotr Bucichowski and Ylva Holzhausen and Inga Hege and Harm Peters and Julius Josef Kaminski},
journal = {JMIR medical education},
year = {2026},
month = oct,
doi = {10.2196/94666},
url = {https://europepmc.org/article/MED/42906147},
note = {CC BY},
}
RIS
TY - JOUR
TI - Test-Retest Reliability of Large Language Models and Their Concordance With Human Experts in Rubric-Based Scoring of Clinical Summary Statements: Experimental Comparative Study.
AU - Patrick Stadler
AU - Piotr Bucichowski
AU - Ylva Holzhausen
AU - Inga Hege
AU - Harm Peters
AU - Julius Josef Kaminski
PY - 2026
DA - 2026/10/21
JO - JMIR medical education
DO - 10.2196/94666
UR - https://europepmc.org/article/MED/42906147
AB - Background Clinical summary statements convert unstructured patient information into a concise format. They are widely used in virtual patients (VPs) to train medical students in diagnostic reasoning. Most VP systems provide little individualized feedback on the quality of learners' summaries, and earlier automated approaches have required complex natural language processing, limiting their scalability. Large language models (LLMs) may offer a more flexible, rubric-guided way to score summary statements and support feedback. Still, their test-retest reliability and concordance with expert ratings in this context remain unclear.Objective This study aims to evaluate the test-retest reliability-specifically, whether the models produce stable ratings across repetitions-of LLM-generated medical summary statement ratings, and to measure their concordance with human expert assessments across different prompting strategies.Methods We conducted an experimental comparative study using 122 rubric-scored summary statements (English and German) from 6 internal medicine VP cases previously rated by 2 expert raters. Statements were evaluated along 4 ternary components (semantic qualifiers, transformation, narrowing, and global rating), and 2 binary components (factual accuracy and person). Four LLMs (Anthropic's Claude, OpenAI's GPT-4o, Mistral AI's Mistral Large, and Mistral AI's Mistral 7B) were each run 5 times under 4 prompting strategies (zero-shot, zero-shot with chain-of-thought [CoT], few-shot, and few-shot+CoT). The test-retest reliability was assessed with intraclass correlation coefficients (ICCs), Fleiss kappa (κ), and concordance with human ratings with accuracy (%) and chance-corrected interrater agreement (Gwet AC1 and AC2).Results Large closed-weights models (Claude, GPT-4o, and Mistral Large) showed high test-retest reliability (ICC≥0.92; Fleiss κ=0.59-1.00). Claude was essentially deterministic (ICC and Fleiss κ≥0.99); GPT-4o and Mistral Large reached ICC≥0.97 and Fleiss κ≥0.80 except under zero-shot. Mistral 7B, a small open-weights model, had high ICCs (≥0.88) but low Fleiss κ (0.32-0.52). Few-shot+CoT prompting achieved the best concordance with human ratings for all the models (means across runs and rubric components), peaking for Mistral Large (accuracy 77%; Gwet AC1 and AC2 0.85, 95% CI 0.83-0.87), followed by Claude (71%; 0.77, 0.74-0.81), GPT-4o (67%; 0.72, 0.68-0.77), and Mistral 7B (60%; 0.64, 0.56-0.73). Concordance was higher for the binary rubric components (person and factual accuracy) than for the ternary components. Extreme deviations and invalid outputs were rare but more frequent in ternary components and the smaller local model.Conclusions Large, general-purpose LLMs can function as reliable, rubric-guided raters of VP-based clinical summary statements, achieving high concordance with expert ratings under clearly defined criteria in binary rubric components and when using few-shot with CoT prompting. The performance on more nuanced, ternary reasoning dimensions was more variable. LLMs can support scalable, rubric-based formative feedback on clinical reasoning within VP systems but should complement rather than replace human judgment.
SN - 2369-3762
ER -