Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study
- 1Department of Orthopaedics, Balgrist University Hospital, University of Zurich, 8008 Zurich, Switzerland.
Abstract
Objectives
To determine whether large language models can compute the Charlson Comorbidity Index (CCI) from unstructured, German-language diagnosis lists with reliability approaching that of expert human consensus.Materials and methods
We retrospectively analyzed the complete free-text diagnosis lists of 200 patients from a tertiary orthopedic center, stratified by American Society of Anesthesiologists (ASA) physical status I-IV. Two board-certified orthopedic surgeons and 1 chiropractor independently assigned CCI values; a consensus reading served as the reference standard. Twelve open-source models, hosted on-premise, and 5 commercial models were each evaluated with and without guideline-based prompting. Agreement was quantified by accuracy, Cohen kappa, and the coefficient of determination, and discrepancies were classified using an adapted error taxonomy.Results
Human interrater reliability was excellent (intraclass correlation 0.97). Across models, accuracy against consensus ranged from 0.34 to 0.79 and kappa from 0.27 to 0.75. The best open-source model (GPT-OSS-120B with guidelines) matched the best commercial model (coefficient of determination 0.93). Agreement declined with increasing ASA class, falling to 0.34-0.56 in ASA IV. Most discrepancies were knowledge-related (41.7%-58.3%); computation errors were rare, and 22.2%-32.0% arose from diagnosis lists too incomplete for any classification.Discussion
Performance approached expert level in low-complexity records but degraded as documentation complexity increased, indicating that record quality rather than model reasoning is the dominant constraint. Locally hosted open-source models performed comparably to proprietary systems, allowing comorbidity extraction without transferring data off-premise.Conclusion
Large language models can approximate expert CCI scoring directly from unstructured German-language clinical text, supporting scalable risk adjustment where documentation is adequate.
Related
- Harnessing the VO2 Phase Transition for Automatic Gain Control in Transimpedance Amplifiers
- Reliable and Efficient Agentic Protein Design with Toolset Harnessing
- Cross-Cultural Adaptation and Psychometric Validation of the Arabic Duruöz Hand Index: A Reliable Patient-Reported Measure for Hand Disability in Arabic-Speaking Populations
- GeneEmbedLLM: Harnessing Large Language Models for Gene Functional Embedding and Association Prediction
- Fine-Tuning Small Language Models for Reliable VASP INCAR Generation
BibTeX
@article{borcard2026harnessing,
title = {{Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study}},
author = {Lana Mei Borcard and Anna-Katharina Calek and Michael Rebsamen and Benedikt Herzog and Sebastiano Caprara and Mazda Farshad and David Ephraim Bauer},
journal = {JAMIA open},
year = {2026},
month = oct,
doi = {10.1093/jamiaopen/ooag212},
url = {https://europepmc.org/article/PMC/PMC13726996},
note = {CC BY},
}
RIS
TY - JOUR
TI - Harnessing large language models for reliable automatic Charlson Comorbidity Index scoring: a validation study
AU - Lana Mei Borcard
AU - Anna-Katharina Calek
AU - Michael Rebsamen
AU - Benedikt Herzog
AU - Sebastiano Caprara
AU - Mazda Farshad
AU - David Ephraim Bauer
PY - 2026
DA - 2026/10/01
JO - JAMIA open
DO - 10.1093/jamiaopen/ooag212
UR - https://europepmc.org/article/PMC/PMC13726996
AB - Objectives To determine whether large language models can compute the Charlson Comorbidity Index (CCI) from unstructured, German-language diagnosis lists with reliability approaching that of expert human consensus.Materials and methods We retrospectively analyzed the complete free-text diagnosis lists of 200 patients from a tertiary orthopedic center, stratified by American Society of Anesthesiologists (ASA) physical status I-IV. Two board-certified orthopedic surgeons and 1 chiropractor independently assigned CCI values; a consensus reading served as the reference standard. Twelve open-source models, hosted on-premise, and 5 commercial models were each evaluated with and without guideline-based prompting. Agreement was quantified by accuracy, Cohen kappa, and the coefficient of determination, and discrepancies were classified using an adapted error taxonomy.Results Human interrater reliability was excellent (intraclass correlation 0.97). Across models, accuracy against consensus ranged from 0.34 to 0.79 and kappa from 0.27 to 0.75. The best open-source model (GPT-OSS-120B with guidelines) matched the best commercial model (coefficient of determination 0.93). Agreement declined with increasing ASA class, falling to 0.34-0.56 in ASA IV. Most discrepancies were knowledge-related (41.7%-58.3%); computation errors were rare, and 22.2%-32.0% arose from diagnosis lists too incomplete for any classification.Discussion Performance approached expert level in low-complexity records but degraded as documentation complexity increased, indicating that record quality rather than model reasoning is the dominant constraint. Locally hosted open-source models performed comparably to proprietary systems, allowing comorbidity extraction without transferring data off-premise.Conclusion Large language models can approximate expert CCI scoring directly from unstructured German-language clinical text, supporting scalable risk adjustment where documentation is adequate.
SN - 2574-2531
ER -