Gold-standard-guided direct preference optimisation for ICD-10-AM and ACHI code assignment from discharge summaries.
- 1School of Computer, Data and Mathematical Sciences, Western Sydney University, Parramatta, NSW, 2116, Australia. [email protected].
- 2School of Computer, Data and Mathematical Sciences, Western Sydney University, Parramatta, NSW, 2116, Australia.
- 3School of Computer Science and Engineering, University of New South Wales, Kensington, NSW, 2052, Australia.
Abstract
Clinical codes assigned from discharge summaries support clinical research, reimbursement, health resource allocation, and service planning. In Australia, ICD-10-AM and ACHI are used to code diagnoses and interventions for admitted episodes of care, but manual coding is time-consuming, labour-intensive, and subject to variability. This study investigates large language models for automated ICD-10-AM and ACHI code assignment from discharge summaries. We first normalised a public ICD-9-CM coded dataset into the Australian coding system using General Equivalence Mappings and semantic similarity over code descriptions, with validation by a domain expert clinical coder. We then fine-tuned Meta-Llama-3-8B and compared full-parameter fine-tuning with a parameter-efficient QLoRA baseline. Finally, we applied Direct Preference Optimisation (DPO) using gold-standard-guided preference pairs, where the gold-standard ICD-10-AM/ACHI code list was treated as the preferred response and the supervised fine-tuned model output as the non-preferred response. Experiments were conducted on a mapped MIMIC-III subset and a de-identified Australian hospital dataset. Among the evaluated model variants, the DPO-aligned full-parameter model achieved the highest score, with micro-F1 scores of 0.7397 on the mapped dataset and 0.7834 on the Australian hospital dataset. These findings suggest that gold-standard-guided DPO can improve agreement with reference code sets compared with supervised fine-tuning alone. However, the results are limited to high-frequency diagnosis and procedure codes, short-document evaluation settings, and DPO preference pairs reviewed by one annotator.
Related
- Structured extraction from Russian/Kazakh discharge summaries: three model families at the human-agreement ceiling
- Large Language Models' Simulation of Patient Comprehension of Discharge Summaries: Multimodel Evaluation Study.
- AI-assisted identification of potential treatment injury from inpatient discharge summaries: an early-stage sociotechnical evaluation in a New Zealand district hospital
- Long COVID undercoding and prediction from releasable electronic health record summaries in All of Us
- Staging and Grading Advanced Periodontitis with Large Language Models from Standardized Case Summaries: A Multicenter Concordance and Reproducibility Study
BibTeX
@article{kaur2026goldstandardguided,
title = {{Gold-standard-guided direct preference optimisation for ICD-10-AM and ACHI code assignment from discharge summaries.}},
author = {Rajvir Kaur and Jeewani Anupama Ginige and Oliver Obst},
journal = {BMC medical informatics and decision making},
year = {2026},
month = jul,
doi = {10.1186/s12911-026-03631-w},
url = {https://europepmc.org/article/MED/42902234},
note = {CC BY-NC-ND},
}
RIS
TY - JOUR
TI - Gold-standard-guided direct preference optimisation for ICD-10-AM and ACHI code assignment from discharge summaries.
AU - Rajvir Kaur
AU - Jeewani Anupama Ginige
AU - Oliver Obst
PY - 2026
DA - 2026/07/01
JO - BMC medical informatics and decision making
DO - 10.1186/s12911-026-03631-w
UR - https://europepmc.org/article/MED/42902234
AB - Clinical codes assigned from discharge summaries support clinical research, reimbursement, health resource allocation, and service planning. In Australia, ICD-10-AM and ACHI are used to code diagnoses and interventions for admitted episodes of care, but manual coding is time-consuming, labour-intensive, and subject to variability. This study investigates large language models for automated ICD-10-AM and ACHI code assignment from discharge summaries. We first normalised a public ICD-9-CM coded dataset into the Australian coding system using General Equivalence Mappings and semantic similarity over code descriptions, with validation by a domain expert clinical coder. We then fine-tuned Meta-Llama-3-8B and compared full-parameter fine-tuning with a parameter-efficient QLoRA baseline. Finally, we applied Direct Preference Optimisation (DPO) using gold-standard-guided preference pairs, where the gold-standard ICD-10-AM/ACHI code list was treated as the preferred response and the supervised fine-tuned model output as the non-preferred response. Experiments were conducted on a mapped MIMIC-III subset and a de-identified Australian hospital dataset. Among the evaluated model variants, the DPO-aligned full-parameter model achieved the highest score, with micro-F1 scores of 0.7397 on the mapped dataset and 0.7834 on the Australian hospital dataset. These findings suggest that gold-standard-guided DPO can improve agreement with reference code sets compared with supervised fine-tuning alone. However, the results are limited to high-frequency diagnosis and procedure codes, short-document evaluation settings, and DPO preference pairs reviewed by one annotator.
SN - 1472-6947
ER -