Structured extraction from Russian/Kazakh discharge summaries: three model families at the human-agreement ceiling
- 1Nazarbayev University School of Medicine
Abstract
ObjectiveTo evaluate task-dependent selection of local and cloud models for structured extraction from multilingual non-English stroke discharge summaries, distinguishing entity detection, record assembly, and raw-document processing.
MethodsThis retrospective system evaluation compared multilingual encoders, locally fine-tuned Qwen3-4B models, and zero-shot GPT-5.5. Primary test cohorts comprised 332 section cases, 149 medication cases with 2,475 reference records after identity-based exclusions, and 191 laboratory cases. Outcomes were section-span F1, drug-name F1, normalized seven-field record recovery, test-name F1, and laboratory quintuple F1. Paired human comparisons used a common second-annotator reference on 50 cases per task. Saved cascade outputs were compared with curated-section controls using document-bootstrap intervals.
ResultsSection F1 was 0.919 for the encoder, 0.926 for GPT-5.5, and 0.932 for Qwen. GPT-5.5 led medication detection (0.966 versus 0.940 for Qwen), whereas Qwen led normalized medication recovery (0.381 versus 0.311; difference 0.070, 95% CI 0.026-0.114) and laboratory quintuple F1 (0.892 versus 0.822). With matched inference stacks, medication recovery declined from 0.377 on curated inputs to 0.204 for single-window and 0.246 for all-block cascades; quintuple F1 on the original laboratory cohort declined from 0.898 to 0.811. Corpus-wide extraction processed 193,101 summaries in 13.3 H100 GPU-hours.
ConclusionLocal models achieved strong detection performance without frontier-scale inference in this setting. Fine-tuned local LLMs were advantageous for record assembly under the evaluated recipes; annotation conventions, normalization, and upstream section extraction remained important constraints.
Related
- Gold-standard-guided direct preference optimisation for ICD-10-AM and ACHI code assignment from discharge summaries.
- Large Language Models' Simulation of Patient Comprehension of Discharge Summaries: Multimodel Evaluation Study.
- AI-assisted identification of potential treatment injury from inpatient discharge summaries: an early-stage sociotechnical evaluation in a New Zealand district hospital
- Structured Teach-Back Discharge Education and Heart Failure Self-Care
- Piloting a data-extraction form for verbatim terminology: a blinded inter-reviewer agreement study before a surgical evidence map
BibTeX
@misc{aimyshev2026structured,
title = {{Structured extraction from Russian/Kazakh discharge summaries: three model families at the human-agreement ceiling}},
author = {T. Aimyshev and I. Arupzhanov and G. Zhakhina and A. Rakhmankulova and S. Smagul and A. Gaipov},
howpublished = {medRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.03.26362149},
url = {https://www.medrxiv.org/content/10.64898/2026.09.03.26362149v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - Structured extraction from Russian/Kazakh discharge summaries: three model families at the human-agreement ceiling
AU - T. Aimyshev
AU - I. Arupzhanov
AU - G. Zhakhina
AU - A. Rakhmankulova
AU - S. Smagul
AU - A. Gaipov
PY - 2026
DA - 2026/09/03
PB - medRxiv
DO - 10.64898/2026.09.03.26362149
UR - https://www.medrxiv.org/content/10.64898/2026.09.03.26362149v1
AB - ObjectiveTo evaluate task-dependent selection of local and cloud models for structured extraction from multilingual non-English stroke discharge summaries, distinguishing entity detection, record assembly, and raw-document processing. MethodsThis retrospective system evaluation compared multilingual encoders, locally fine-tuned Qwen3-4B models, and zero-shot GPT-5.5. Primary test cohorts comprised 332 section cases, 149 medication cases with 2,475 reference records after identity-based exclusions, and 191 laboratory cases. Outcomes were section-span F1, drug-name F1, normalized seven-field record recovery, test-name F1, and laboratory quintuple F1. Paired human comparisons used a common second-annotator reference on 50 cases per task. Saved cascade outputs were compared with curated-section controls using document-bootstrap intervals. ResultsSection F1 was 0.919 for the encoder, 0.926 for GPT-5.5, and 0.932 for Qwen. GPT-5.5 led medication detection (0.966 versus 0.940 for Qwen), whereas Qwen led normalized medication recovery (0.381 versus 0.311; difference 0.070, 95% CI 0.026-0.114) and laboratory quintuple F1 (0.892 versus 0.822). With matched inference stacks, medication recovery declined from 0.377 on curated inputs to 0.204 for single-window and 0.246 for all-block cascades; quintuple F1 on the original laboratory cohort declined from 0.898 to 0.811. Corpus-wide extraction processed 193,101 summaries in 13.3 H100 GPU-hours. ConclusionLocal models achieved strong detection performance without frontier-scale inference in this setting. Fine-tuned local LLMs were advantageous for record assembly under the evaluated recipes; annotation conventions, normalization, and upstream section extraction remained important constraints.
ER -