Assessing the Concordance Between LLM-Based ICD-10 Coding and a Drug Database for Antihypertensive Drug Contraindications: A Weighted Inter-Rater Agreement Study
Abstract
Background: Manual ICD-10 coding of drug contraindications is resource-intensive and difficult to scale. Large language models (LLMs) have shown early promise for clinical information extraction, but their reliability for structured pharmacological coding relative to validated expert databases remains unknown. Objective: To assess the concordance between LLM-based automated ICD-10 coding and Thésorimed drug database1 for antihypertensive drug contraindications. Methods: Contraindications were extracted from Summary of Product Characteristics free-text documents for 301 antihypertensive drugs using a retrieval-augmented generation pipeline and aligned against Thésorimed ICD-10 codes. Agreement was assessed using a binary kappa (κ1, entity presence) and a hierarchical weighted kappa (κ2, ICD-10 code concordance) on 5,074 entities. Results: Entity-level agreement was slight (κ1 = -0.30), reflecting structural differences in source coverage. Code-level agreement among matched pairs was substantial (κ2 = 0.70, Po = 0.871), with 76.9% identical codes. Conclusion: When both sources identify the same contraindication, LLM-based coding shows substantial concordance with expert-curated ICD-10 assignments. However, low entity-level agreement highlights important differences in coverage and representation between sources.
Related
- Adverse effects of gender‐affirming hormonal therapy in transgender persons: Assessing reports in the French pharmacovigilance database
- Antihypertensive drug targets and pancreatitis reporting signals: An integrative mendelian randomization and FAERS pharmacovigilance study
- Causal effects of antihypertensive drugs on thyroid diseases: A trans-ancestry drug-target Mendelian randomization study.
- Rising contraceptive use, falling couple concordance: A mixed-methods analysis of the concordance paradox in Nigeria
- Autoantibodies neutralizing type I interferons in older twin pairs: Low concordance.
BibTeX
@article{adrouji2026assessing,
title = {{Assessing the Concordance Between LLM-Based ICD-10 Coding and a Drug Database for Antihypertensive Drug Contraindications: A Weighted Inter-Rater Agreement Study}},
author = {Samya Adrouji and Abdelmalek Mouazer and Jean-Baptiste Lamy},
journal = {Studies in Health Technology and Informatics},
year = {2026},
month = jun,
doi = {10.3233/shti260902},
url = {https://hal.science/hal-05709884v1},
note = {CC BY-NC 4.0},
}
RIS
TY - JOUR
TI - Assessing the Concordance Between LLM-Based ICD-10 Coding and a Drug Database for Antihypertensive Drug Contraindications: A Weighted Inter-Rater Agreement Study
AU - Samya Adrouji
AU - Abdelmalek Mouazer
AU - Jean-Baptiste Lamy
PY - 2026
DA - 2026/06/29
JO - Studies in Health Technology and Informatics
DO - 10.3233/shti260902
UR - https://hal.science/hal-05709884v1
AB - Background: Manual ICD-10 coding of drug contraindications is resource-intensive and difficult to scale. Large language models (LLMs) have shown early promise for clinical information extraction, but their reliability for structured pharmacological coding relative to validated expert databases remains unknown. Objective: To assess the concordance between LLM-based automated ICD-10 coding and Thésorimed drug database1 for antihypertensive drug contraindications. Methods: Contraindications were extracted from Summary of Product Characteristics free-text documents for 301 antihypertensive drugs using a retrieval-augmented generation pipeline and aligned against Thésorimed ICD-10 codes. Agreement was assessed using a binary kappa (κ1, entity presence) and a hierarchical weighted kappa (κ2, ICD-10 code concordance) on 5,074 entities. Results: Entity-level agreement was slight (κ1 = -0.30), reflecting structural differences in source coverage. Code-level agreement among matched pairs was substantial (κ2 = 0.70, Po = 0.871), with 76.9% identical codes. Conclusion: When both sources identify the same contraindication, LLM-based coding shows substantial concordance with expert-curated ICD-10 assignments. However, low entity-level agreement highlights important differences in coverage and representation between sources.
SN - 0926-9630
SN - 1879-8365
ER -