A Comparative Benchmark of Biomedical Language Models for Concept Normalization from Real-World Text
- 1CSIR-Institute of Genomics and Integrative Biology
Abstract
Patient-reported and clinically documented narratives often contain informal, fragmented and linguistically heterogeneous expressions, complicating medical concept normalization (MCN) and frequently necessitating pre-processing before terminology mapping. Despite rapid advances in biomedical language modeling, the comparative utility of representation models for MCN and their integration with instruction-tuned LLMs in automated normalization workflows remain underexplored. In this study, we systematically benchmarked 15 general, biomedical and clinical transformer-based representation models together with 12 instruction-tuned LLMs across 12,713 instances from five established datasets: TAC2017_ADR, TwADR-L, TwiMed, CADEC and SMM4H2017. Representation models were evaluated using embedding-based semantic retrieval, whereas instruction-tuned LLMs were assessed as upstream text-correction modules. SapBERT achieved the highest Top-5 accuracy among representation models, reaching 63.8% for SNOMED CT and 58.0% for MedDRA without correction. Qwen 2 Instruct was selected as the preferred corrector on the basis of its favorable balance between Top-1 normalization performance and computational efficiency relative to substantially larger models, including Llama 3.1 Instruct (70B). Incorporation of Qwen 2 Instruct increased Top-5 accuracy to 69.4% for SNOMED CT and 63.2% for MedDRA. The resulting framework accepts heterogeneous short medical expressions without manual input pre-processing and automatically performs text refinement, semantic retrieval and terminology mapping to standardized concepts and vocabulary codes. These findings establish a benchmark-guided, scalable framework for automated medical terminology standardization.
Related
- An Expert-Curated Benchmark for Frontier and Local LLMs Querying a Biomedical Knowledge Graph
- Contextual Biomedical Language Models for Imbalance-Aware Drug-Food Interaction Classification
- Tracking claim changes from preprint to publication across 72,644 biomedical studies using large language models
- VibeProteinBench: An Evaluation Benchmark for Language-interfaced Vibe Protein Design
- RadMultiBench: A Reproducible Dual-Task Benchmark Exposing the Limits of Pre-Trained Vision-Language Models Across Thoracic and Breast Imaging
BibTeX
@misc{verma2026comparative,
title = {{A Comparative Benchmark of Biomedical Language Models for Concept Normalization from Real-World Text}},
author = {A. Verma and Abhijay and M. Vangani and S. Prakash and K. Chaudhary},
howpublished = {bioRxiv},
year = {2026},
month = sep,
doi = {10.64898/2026.09.19.752843},
url = {https://www.biorxiv.org/content/10.64898/2026.09.19.752843v1},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - A Comparative Benchmark of Biomedical Language Models for Concept Normalization from Real-World Text
AU - A. Verma
AU - Abhijay
AU - M. Vangani
AU - S. Prakash
AU - K. Chaudhary
PY - 2026
DA - 2026/09/19
PB - bioRxiv
DO - 10.64898/2026.09.19.752843
UR - https://www.biorxiv.org/content/10.64898/2026.09.19.752843v1
AB - Patient-reported and clinically documented narratives often contain informal, fragmented and linguistically heterogeneous expressions, complicating medical concept normalization (MCN) and frequently necessitating pre-processing before terminology mapping. Despite rapid advances in biomedical language modeling, the comparative utility of representation models for MCN and their integration with instruction-tuned LLMs in automated normalization workflows remain underexplored. In this study, we systematically benchmarked 15 general, biomedical and clinical transformer-based representation models together with 12 instruction-tuned LLMs across 12,713 instances from five established datasets: TAC2017_ADR, TwADR-L, TwiMed, CADEC and SMM4H2017. Representation models were evaluated using embedding-based semantic retrieval, whereas instruction-tuned LLMs were assessed as upstream text-correction modules. SapBERT achieved the highest Top-5 accuracy among representation models, reaching 63.8% for SNOMED CT and 58.0% for MedDRA without correction. Qwen 2 Instruct was selected as the preferred corrector on the basis of its favorable balance between Top-1 normalization performance and computational efficiency relative to substantially larger models, including Llama 3.1 Instruct (70B). Incorporation of Qwen 2 Instruct increased Top-5 accuracy to 69.4% for SNOMED CT and 63.2% for MedDRA. The resulting framework accepts heterogeneous short medical expressions without manual input pre-processing and automatically performs text refinement, semantic retrieval and terminology mapping to standardized concepts and vocabulary codes. These findings establish a benchmark-guided, scalable framework for automated medical terminology standardization.
ER -