Domain-adaptive Bengali text summarization: Harnessing transformer-based large language models in closed-set contexts
- 1Electrical and Computer Engineering, North South University, Dhaka, Bangladesh
Abstract
This work aims to develop a domain-adapted Bengali text summarization model by training and fine-tuning on general and domain-specific datasets with categories such as state, international, and sports. Flan-T5 and mT5 LLMs were trained on the XLSUM Bengali dataset for text summarization, and they were trained on source domains and fine-tuned on the target domains for domain adaptation. In general-domain summarization on the XLSum Bengali dataset, mT5 achieved stronger overall performance than Flan-T5, with ROUGE-1 and BERTScore values of 0.21 and 0.72, respectively. The Flan-T5-XLSUM model outperformed other LLMs, achieving a ROUGE score of 0.79, a BLEU score of 0.13 and a BERTScore of 0.86. A human evaluation involving 28 participants was conducted to assess summary fluency, adequacy, and factual consistency across domains, confirming the qualitative effectiveness of the proposed models. A custom dataset is developed with manually annotated categories for domain adaptation purposes, contributing to domain-specific Bengali data. The ablation study illustrated that pre-training and fine-tuning on domain-specific data significantly enhanced model performance, with Flan-T5 fine-tuned on sports data. Three explainability methods (LIME, SHAP, and BertViz) revealed that geographically and contextually meaningful tokens strongly influenced the domain-adaptive Flan-T5 Bengali summarization model. The adapter-based Flan-T5 architecture enables effective multi-domain fine-tuning with less than 1% of trainable parameters while preserving model performance.
Related
- Early ERP processing of self-referential language processing in voluntary and involuntary action contexts: N170 and EPN.
- An adaptive transformer-based model with entropy-driven knowledge distillation for single-channel EEG REM sleep detection
- Learning from Adaptive Oscillatory Modes: An Interpretable Synchrosqueezing Transformer for Patient-Independent Seizure Detection
- Pruning the Search, Not the Signal: Adaptive-Banding Needleman-Wunsch Sequence Alignment via Protein Language Model Confidence
- Altered Sensorimotor-Association Axis Patterning of Global Functional Connectivity in an Autism Subtype With Low Levels of Language, Intellectual, and Adaptive Functioning
BibTeX
@article{roy2026domainadaptive,
title = {{Domain-adaptive Bengali text summarization: Harnessing transformer-based large language models in closed-set contexts}},
author = {Anusree Roy and Fahrin Hossain Sunaira and Tateyama Orpa and Farasha Shamma Yussouf and Intisar Tahmid Naheen and Riasat Khan},
journal = {PLOS One},
year = {2026},
month = oct,
doi = {10.1371/journal.pone.0357890},
url = {https://doi.org/10.1371/journal.pone.0357890},
note = {CC BY 4.0},
}
RIS
TY - JOUR
TI - Domain-adaptive Bengali text summarization: Harnessing transformer-based large language models in closed-set contexts
AU - Anusree Roy
AU - Fahrin Hossain Sunaira
AU - Tateyama Orpa
AU - Farasha Shamma Yussouf
AU - Intisar Tahmid Naheen
AU - Riasat Khan
PY - 2026
DA - 2026/10/05
JO - PLOS One
DO - 10.1371/journal.pone.0357890
UR - https://doi.org/10.1371/journal.pone.0357890
AB - This work aims to develop a domain-adapted Bengali text summarization model by training and fine-tuning on general and domain-specific datasets with categories such as state, international, and sports. Flan-T5 and mT5 LLMs were trained on the XLSUM Bengali dataset for text summarization, and they were trained on source domains and fine-tuned on the target domains for domain adaptation. In general-domain summarization on the XLSum Bengali dataset, mT5 achieved stronger overall performance than Flan-T5, with ROUGE-1 and BERTScore values of 0.21 and 0.72, respectively. The Flan-T5-XLSUM model outperformed other LLMs, achieving a ROUGE score of 0.79, a BLEU score of 0.13 and a BERTScore of 0.86. A human evaluation involving 28 participants was conducted to assess summary fluency, adequacy, and factual consistency across domains, confirming the qualitative effectiveness of the proposed models. A custom dataset is developed with manually annotated categories for domain adaptation purposes, contributing to domain-specific Bengali data. The ablation study illustrated that pre-training and fine-tuning on domain-specific data significantly enhanced model performance, with Flan-T5 fine-tuned on sports data. Three explainability methods (LIME, SHAP, and BertViz) revealed that geographically and contextually meaningful tokens strongly influenced the domain-adaptive Flan-T5 Bengali summarization model. The adapter-based Flan-T5 architecture enables effective multi-domain fine-tuning with less than 1% of trainable parameters while preserving model performance.
SN - 1932-6203
ER -