Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting

@article{hosen2026evaluating,
  title = {{Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting}},
  author = {Md. Hamid Hosen and Sadia Nawar and Rituparna Chowdhury and Altaf Uddin and Riadul Islam Rabbi and Khair Razlan Othman and Nurul Karim Symon},
  journal = {PLOS One},
  year = {2026},
  month = nov,
  doi = {10.1371/journal.pone.0361368},
  url = {https://doi.org/10.1371/journal.pone.0361368},
  note = {CC BY 4.0},
}

TY  - JOUR
TI  - Evaluating the effectiveness of open-weight LLMs for multi-domain Bengali text classification with zero-shot and few-shot prompting
AU  - Md. Hamid Hosen
AU  - Sadia Nawar
AU  - Rituparna Chowdhury
AU  - Altaf Uddin
AU  - Riadul Islam Rabbi
AU  - Khair Razlan Othman
AU  - Nurul Karim Symon
PY  - 2026
DA  - 2026/11/06
JO  - PLOS One
DO  - 10.1371/journal.pone.0361368
UR  - https://doi.org/10.1371/journal.pone.0361368
AB  - Large Language Models (LLMs) have advanced multilingual NLP significantly, yet their effectiveness for Bengali, a globally important but low-resource language, remains understudied due to limited linguistic resources and the absence of a standardized evaluation benchmark. This is due to the unavailability of large linguistic resources and the lack of a standard benchmarking framework, which has yet to be established. This study aims to address the underrepresentation of Bengali text classification by leading a systematic evaluation. It involves assessing multiple open-weight LLMs on Bengali text classification tasks and observing their limitations and regular error patterns. The procedure consists of six open-weight LLMs, which include LLaMA-3.2-3B, Mistral-V3-7B, DeepSeek-R1-8B, Phi-4-14B, Gemma-2-27B, and Qwen-2.5-72B, and are evaluated using four critical text classification tasks in Bengali, namely sentiment analysis, emotion recognition, hate-speech detection, and fake-news classification. Furthermore, two prompting methods are compared in the context of zero-shot and few-shot prompting using both the vLLM inference engine and Hugging Face Transformers pipelines to investigate the adaptability of the models in resource-limited settings. Across all four tasks, the two largest models (Qwen-2.5-72B and Gemma-2-27B) obtained the highest scores, although the ranking was not strictly monotonic in parameter count and, because each scale is represented by a single model family, the design cannot attribute these differences to model size alone. When training in zero-shot, Qwen-2.5-72B showed the best performance at 79.90% accuracy and 79.88% Macro-F1 score, showing amazing accuracy on hate-speech detection. In the case of fake news, Gemma-2-27B performed best at 84.45% accuracy and 84.33% Macro-F1 score. With a few-shot set-up, Gemma-2-27B and Qwen-2.5-72B demonstrated a high level of performance in terms of Macro-F1 score, reaching 84.17% for fake-news detection and 66.80% for emotion recognition, respectively. Paired multi-model tests (Cochran’s Q with Holm-corrected McNemar post-hoc comparisons) confirmed significant differences among the six models across all tasks and prompting settings, whereas the effect of five-shot prompting, obtained with a single fixed demonstration set, was significant for only 4 of 24 model–task combinations after correction and never for the three largest models. The findings characterise benchmark classification performance under the evaluated settings rather than real-world effectiveness, and indicate that model scale, multilingual pre-training, and in-context demonstrations are jointly associated with, but cannot be causally isolated as drivers of, performance on Bengali text classification.
SN  - 1932-6203
ER  -