Benchmarking Embedding Models for POJK Regulatory Retrieval in Indonesian Insurance Documents
Abstract
Indonesian insurance regulations are clause-dense and terminologically specific, making manual retrieval of applicable provisions difficult. This study presents a retrieval-only benchmark comparing embedding and lexical methods on POJK 26/2025 and POJK 36/2025. The corpus contains 181 context chunks and 181 synthetic Indonesian queries, each assigned one primary relevant chunk. Queries were generated in JSON mode using Gemma 4 E4B via Ollama (ollama:gemma4:e4b), with malformed outputs retried and heuristic fallbacks prohibited. Evaluation comprised a four-model local dense comparison and a separate native BGE-M3/BM25 ablation using exact exhaustive ranking. The local comparison reports MRR and Hit rates, while the native ablation additionally reports NDCG@10, MAP@10, bootstrap confidence intervals, paired tests, and rank bins. Qwen3 Embedding achieved the highest local MRR (0.8075). Under a prespecified descriptive composite rule combining MRR, Hit rates, and measured latency, BGE-M3 ranked first. In the native ablation, BGE-M3 combined retrieval achieved NDCG@10 of 0.8810. A practitioner-validated subset audit and artifact-backed qualitative analysis supplemented the quantitative results. Findings are limited to this two-document benchmark and do not evaluate answer generation, citation correctness, or end-to-end RAG faithfulness.
Related
- Context-Aware Detection of Content Evidencing Illegal or Predatory Fintech Lending Practices on Indonesian Digital Platforms Using Transformer-Based Language Models
- The effects of expanding the generosity of statutory sick leave insurance: The case of a French reform
- Existence and uniqueness for non-linear forward equations: Addendum to "Mean-field approximations in insurance" (Hornung, 2026)
- Comparative phytochemical profiling and bioactivity evaluation of three Indonesian pomegranate (Punica granatum) peel varieties: Antioxidant, antibacterial, and in vitro antiproliferative activities
- Text Data Analysis and Classification Methods - Insights from Customer Letters in Life Insurance
BibTeX
@article{nurdin2026benchmarking,
title = {{Benchmarking Embedding Models for POJK Regulatory Retrieval in Indonesian Insurance Documents}},
author = {Daeng Ahmad Nurdin and Bambang Jokonowo and Bagus Priambodo},
journal = {Journal of Information Systems and Informatics},
year = {2026},
month = oct,
doi = {10.63158/journalisi.v8i5.1781},
url = {https://doi.org/10.63158/journalisi.v8i5.1781},
note = {CC BY},
}
RIS
TY - JOUR
TI - Benchmarking Embedding Models for POJK Regulatory Retrieval in Indonesian Insurance Documents
AU - Daeng Ahmad Nurdin
AU - Bambang Jokonowo
AU - Bagus Priambodo
PY - 2026
DA - 2026/10/24
JO - Journal of Information Systems and Informatics
DO - 10.63158/journalisi.v8i5.1781
UR - https://doi.org/10.63158/journalisi.v8i5.1781
AB - Indonesian insurance regulations are clause-dense and terminologically specific, making manual retrieval of applicable provisions difficult. This study presents a retrieval-only benchmark comparing embedding and lexical methods on POJK 26/2025 and POJK 36/2025. The corpus contains 181 context chunks and 181 synthetic Indonesian queries, each assigned one primary relevant chunk. Queries were generated in JSON mode using Gemma 4 E4B via Ollama (ollama:gemma4:e4b), with malformed outputs retried and heuristic fallbacks prohibited. Evaluation comprised a four-model local dense comparison and a separate native BGE-M3/BM25 ablation using exact exhaustive ranking. The local comparison reports MRR and Hit rates, while the native ablation additionally reports NDCG@10, MAP@10, bootstrap confidence intervals, paired tests, and rank bins. Qwen3 Embedding achieved the highest local MRR (0.8075). Under a prespecified descriptive composite rule combining MRR, Hit rates, and measured latency, BGE-M3 ranked first. In the native ablation, BGE-M3 combined retrieval achieved NDCG@10 of 0.8810. A practitioner-validated subset audit and artifact-backed qualitative analysis supplemented the quantitative results. Findings are limited to this two-document benchmark and do not evaluate answer generation, citation correctness, or end-to-end RAG faithfulness.
SN - 2656-5935
SN - 2656-4882
ER -