Design and Evaluation of a Route-Aware, Evidence-Grounded RAG Chatbot for Accessing Village Monographs and Census Data within Data Desa Presisi
Abstract
Data Desa Presisi integrates narrative village monographs and structured census records, creating challenges for contextual, numerical, and cross-source information access. This study designed and evaluated a locally deployed, route-aware, evidence-grounded retrieval-augmented generation (RAG) chatbot that combines deterministic DuckDB queries, document retrieval, provenance, and privacy-aware routing. Evaluation used 100 ground-truth questions from Setu and Gunung Putri. Eighty-four database, monograph, and mixed-evidence questions were assessed using RAGAS and ROUGE-L, whereas 16 conversational and policy questions were evaluated through deterministic checks. Elasticsearch at a temperature of 0.1 produced the highest composite score (0.592018), although post hoc Wilcoxon tests found no statistically significant superiority over alternative configurations. The selected configuration achieved a 98.0% functional pass rate and a 97.92% numeric-match rate. Public-mode usability evaluation involving 21 village officials and staff yielded 96.03% effectiveness. Scalability testing covered pipeline construction and retrieval across 30 census inputs and 28 monographs, but did not assess concurrent users or full end-to-end production workloads. These findings provide preliminary evidence that route-aware RAG can improve transparent, evidence-grounded access to heterogeneous village data while preserving deterministic handling of numerical queries and privacy-sensitive information.
Related
- Evaluation Choices Decide the Forecasting Leaderboard: Evidence from a Production Marketplace Panel
- A Formal Proof of Riemann Hypothesis from Existence Alone, the Sign Excepted by Theorem: The Constructed Witness, the Identity Cone over the Fixed Locus, the Grounded Counter-Model, and the Cure Theorem Proved in Lean 4 and Re-executed in Fortran
- The grounded spirituality of mathematics
- Latent Similarity Gaussian Processes: A Theory-Grounded Approach to Personalized Suicide-Risk Forecasting for Clinical Decision-Support
- CausalSmith: A Formally Grounded, Self-Improving Agentic Framework for Automated Research in Causal Inference
BibTeX
@article{amal2026design,
title = {{Design and Evaluation of a Route-Aware, Evidence-Grounded RAG Chatbot for Accessing Village Monographs and Census Data within Data Desa Presisi}},
author = {Ikhlasul Amal and Irman Hermadi and Yani Nurhadryani},
journal = {Journal of Information Systems and Informatics},
year = {2026},
month = oct,
doi = {10.63158/journalisi.v8i5.1856},
url = {https://doi.org/10.63158/journalisi.v8i5.1856},
note = {CC BY},
}
RIS
TY - JOUR
TI - Design and Evaluation of a Route-Aware, Evidence-Grounded RAG Chatbot for Accessing Village Monographs and Census Data within Data Desa Presisi
AU - Ikhlasul Amal
AU - Irman Hermadi
AU - Yani Nurhadryani
PY - 2026
DA - 2026/10/24
JO - Journal of Information Systems and Informatics
DO - 10.63158/journalisi.v8i5.1856
UR - https://doi.org/10.63158/journalisi.v8i5.1856
AB - Data Desa Presisi integrates narrative village monographs and structured census records, creating challenges for contextual, numerical, and cross-source information access. This study designed and evaluated a locally deployed, route-aware, evidence-grounded retrieval-augmented generation (RAG) chatbot that combines deterministic DuckDB queries, document retrieval, provenance, and privacy-aware routing. Evaluation used 100 ground-truth questions from Setu and Gunung Putri. Eighty-four database, monograph, and mixed-evidence questions were assessed using RAGAS and ROUGE-L, whereas 16 conversational and policy questions were evaluated through deterministic checks. Elasticsearch at a temperature of 0.1 produced the highest composite score (0.592018), although post hoc Wilcoxon tests found no statistically significant superiority over alternative configurations. The selected configuration achieved a 98.0% functional pass rate and a 97.92% numeric-match rate. Public-mode usability evaluation involving 21 village officials and staff yielded 96.03% effectiveness. Scalability testing covered pipeline construction and retrieval across 30 census inputs and 28 monographs, but did not assess concurrent users or full end-to-end production workloads. These findings provide preliminary evidence that route-aware RAG can improve transparent, evidence-grounded access to heterogeneous village data while preserving deterministic handling of numerical queries and privacy-sensitive information.
SN - 2656-5935
SN - 2656-4882
ER -