Evaluating Retrieval-Augmented Generation Assistants for Higher-Education Information Services: A Systematic Review of Metrics, Test Sets and Failure Modes
- 1South Ural State University
Abstract
Background. Universities are increasingly adopting retrieval-augmented generation (RAG) assistants to answer student questions about admissions, regulations and services, but it is unclear how well the published evaluations support the accuracy these systems claim. Methods. We searched OpenAlex with five documented queries on 18 October 2026 (378 records, 310 after de-duplication), added 11 studies found by web search, screened 321 records and assessed 76 reports. Fifty-nine studies from 17 countries met the criteria. Screening and extraction were done with an AI assistant under the supervision of one author, and the review was registered retrospectively on OSF. Each study was coded on 13 fields and scored against an 11-item reporting checklist. Sixty-seven sought reports could not be retrieved. Results. Fifty-three of the 59 studies date from 2025 or 2026. Twenty-five name a retrieval metric and 23 a faithfulness or hallucination metric. Twenty-four of the 38 studies that report a test-set size use 100 questions or fewer, and 10 test on real user questions. Eighteen use an LLM-based judge, but only 3 report agreement between human raters. No study tests for outdated information, and only 3 report results by language. Reported faithfulness scores range from 0.19 to 0.96 across 12 studies. The median checklist score is 2 of 11 and the highest is 7. Conclusion. Retrieval clearly helps on questions about an institution’s own facts, but the evidence on how well these assistants work is thin and not comparable across studies. We offer the checklist as a minimum reporting standard.
Related
- Evaluating Retrieval-Augmented Generation vs. Long-Context Input for Clinical Reasoning over EHRs
- Evaluating Retrieval-Augmented Generation for Industrial Cybersecurity Compliance Assessment
- Evaluating AI agents on the IUCN Red List assessor exam: Tradeoffs and opportunities for AI assistants in extinction risk assessments
- Graph-Enhanced Retrieval-Augmented Generation for Postoperative Patient Question Answering: A Comparative Evaluation of Knowledge Graph, Retrieval-Augmented Generation, and Hybrid GraphRAG Approaches
- Efficient training-free query routing framework for federated retrieval-augmented generation
BibTeX
@misc{dey2026evaluating,
title = {{Evaluating Retrieval-Augmented Generation Assistants for Higher-Education Information Services: A Systematic Review of Metrics, Test Sets and Failure Modes}},
author = {Abhishek Dey},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23659588},
url = {https://zenodo.org/records/23659588},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating Retrieval-Augmented Generation Assistants for Higher-Education Information Services: A Systematic Review of Metrics, Test Sets and Failure Modes
AU - Abhishek Dey
PY - 2026
DA - 2026/10/26
PB - Zenodo
DO - 10.5281/zenodo.23659588
UR - https://zenodo.org/records/23659588
AB - Background. Universities are increasingly adopting retrieval-augmented generation (RAG) assistants to answer student questions about admissions, regulations and services, but it is unclear how well the published evaluations support the accuracy these systems claim. Methods. We searched OpenAlex with five documented queries on 18 October 2026 (378 records, 310 after de-duplication), added 11 studies found by web search, screened 321 records and assessed 76 reports. Fifty-nine studies from 17 countries met the criteria. Screening and extraction were done with an AI assistant under the supervision of one author, and the review was registered retrospectively on OSF. Each study was coded on 13 fields and scored against an 11-item reporting checklist. Sixty-seven sought reports could not be retrieved. Results. Fifty-three of the 59 studies date from 2025 or 2026. Twenty-five name a retrieval metric and 23 a faithfulness or hallucination metric. Twenty-four of the 38 studies that report a test-set size use 100 questions or fewer, and 10 test on real user questions. Eighteen use an LLM-based judge, but only 3 report agreement between human raters. No study tests for outdated information, and only 3 report results by language. Reported faithfulness scores range from 0.19 to 0.96 across 12 studies. The median checklist score is 2 of 11 and the highest is 7. Conclusion. Retrieval clearly helps on questions about an institution’s own facts, but the evidence on how well these assistants work is thin and not comparable across studies. We offer the checklist as a minimum reporting standard.
ER -