{
  "abstract": "Background. Universities are increasingly adopting retrieval-augmented generation (RAG) assistants to answer student questions about admissions, regulations and services, but it is unclear how well the published evaluations support the accuracy these systems claim. Methods. We searched OpenAlex with five documented queries on 18 October 2026 (378 records, 310 after de-duplication), added 11 studies found by web search, screened 321 records and assessed 76 reports. Fifty-nine studies from 17 countries met the criteria. Screening and extraction were done with an AI assistant under the supervision of one author, and the review was registered retrospectively on OSF. Each study was coded on 13 fields and scored against an 11-item reporting checklist. Sixty-seven sought reports could not be retrieved. Results. Fifty-three of the 59 studies date from 2025 or 2026. Twenty-five name a retrieval metric and 23 a faithfulness or hallucination metric. Twenty-four of the 38 studies that report a test-set size use 100 questions or fewer, and 10 test on real user questions. Eighteen use an LLM-based judge, but only 3 report agreement between human raters. No study tests for outdated information, and only 3 report results by language. Reported faithfulness scores range from 0.19 to 0.96 across 12 studies. The median checklist score is 2 of 11 and the highest is 7. Conclusion. Retrieval clearly helps on questions about an institution’s own facts, but the evidence on how well these assistants work is thin and not comparable across studies. We offer the checklist as a minimum reporting standard.",
  "abstract_shown": true,
  "affiliations": [
    "South Ural State University"
  ],
  "article_type": null,
  "arxiv": null,
  "author_total": 1,
  "authors": [
    {
      "affs": [
        0
      ],
      "name": "Abhishek Dey",
      "orcid": "0009-0007-4913-7090"
    }
  ],
  "bibtex": "https://research.draconitic.com/paper/1f9qwqhegk4v1.bib",
  "cited_by": 0,
  "doi": "10.5281/zenodo.23659588",
  "fulltext_url": null,
  "id": "1f9qwqhegk4v1",
  "issns": [],
  "landing_url": "https://zenodo.org/records/23659588",
  "language": "eng",
  "licence": "CC-BY-4.0",
  "licence_label": "CC BY 4.0",
  "licence_source": "article",
  "licence_url": "https://creativecommons.org/licenses/by/4.0/",
  "metadata_licence": null,
  "native_subjects": [],
  "openalex": null,
  "pdf_url": "https://zenodo.org/api/records/23659588/files/Dey_RAG_HigherEd_SystematicReview_FIXED.pdf/content",
  "pmcid": null,
  "pmid": null,
  "published": "2026-10-26",
  "publisher": null,
  "ris": "https://research.draconitic.com/paper/1f9qwqhegk4v1.ris",
  "sources": [
    {
      "first_seen": "2026-11-09T12:21:03Z",
      "native_id": "23659588",
      "source": "zenodo",
      "url": "https://zenodo.org/records/23659588"
    }
  ],
  "subjects": [],
  "title": "Evaluating Retrieval-Augmented Generation Assistants for Higher-Education Information Services: A Systematic Review of Metrics, Test Sets and Failure Modes",
  "title_text": "Evaluating Retrieval-Augmented Generation Assistants for Higher-Education Information Services: A Systematic Review of Metrics, Test Sets and Failure Modes",
  "type": "preprint",
  "updated_at": "2026-11-09T12:21:03Z",
  "url": "https://research.draconitic.com/paper/1f9qwqhegk4v1",
  "venue": "Zenodo",
  "versions": []
}