Evaluating Retrieval-Augmented Generation for Industrial Cybersecurity Compliance Assessment
- 1Axix Technologies LLC USA
Abstract
This paper presents a methodology for evaluating Retrieval-Augmented Generation (RAG) in industrial cybersecurity compliance assessment. The study focuses on whether AI-generated compliance judgments are supported by authoritative cybersecurity guidance and evidence of control implementation.
The proposed evaluation uses NIST Special Publication 800-82 Revision 3 as its primary reference source and defines three assessment outcomes: compliant, non-compliant, and insufficient evidence. The planned benchmark examines five evidence conditions: full supporting evidence, partial supporting evidence, irrelevant evidence, absent evidence, and contradictory evidence.
The methodology describes a human annotation and adjudication process and proposes metrics for classification performance, retrieval relevance, evidence grounding, citation correctness, and uncertainty handling.
This manuscript is a methodology-focused research preprint. The full benchmark experiments and quantitative evaluation have not been completed, and no empirical performance claims are made. Further dataset construction, experimental execution, and independent evaluation are required.
Related
- Beyond compliance: evaluating the impact of assent-based training on equine behavior and the human-horse relationship.
- WardBench: Evaluating Policy Compliance, Dynamic FHIR Consent, and Construct Validity in Clinical EHR Agents
- A quantitative model for embedding quantum-secure communications and evaluating communication autonomy in industrial control systems
- Assessment of Compliance With Infection Prevention Guidelines/Standard Precautions Among Healthcare Workers of Debark and Metema General Hospitals, Northwest Ethiopia
- Evaluating FoundationPose object registration accuracy for AR-guided industrial inspection
BibTeX
@misc{arshad2026evaluating,
title = {{Evaluating Retrieval-Augmented Generation for Industrial Cybersecurity Compliance Assessment}},
author = {Muhammad Ali Arshad},
howpublished = {Zenodo},
year = {2026},
month = oct,
doi = {10.5281/zenodo.23498011},
url = {https://zenodo.org/records/23498011},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Evaluating Retrieval-Augmented Generation for Industrial Cybersecurity Compliance Assessment
AU - Muhammad Ali Arshad
PY - 2026
DA - 2026/10/20
PB - Zenodo
DO - 10.5281/zenodo.23498011
UR - https://zenodo.org/records/23498011
AB - This paper presents a methodology for evaluating Retrieval-Augmented Generation (RAG) in industrial cybersecurity compliance assessment. The study focuses on whether AI-generated compliance judgments are supported by authoritative cybersecurity guidance and evidence of control implementation. The proposed evaluation uses NIST Special Publication 800-82 Revision 3 as its primary reference source and defines three assessment outcomes: compliant, non-compliant, and insufficient evidence. The planned benchmark examines five evidence conditions: full supporting evidence, partial supporting evidence, irrelevant evidence, absent evidence, and contradictory evidence. The methodology describes a human annotation and adjudication process and proposes metrics for classification performance, retrieval relevance, evidence grounding, citation correctness, and uncertainty handling. This manuscript is a methodology-focused research preprint. The full benchmark experiments and quantitative evaluation have not been completed, and no empirical performance claims are made. Further dataset construction, experimental execution, and independent evaluation are required.
ER -