Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG
Abstract
Question-answering services built on retrieval-augmented generation (RAG), in which a language model answers from retrieved documents, are inspected continuously and upgraded repeatedly, so their reliability guarantee must survive both. We study federated conformal RAG: nodes holding private corpora score candidate answers with a shared language model and send compressed scores to a hub that returns an answer set; a miss omits the true answer. We formulate monitoring as a sequential test: alarm when misses exceed a certified bound on the miss rate (the allowance). But a conformal certificate covers one frozen configuration at one look fixed in advance, so it gives no anytime-valid alarm; a threshold calibrated for one model certifies nothing about its replacement; and re-certifying each upgrade at full error level compounds failures. We propose Anytime-FC-RAG with three auditable rules: pick each configuration before its fresh calibration, charge every certificate attempt to one trajectory-wide budget δcal, and fix threshold, allowance and bet before each query. One betting wealth Et, growing in expectation only when misses exceed the allowance, never resets across deployments. Our certified adaptive-refresh theorem shows that, with evidence budget δe, alarming when Et reaches 1/δe has false-alarm probability at most δe + δcal under a random number of history-selected, evidence-driven refreshes of model, retriever, corpus, score or threshold; δcal pays for wrong certificates. With an exact order-statistic certificate the bound is distribution-free, architecture-agnostic and about misses, not per-input coverage or distribution shift as such. On Qwen2.5/MMLU-Pro the alarm separated audited-harmful from benign shifts completely at a 5% budget; in a model swap, only fresh calibration kept a downgrade's miss rate in check.
Related
- Common-Center Geometry and Certified Radial Reconstruction for Energy-Form Full Conformal Regions
- Region bounds, error margins and certified positivity in the numerical conformal bootstrap; Or the first mathematically rigorous bounds on the 3d Ising CFT critical exponents
- Federated Language Models Under Bandwidth Budgets: Distillation Rates and Conformal Coverage
- Byzantine-Robust Federated RAG via Aligned Calibration and Fixed-Membership Conformal Prediction
- Certified exact bounds for adaptive quantitative group testing with two defectives
BibTeX
@misc{dubey2026certified,
title = {{Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG}},
author = {Prasanjit Dubey and Xiaoming Huo},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2605.29139},
eprint = {2605.29139},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2605.29139},
}
RIS
TY - UNPB
TI - Certified Adaptive Refresh: Anytime-Valid Monitoring for Federated Conformal RAG
AU - Prasanjit Dubey
AU - Xiaoming Huo
PY - 2026
DA - 2026/09/29
PB - arXiv
DO - 10.48550/arxiv.2605.29139
UR - https://arxiv.org/abs/2605.29139
AB - Question-answering services built on retrieval-augmented generation (RAG), in which a language model answers from retrieved documents, are inspected continuously and upgraded repeatedly, so their reliability guarantee must survive both. We study federated conformal RAG: nodes holding private corpora score candidate answers with a shared language model and send compressed scores to a hub that returns an answer set; a miss omits the true answer. We formulate monitoring as a sequential test: alarm when misses exceed a certified bound on the miss rate (the allowance). But a conformal certificate covers one frozen configuration at one look fixed in advance, so it gives no anytime-valid alarm; a threshold calibrated for one model certifies nothing about its replacement; and re-certifying each upgrade at full error level compounds failures. We propose Anytime-FC-RAG with three auditable rules: pick each configuration before its fresh calibration, charge every certificate attempt to one trajectory-wide budget δ_cal, and fix threshold, allowance and bet before each query. One betting wealth E_t, growing in expectation only when misses exceed the allowance, never resets across deployments. Our certified adaptive-refresh theorem shows that, with evidence budget δ_e, alarming when E_t reaches 1/δ_e has false-alarm probability at most δ_e + δ_cal under a random number of history-selected, evidence-driven refreshes of model, retriever, corpus, score or threshold; δ_cal pays for wrong certificates. With an exact order-statistic certificate the bound is distribution-free, architecture-agnostic and about misses, not per-input coverage or distribution shift as such. On Qwen2.5/MMLU-Pro the alarm separated audited-harmful from benign shifts completely at a 5% budget; in a model swap, only fresh calibration kept a downgrade's miss rate in check.
ER -