Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting
Abstract
Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically control false alarms by requiring the discrepancy measure used to detect shifts to be *pre-specified*. However, for unstructured data, pre-specification creates a sensitivity--breadth trade-off: a narrowly defined measure is sensitive to the anticipated shift but may miss unexpected changes, whereas a broadly defined measure may detect a wider range of shifts but with reduced sensitivity. Although adaptively hypothesizing the shift direction can overcome this limitation, such data-dependent adaptation generally invalidates conventional false-alarm guarantees. We reconcile adaptivity with statistical validity through testing-by-betting, reframing sequential changepoint detection as a betting game in which the bettor continually updates its hypothesized shift direction, or witness function, as new observations arrive. We introduce *asymptotic e-detectors*, which approximate the pre-change distribution using a calibration set of size m, and prove that they asymptotically control the average run length to a false alarm as m increases. Empirically, calibration sets as small as m=30 provide effective control. We then develop adaptive monitoring strategies ranging from black-box bettors that learn from embeddings to interpretable bettors based on LLM-proposed concepts. Across three real-world datasets, including data from a clinical AI scribe deployment, adaptive monitors detect shifts faster than fixed monitors at the same false-alarm level. Further, when the true shifts are sparse and interpretable, LLM-concept monitors provide the fastest and most interpretable detection, while also guiding root-cause diagnosis for complex shifts.
Related
- Application of Machine Learning to the Computation of Parameters in the SUPG Method for Unstructured and Anisotropic Meshes
- Identifying ODEs from Unstructured Data with Causal Representation Learning
- Integrity at Stake: Statistical Detection of Anomalous In-Play Betting in Football
- Sign Betting for Misspecified Regression: Variance Adaptivity and Optimal Approximation
- Bits per Spike as a Betting Game: An Interpretable Unit for Held-Out Log-Likelihood in Neural Data Analysis
BibTeX
@misc{zhou2026learning,
title = {{Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting}},
author = {Siyu Zhou and Jialin Ouyang and Fan Xia and Adarsh Subbaswamy and Drew Prinster and Suchi Saria and Augusto Garcia-Agundez and Sarah Pollet and Jinoos Yazdany and Patrick Vossler and Jean Feng},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.17399},
eprint = {2610.17399},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.17399},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Learning to Detect Shifts in Unstructured Data Streams via Monitoring-by-Betting
AU - Siyu Zhou
AU - Jialin Ouyang
AU - Fan Xia
AU - Adarsh Subbaswamy
AU - Drew Prinster
AU - Suchi Saria
AU - Augusto Garcia-Agundez
AU - Sarah Pollet
AU - Jinoos Yazdany
AU - Patrick Vossler
AU - Jean Feng
PY - 2026
DA - 2026/10/12
PB - arXiv
DO - 10.48550/arxiv.2610.17399
UR - https://arxiv.org/abs/2610.17399
AB - Continuous monitoring of generative AI systems requires detecting distribution shifts in the unstructured data they ingest and generate. Existing monitoring methods typically control false alarms by requiring the discrepancy measure used to detect shifts to be *pre-specified*. However, for unstructured data, pre-specification creates a sensitivity--breadth trade-off: a narrowly defined measure is sensitive to the anticipated shift but may miss unexpected changes, whereas a broadly defined measure may detect a wider range of shifts but with reduced sensitivity. Although adaptively hypothesizing the shift direction can overcome this limitation, such data-dependent adaptation generally invalidates conventional false-alarm guarantees. We reconcile adaptivity with statistical validity through testing-by-betting, reframing sequential changepoint detection as a betting game in which the bettor continually updates its hypothesized shift direction, or witness function, as new observations arrive. We introduce *asymptotic e-detectors*, which approximate the pre-change distribution using a calibration set of size m, and prove that they asymptotically control the average run length to a false alarm as m increases. Empirically, calibration sets as small as m=30 provide effective control. We then develop adaptive monitoring strategies ranging from black-box bettors that learn from embeddings to interpretable bettors based on LLM-proposed concepts. Across three real-world datasets, including data from a clinical AI scribe deployment, adaptive monitors detect shifts faster than fixed monitors at the same false-alarm level. Further, when the true shifts are sparse and interpretable, LLM-concept monitors provide the fastest and most interpretable detection, while also guiding root-cause diagnosis for complex shifts.
ER -