Detecting Unseen Jailbreak Sources: A Multi-Source Conformal Detection Perspective
Abstract
Jailbreak defenses for large language models are usually calibrated on a fixed set of known attack methods. However, new attack methods keep appearing after deployment, and existing detectors are observed to degrade under the new attacks (Piet et al., 2025). This paper studies how to detect, from a stream of attack prompts, that the prompts come from an attack source outside the known ones. We formulate it as a sequential test whose null hypothesis is that the stream comes from one of the known sources. Methodology-wise, we compute a conformal p-value against each known source, and reject the null only when the evidence against every known source is large. Using these source-wise p-values, we build Persist-3 and analyze its stationary false-alarm probability and detection power. The technical contribution of Persist-3 lies in a detection guarantee that uses only a finite window of recent observations. Our theoretical analysis reveals that its detection probability approaches one exponentially fast as the window grows, and thus a short window suffices to detect unseen sources that are well separated from all known sources. Empirically, we validate Persist-3 on top of two recent jailbreak defenses, MTK (Zhang et al., 2026) and Prompt Guard 2 (Chennabasappa et al., 2025), and observe a significant defense enhancement.
Related
- Notes on algebraic perspective on neural network conformal theories
- How Synthetic Labels Improve Conformal Prediction: A Perspective on Conditional Coverage
- A Unified Perspective on Conformal Prediction and Wasserstein Distributionally Robust Optimization for Uncertainty Quantification
- Conformal Novelty Detection via Data-enhanced Localization
- Improving the Sensitivity of Gravitational Wave Detection with Weighted Conformal Prediction
BibTeX
@misc{qian2026detecting,
title = {{Detecting Unseen Jailbreak Sources: A Multi-Source Conformal Detection Perspective}},
author = {Xunlei Qian and Yingqian Cui and Yuping Lin and Jun Wu and Yue Xing},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.09167},
eprint = {2610.09167},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.09167},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Detecting Unseen Jailbreak Sources: A Multi-Source Conformal Detection Perspective
AU - Xunlei Qian
AU - Yingqian Cui
AU - Yuping Lin
AU - Jun Wu
AU - Yue Xing
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.09167
UR - https://arxiv.org/abs/2610.09167
AB - Jailbreak defenses for large language models are usually calibrated on a fixed set of known attack methods. However, new attack methods keep appearing after deployment, and existing detectors are observed to degrade under the new attacks (Piet et al., 2025). This paper studies how to detect, from a stream of attack prompts, that the prompts come from an attack source outside the known ones. We formulate it as a sequential test whose null hypothesis is that the stream comes from one of the known sources. Methodology-wise, we compute a conformal p-value against each known source, and reject the null only when the evidence against every known source is large. Using these source-wise p-values, we build Persist-3 and analyze its stationary false-alarm probability and detection power. The technical contribution of Persist-3 lies in a detection guarantee that uses only a finite window of recent observations. Our theoretical analysis reveals that its detection probability approaches one exponentially fast as the window grows, and thus a short window suffices to detect unseen sources that are well separated from all known sources. Empirically, we validate Persist-3 on top of two recent jailbreak defenses, MTK (Zhang et al., 2026) and Prompt Guard 2 (Chennabasappa et al., 2025), and observe a significant defense enhancement.
ER -