Revisiting Certified Defense with Differential Privacy on Vision Transformers
Abstract
Certified defenses that incorporate differential privacy have proven effective on Convolutional Neural Networks (CNNs), furnishing rigorous robustness guarantees against norm-bounded adversaries. However, the certified robustness behavior of Pixel Differential Privacy (PixelDP) remains largely unexplored with the self-attention architecture now dominating the deep-learning landscape. Given that the Transformer has a profound impact on our daily applications from the digital world to the physical world, it is crucial to study certified robustness through differential-privacy-style stability. To fill this research gap, we revisit this construction in Vision Transformers and identify a failure mode that is largely hidden in the convolutional setting. When noise is injected after the patch embedding, the Laplace mechanism with the inherited grouped ℓ1 sensitivity bound collapses to chance-level accuracy across noise scales, whereas the Gaussian mechanism remains trainable. This contrast isolates the source of failure: not the injected noise itself, but the geometry of the sensitivity constraint. We show that the attenuation induced by the inherited Δ1,1 projection increases with layer width and kernel size according to a random-matrix scale C/(√M+√N). Replacing the ℓ1-type constraint with a spectral-norm constraint eliminates the collapse across datasets and architectures, but creates a fundamental obstacle: the repaired models no longer satisfy the sensitivity condition required by the standard Laplace certificate. We resolve this mismatch by deriving a dimension-free (ε, δ)-privacy guarantee for the Laplace mechanism under ℓ2 sensitivity through concentration of the privacy loss.
Related
- Prophet Inequalities under Local Differential Privacy
- Performative Privacy: When Differential Privacy Maximizes Utility
- Task-aware Kernel Stein certificates for Rényi differential privacy
- Minimax Wasserstein Testing under Differential Privacy
- Asymptotic Anytime-Valid Quantile Inference under Local Differential Privacy
BibTeX
@misc{yan2026revisiting,
title = {{Revisiting Certified Defense with Differential Privacy on Vision Transformers}},
author = {Jun Yan and Weiquan Huang and Qixian Zhang and Yan Bai and Shutai Zhang},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.31310},
eprint = {2609.31310},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.31310},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Revisiting Certified Defense with Differential Privacy on Vision Transformers
AU - Jun Yan
AU - Weiquan Huang
AU - Qixian Zhang
AU - Yan Bai
AU - Shutai Zhang
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.31310
UR - https://arxiv.org/abs/2609.31310
AB - Certified defenses that incorporate differential privacy have proven effective on Convolutional Neural Networks (CNNs), furnishing rigorous robustness guarantees against norm-bounded adversaries. However, the certified robustness behavior of Pixel Differential Privacy (PixelDP) remains largely unexplored with the self-attention architecture now dominating the deep-learning landscape. Given that the Transformer has a profound impact on our daily applications from the digital world to the physical world, it is crucial to study certified robustness through differential-privacy-style stability. To fill this research gap, we revisit this construction in Vision Transformers and identify a failure mode that is largely hidden in the convolutional setting. When noise is injected after the patch embedding, the Laplace mechanism with the inherited grouped ℓ_1 sensitivity bound collapses to chance-level accuracy across noise scales, whereas the Gaussian mechanism remains trainable. This contrast isolates the source of failure: not the injected noise itself, but the geometry of the sensitivity constraint. We show that the attenuation induced by the inherited Δ_1,1 projection increases with layer width and kernel size according to a random-matrix scale C/(√M+√N). Replacing the ℓ_1-type constraint with a spectral-norm constraint eliminates the collapse across datasets and architectures, but creates a fundamental obstacle: the repaired models no longer satisfy the sensitivity condition required by the standard Laplace certificate. We resolve this mismatch by deriving a dimension-free (ε, δ)-privacy guarantee for the Laplace mechanism under ℓ_2 sensitivity through concentration of the privacy loss.
ER -