Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks
Abstract
Understanding why deep neural networks make a given prediction is of great importance for their safe deployment. In computer vision, saliency maps, which highlight the image region most influential for a prediction, remain a widely-used form of explanation. An empirical observation is the apparent sparsity of gradient saliency maps of adversarially-trained neural networks. In this paper, we propose a theoretical explanation of this phenomenon for two-layer ReLU networks. We build on the established equivalence of adversarial training to the minimization of the empirical risk with weight-decay penalization and an added adversarial total variation term -- valid for certain loss functions. As the number of data points and neurons grows and the regularization parameters are sent to zero at appropriate rates, we prove that minimizers converge to a Bayes classifier with minimal gradient and Barron norm. Sparsity appears since for adversarial training with ℓ∞-attacks the gradient norm is anisotropic and favors axis-aligned / sparse gradients. We illustrate our theoretical findings experimentally by evaluating the gradient ℓ1-norm and thresholded sparsity of naturally versus adversarially trained models.
Related
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Linear-Time Bilevel Clipping operators for Structured Sparsity in Neural Networks
- Sparsity-Corrected Tensor Clustering for Multi-Layer Networks
- Circuit-Embedded Feeder-State Reconstruction for Low-Voltage Distribution Networks under Extreme Measurement Sparsity
- Edge-preserving Bayesian neural networks via hierarchical sparsity-promoting priors
BibTeX
@misc{lunk2026explaining,
title = {{Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks}},
author = {Yannick Lunk and Atell Yehor Krasnopolsky and Damien Garreau and Leon Bungert},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.10666},
eprint = {2610.10666},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.10666},
}
RIS
TY - UNPB
TI - Explaining the Saliency Map Sparsity of Adversarially-Trained Neural Networks
AU - Yannick Lunk
AU - Atell Yehor Krasnopolsky
AU - Damien Garreau
AU - Leon Bungert
PY - 2026
DA - 2026/10/07
PB - arXiv
DO - 10.48550/arxiv.2610.10666
UR - https://arxiv.org/abs/2610.10666
AB - Understanding why deep neural networks make a given prediction is of great importance for their safe deployment. In computer vision, saliency maps, which highlight the image region most influential for a prediction, remain a widely-used form of explanation. An empirical observation is the apparent sparsity of gradient saliency maps of adversarially-trained neural networks. In this paper, we propose a theoretical explanation of this phenomenon for two-layer ReLU networks. We build on the established equivalence of adversarial training to the minimization of the empirical risk with weight-decay penalization and an added adversarial total variation term -- valid for certain loss functions. As the number of data points and neurons grows and the regularization parameters are sent to zero at appropriate rates, we prove that minimizers converge to a Bayes classifier with minimal gradient and Barron norm. Sparsity appears since for adversarial training with ℓ_∞-attacks the gradient norm is anisotropic and favors axis-aligned / sparse gradients. We illustrate our theoretical findings experimentally by evaluating the gradient ℓ_1-norm and thresholded sparsity of naturally versus adversarially trained models.
ER -