An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
Abstract
The Dobrushin coefficient of each attention-rollout factor satisfies κ(1/2(I+A))≤1/2(1+κ(A)), and multiplying these inequalities over layers bounds the coefficient of the whole rollout. We characterise exactly when the layerwise bound is tight: equality holds if and only if some token pair attaining κ(A) is mutually self-dominant - each of the two attends to itself at least as strongly as the other attends to it. The condition is far from automatic: uniformly random stochastic matrices satisfy it only 24-30% of the time. When tested on the head-averaged attention of each individual input and restricted to content tokens - image patches, words or tabular features, excluding cls, register and separator tokens - the condition holds for essentially every input at every layer of DINOv2 (three model sizes), RoBERTa and DistilBERT. In the supervised models DeiT-B and ViT-B/16 it holds for 91% and 64% of input-layer pairs respectively, with all failures occurring late in depth. In FT-Transformer trained on two standard tabular benchmarks it holds for only 11-44% of input-layer pairs. The special tokens account for almost all failures in DINOv2 and the language models: when they are included, the condition holds for only 82-97% of input-layer pairs.
Related
- Jensen' s trace inequality with equality condition and noncommutative Lamperti's theorem
- Asymptotic Analysis of Discrete-Time Self-Attention Dynamics in Transformers
- Local Geometric Mixing via Dobrushin Contraction with Applications to Diffusion Path Monte Carlo and the Proximal Sampler
- BASIS: Batchwise Advantage Estimation from Single-Rollout Information Sharing for LLM Reasoning
- Occupation-Weighted Performance Bounds for Rollout Policies in Stochastic Shortest Path Problems
BibTeX
@misc{rola2026equality,
title = {{An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers}},
author = {Przemysław Rola},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.05558},
eprint = {2610.05558},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.05558},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - An equality condition for the Dobrushin bound on attention rollout and how often it holds in trained transformers
AU - Przemysław Rola
PY - 2026
DA - 2026/10/04
PB - arXiv
DO - 10.48550/arxiv.2610.05558
UR - https://arxiv.org/abs/2610.05558
AB - The Dobrushin coefficient of each attention-rollout factor satisfies κ(1/2(I+A))≤1/2(1+κ(A)), and multiplying these inequalities over layers bounds the coefficient of the whole rollout. We characterise exactly when the layerwise bound is tight: equality holds if and only if some token pair attaining κ(A) is mutually self-dominant - each of the two attends to itself at least as strongly as the other attends to it. The condition is far from automatic: uniformly random stochastic matrices satisfy it only 24-30% of the time. When tested on the head-averaged attention of each individual input and restricted to content tokens - image patches, words or tabular features, excluding cls, register and separator tokens - the condition holds for essentially every input at every layer of DINOv2 (three model sizes), RoBERTa and DistilBERT. In the supervised models DeiT-B and ViT-B/16 it holds for 91% and 64% of input-layer pairs respectively, with all failures occurring late in depth. In FT-Transformer trained on two standard tabular benchmarks it holds for only 11-44% of input-layer pairs. The special tokens account for almost all failures in DINOv2 and the language models: when they are included, the condition holds for only 82-97% of input-layer pairs.
ER -