RoPE attention is an exact forward-pass gradient step with softmax intact
Abstract
We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix ΔMi satisfying yi = μi + ui⊤ ΔMi, where μi is the uniform mean of the attended values and ui is the augmented query input. The construction applies the classical exponential divided difference ρ= ϕ1 to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same function connects the RoPE generator to exact positional finite differences. We derive a tokenwise formula for the error of reusing one query's matrix and prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout. Reconstruction checks and frozen-reuse calibration on one pretrained Qwen2.5-0.5B layer verify the representation and quantify the correction required when one query's matrix is reused.
Related
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
- In-context classification: Calibration and noise in linear attention with softmax readout
- Bulk-edge sticking beyond the Perron mode in Gaussian softmax attention
- Learning Decision-Stump Thresholds in Context: Dynamics of Softmax Attention
BibTeX
@misc{huang2026rope,
title = {{RoPE attention is an exact forward-pass gradient step with softmax intact}},
author = {Julie Huang and Maggie Chlon and Leon Chlon},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.06685},
eprint = {2609.06685},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.06685},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - RoPE attention is an exact forward-pass gradient step with softmax intact
AU - Julie Huang
AU - Maggie Chlon
AU - Leon Chlon
PY - 2026
DA - 2026/09/06
PB - arXiv
DO - 10.48550/arxiv.2609.06685
UR - https://arxiv.org/abs/2609.06685
AB - We derive an exact gradient-step representation of the RoPE-softmax forward pass. For every deterministic RoPE-softmax attention head with arbitrary affine projection weights, we construct a query-dependent effective matrix ΔM_i satisfying y_i = μ_i + u_i^⊤ ΔM_i, where μ_i is the uniform mean of the attended values and u_i is the augmented query input. The construction applies the classical exponential divided difference ρ= ϕ_1 to retain the softmax exactly. Its positive coefficients give a unit gradient-step representation on a query-conditioned quadratic objective. The same function connects the RoPE generator to exact positional finite differences. We derive a tokenwise formula for the error of reusing one query's matrix and prove that a nonconstant finite-cache head cannot admit a globally exact affine query readout. Reconstruction checks and frozen-reuse calibration on one pretrained Qwen2.5-0.5B layer verify the representation and quantify the correction required when one query's matrix is reused.
ER -