Exact Finite Attention Responses From RoPE Derivatives
Abstract
We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative ∂p z(p) = A z(p): its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term CKV = ∑j (p'j − pj) εj, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.
Related
- RoPE attention is an exact forward-pass gradient step with softmax intact
- Exact Elliptic and Solitary Waves in a Fourth-Order Generalized Burgers–Huxley-Type Equation with Truncated M-Fractional Derivatives
- Unbiased Gradients, Moving Stability Boundaries: Exact Mini-Batch Geometry in Linear Self-Attention
- Exact Fusion of Softmax Attention Heads Can Require Exponentially Many Atoms
- Coherent differentiation with a monoid of degrees: derivatives, Taylor expansion, finite differences...
BibTeX
@misc{huang2026exact,
title = {{Exact Finite Attention Responses From RoPE Derivatives}},
author = {Julie Huang and Maggie Chlon and Gregory Gutin and Leon Chlon},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.14127},
eprint = {2609.14127},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.14127},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Exact Finite Attention Responses From RoPE Derivatives
AU - Julie Huang
AU - Maggie Chlon
AU - Gregory Gutin
AU - Leon Chlon
PY - 2026
DA - 2026/09/12
PB - arXiv
DO - 10.48550/arxiv.2609.14127
UR - https://arxiv.org/abs/2609.14127
AB - We derive exact local responses for attention interventions, allowing candidate edits to be scored from a cached baseline and one backward pass. The starting point is the RoPE derivative ∂_p z(p) = A z(p): its integral gives the finite positional displacement, which we carry through the softmax without linearising either rotation or normalisation. The resulting predictions achieve 95.36--96.52% sign accuracy across 92,160 executed positional edits on 768 held-out prompt sets, reducing answer-margin MAE by 73.6--82.5% against the positional Jacobian and by 36.2--50.9% against zero. For simultaneous key and value edits, the same divided-difference calculus isolates the interaction term C_KV = ∑_j (p'_j − p_j) ε_j, which is omitted by adding separate attributions. Retaining it reduces downstream margin MAE by more than a factor of nine in every setting of a 5,120-intervention sweep across two Qwen sizes, two tasks, and multiple layers; reductions against a quadratic interaction correction are 75.9--98.5%. Exactness concerns the edited attention write; downstream predictions contract that response with a baseline gradient and are evaluated by native execution. The calculus also yields a KL certificate for local approximation error, an exact query-conditioned gradient-step representation whose curvature identifies attention-preserving query directions, and minimum-norm query control. Sparse evaluation supports candidate ranking and cache decisions under explicit local distortion criteria.
ER -