Rotate the Loop: Blockwise Conformality for Looped Transformers
Abstract
Looped transformers reuse one shared block across many recurrent steps, so depth becomes a quantity that can be spent at runtime rather than a stack of distinct layers. Those extra steps help only if a write made on one recurrent step is still readable on later ones, and that depends on the linear map applied to the residual between recurrent steps. We show that whether the loop learns to track state is a geometric question about the form of this map, and we propose blockwise conformal injections: the residual splits into coordinate-aligned groups and rotates inside each one, so every group has its own timescale while the geometry stays exact within it. Our experiments show a performance increase on a demanding form of state tracking - last-token accuracy when the rest of the prefix is already correct - and on language modelling, where a signal written earlier has to be retrieved at the end of the sequence.
Related
- Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
- Rotate Once, Read Many Times: on the Output Noise of Multi-Value Bootstrapping
- Wait or Rotate: Statistical Repair of the Backward Pass in Over-the-Air Split Learning Under Channel Aging
- Same path, different walk: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG
- Statistical Gains from Looped Estimation under Parameter Budgets
BibTeX
@misc{zemtsov2026rotate,
title = {{Rotate the Loop: Blockwise Conformality for Looped Transformers}},
author = {Alexey Zemtsov and Albina Klepach and Vadim Lomshakov and Andrei Polubarov and Nikita Lyubaykin and Alexander Derevyagin and Vladislav Kurenkov and Alexander Nikulin},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.14030},
eprint = {2610.14030},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.14030},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Rotate the Loop: Blockwise Conformality for Looped Transformers
AU - Alexey Zemtsov
AU - Albina Klepach
AU - Vadim Lomshakov
AU - Andrei Polubarov
AU - Nikita Lyubaykin
AU - Alexander Derevyagin
AU - Vladislav Kurenkov
AU - Alexander Nikulin
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.14030
UR - https://arxiv.org/abs/2610.14030
AB - Looped transformers reuse one shared block across many recurrent steps, so depth becomes a quantity that can be spent at runtime rather than a stack of distinct layers. Those extra steps help only if a write made on one recurrent step is still readable on later ones, and that depends on the linear map applied to the residual between recurrent steps. We show that whether the loop learns to track state is a geometric question about the form of this map, and we propose blockwise conformal injections: the residual splits into coordinate-aligned groups and rotates inside each one, so every group has its own timescale while the geometry stays exact within it. Our experiments show a performance increase on a demanding form of state tracking - last-token accuracy when the rest of the prefix is already correct - and on language modelling, where a signal written earlier has to be retrieved at the end of the sequence.
ER -