Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
Abstract
Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping. In this work, we study the value of truncation in looped transformer training in a systematic way: we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower validation loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and only 8% more training compute.
Related
- Rotate the Loop: Blockwise Conformality for Looped Transformers
- Truncated backpropagation through time induces a two-sided gradient turnpike property
- Scaling Limits of Long-Context Transformers
- A stable rank-adaptive step-and-truncate finite volume method for Vlasov transport on domains with piecewise linear boundaries
- A Constructive Framework for Generalized Fourier Transforms via Truncate-and-Generalized Limits
BibTeX
@misc{xiong2026truncate,
title = {{Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation}},
author = {Xuyuan Xiong and Siyu Chen and Zihan Shao and Yue Su and Weiyang Jin and Ping Luo and Zhuoran Yang and Xihui Liu},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.26030},
eprint = {2610.26030},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.26030},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Truncate or Not? A Scaling Study of Looped Transformers with Truncated Backpropagation
AU - Xuyuan Xiong
AU - Siyu Chen
AU - Zihan Shao
AU - Yue Su
AU - Weiyang Jin
AU - Ping Luo
AU - Zhuoran Yang
AU - Xihui Liu
PY - 2026
DA - 2026/10/20
PB - arXiv
DO - 10.48550/arxiv.2610.26030
UR - https://arxiv.org/abs/2610.26030
AB - Looped transformers reuse a shared block of layers to increase computational depth without increasing the number of trainable parameters. They are trained with backpropagation through time (BPTT). To save training compute, BPTT is often truncated to the last few loops, but truncation yields biased gradients that may weaken the benefit of looping. In this work, we study the value of truncation in looped transformer training in a systematic way: we jointly vary model size, training tokens, and forward and backward loop counts across nearly 600 training runs and fit a unified scaling law that captures the effects of looping and truncation through effective parameters. Despite the training-compute savings of truncation, our fitted law favors full BPTT when forward and backward loop counts are jointly optimized. In matched-compute runs, full BPTT through fewer loops consistently outperforms truncated settings of the same compute. The law further shows that at fixed model size, the compute-optimal loop count and training-token budget both grow with compute. When model size is also free, the law identifies two fully trained loops as the best looped setting. Experimentally, this setting achieves lower validation loss than the compute-optimal non-looped model with about 27% fewer non-embedding parameters and only 8% more training compute.
ER -