Same path, different walk: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG
Abstract
Recurrent Transformers reusing their weights rather than stacking L distinct layers are becoming widely adopted due to their parameter efficiency [1,2,3]. However, the exact representational and dynamical differences between looped and stacked architectures remain uncharacterized. This paper presents a controlled study on the example of bViT model [1] applying one weight-tied block L times. We train two models: bViT and standard ViT [4] on 12-lead electrocardiogram (ECG) classification tasks from the PTB-XL dataset under identical training protocols. Despite an 8.9× parameter reduction, bViT achieves accuracy parity with ViT. Geometric similarity metrics demonstrate that both architectures construct comparable latent representations in an equivalent canonical order. Crucially, their dynamics differ: bViT exhibits smaller step sizes and inter-patient sensitivity, as well as near-neutral behavior away from the data manifold, whereas ViT exhibits collapsing dimensionality of representations and out-of-distribution feature expansion.
Related
- Tensor Decomposition of Transformer Key-Value Caches: Spectral Structure and Format Comparison
- Task-Oriented Quantization for Quadratic Scheduling: Centroid Water-Filling and Power-Diagram Encoders
- Evaluating cross-encoders for semantic similarity assessment in psychological questionnaires
- It Takes Two Moments: Rigidity and Limits of Continuous Encoders for Over-the-Air Computation
- SiamJEPA: On the Role of Siamese Student Encoders in JEPA
BibTeX
@misc{olszowiec2026same,
title = {{Same path, different walk: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG}},
author = {Pawel Olszowiec and Michal Byra and Grzegorz Gruszczynski and Grzegorz Stefanski and Alberto Presta},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2609.15498},
eprint = {2609.15498},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.15498},
}
RIS
TY - UNPB
TI - Same path, different walk: a mechanistic comparison of looped and stacked transformer encoders on 12-lead ECG
AU - Pawel Olszowiec
AU - Michal Byra
AU - Grzegorz Gruszczynski
AU - Grzegorz Stefanski
AU - Alberto Presta
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2609.15498
UR - https://arxiv.org/abs/2609.15498
AB - Recurrent Transformers reusing their weights rather than stacking L distinct layers are becoming widely adopted due to their parameter efficiency [1,2,3]. However, the exact representational and dynamical differences between looped and stacked architectures remain uncharacterized. This paper presents a controlled study on the example of bViT model [1] applying one weight-tied block L times. We train two models: bViT and standard ViT [4] on 12-lead electrocardiogram (ECG) classification tasks from the PTB-XL dataset under identical training protocols. Despite an 8.9× parameter reduction, bViT achieves accuracy parity with ViT. Geometric similarity metrics demonstrate that both architectures construct comparable latent representations in an equivalent canonical order. Crucially, their dynamics differ: bViT exhibits smaller step sizes and inter-patient sensitivity, as well as near-neutral behavior away from the data manifold, whereas ViT exhibits collapsing dimensionality of representations and out-of-distribution feature expansion.
ER -