Replica Fragmentation and Glassy Dynamics in Parity Learning
Abstract
We study how independently trained Transformer neural networks reconstruct a binary string from its local domain walls. Runs sharing the data and training protocol can realize different functions. We treat them as replicas and measure truth alignment m, prediction confidence qself, and cross-replica agreement qcross. Confident disagreement defines the finite-size replica fragmentation that we call glass-like. With small training sets, replicas predict all training examples correctly but remain confident in incorrect predictions for unseen inputs, a regime we call memorization. With larger sets, runs can generalize and then retreat. Retreat occurs when outputs start to deviate from truth while confidence remains high. The frontier between learned and unlearned outputs recedes toward shorter strings. Many later recover as the frontier advances again. The self--cross gap qself−qcross clearly distinguishes the three learning regimes of memorization, retreat, and recovery. With overall and position-dependent truth alignment subtracted, their residual correlations also differ: memorizing replicas have weakly and uniformly correlated residuals, while retreat has the largest fraction of replica pairs whose residuals are anti-correlated. That is, on the inputs where one replica of such a pair does better than its average, the other tends to do worse. These finite-size observations distinguish persistent memorization from ongoing retreat--recovery dynamics. The thermodynamic and long-time limits, and extensions to other learning tasks, remain open.
Related
- Stable valleys in the glassy landscape of a low-density parity-check (LDPC) code
- On non-phononic modes and glassy dynamics
- Glassy dynamics of metropolitan human mobility frozen near the gravitational equilibrium
- Glassy dynamics with softened kinetic constraints on a noisy quantum computer
- Interaction-Dependent Molecular Probing of Heterogeneous Glassy Dynamics in Polymer Melts
BibTeX
@misc{ma2026replica,
title = {{Replica Fragmentation and Glassy Dynamics in Parity Learning}},
author = {Han Ma},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.08503},
eprint = {2610.08503},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.08503},
}
RIS
TY - UNPB
TI - Replica Fragmentation and Glassy Dynamics in Parity Learning
AU - Han Ma
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.08503
UR - https://arxiv.org/abs/2610.08503
AB - We study how independently trained Transformer neural networks reconstruct a binary string from its local domain walls. Runs sharing the data and training protocol can realize different functions. We treat them as replicas and measure truth alignment m, prediction confidence q_self, and cross-replica agreement q_cross. Confident disagreement defines the finite-size replica fragmentation that we call glass-like. With small training sets, replicas predict all training examples correctly but remain confident in incorrect predictions for unseen inputs, a regime we call memorization. With larger sets, runs can generalize and then retreat. Retreat occurs when outputs start to deviate from truth while confidence remains high. The frontier between learned and unlearned outputs recedes toward shorter strings. Many later recover as the frontier advances again. The self--cross gap q_self−q_cross clearly distinguishes the three learning regimes of memorization, retreat, and recovery. With overall and position-dependent truth alignment subtracted, their residual correlations also differ: memorizing replicas have weakly and uniformly correlated residuals, while retreat has the largest fraction of replica pairs whose residuals are anti-correlated. That is, on the inputs where one replica of such a pair does better than its average, the other tends to do worse. These finite-size observations distinguish persistent memorization from ongoing retreat--recovery dynamics. The thermodynamic and long-time limits, and extensions to other learning tasks, remain open.
ER -