Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning

Abstract

During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.

Deep Learning as Neural Low-Degree Filtering: A Spectral Theory of Hierarchical Feature Learning
Yatin Dandi, Matteo Vilucchio, Luca Arnaboldi +2
arXiv4 Sep 2026PreprintCC BY 4.0Full textStatistics Condensed matterarXiv
Learning Kolmogorov Backward Operators for Spectral Analysis and Forecasting: A Data-Driven Variational Galerkin Framework
Yuanchao Xu, Claire Valva, Ben Gao +1
arXiv29 Oct 2026PreprintCC BY 4.0Full textDynamical systemsarXiv
Dual-domain self-supervised feature alignment via spectral–spatial representation learning for deepfake anomaly detection
Yi He
PLOS One11 Sep 2026Peer-reviewedCC BY 4.0Full textBioinformatics & computational biology ProbabilityPLOS
Minimax rates for learning spectral Barron functions by deep ReLU neural networks
Songqiu Ma, Yunfei Yang
arXiv30 Sep 2026PreprintarXiv licenceFull textNumerical analysis & scientific computing StatisticsarXiv
Backward Homogeneous Heat Equation with Ψ-Caputo Time Derivative and Spectral Fractional Laplacian
Nelson R. Ojeda, Ruben D. Spies
arXiv20 Oct 2026PreprintarXiv licenceFull textAnalysis & PDEsarXiv

Cite

BibTeX RIS

BibTeX

@misc{dandi2026neural,
  title = {{Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning}},
  author = {Yatin Dandi and Matteo Vilucchio and Hugo Tabanelli and Florent Krzakala},
  howpublished = {arXiv},
  year = {2026},
  month = oct,
  doi = {10.48550/arxiv.2610.32849},
  eprint = {2610.32849},
  archivePrefix = {arXiv},
  url = {https://arxiv.org/abs/2610.32849},
  note = {CC BY 4.0},
}

RIS

TY  - UNPB
TI  - Neural LoFi with Backward Coupling: A Spectral Theory of Cross-Layer Feature Learning
AU  - Yatin Dandi
AU  - Matteo Vilucchio
AU  - Hugo Tabanelli
AU  - Florent Krzakala
PY  - 2026
DA  - 2026/10/23
PB  - arXiv
DO  - 10.48550/arxiv.2610.32849
UR  - https://arxiv.org/abs/2610.32849
AB  - During joint training, backpropagation allows later layers to reshape earlier representations, a feedback mechanism absent from purely layer-wise learning. We extend Neural Low-Degree Filtering (Neural LoFi), a recently proposed layer-wise spectral surrogate for feature learning, to capture this interaction. Starting from the exact backpropagation gradients, we derive a local coupling between a neuron's incoming weights and its corresponding next-layer column. In a controlled small-initialization regime with smooth activations and fixed surroundings, these dynamics amplify singular modes of a cross-layer correlation operator. Each growing mode pairs an earlier feature with an upper-layer response through shared neuron coefficients. This motivates a backward-coupling algorithm in which upper-layer sensitivities generate sample-dependent effective labels for earlier spectral steps, while corrected features refine upward transmission. A function-space variational principle characterizes the selected feature-response pairs by maximizing the label correlation of their product under kernel-norm constraints. For three-layer networks, we derive self-consistent continuous-time dynamics for the filtered hidden-layer features, the activation kernels, and the network output under the proposed backward coupling. In a hierarchical multi-index model, we establish a feature-recovery separation: backward coupling recovers lower-layer directions invisible to the original forward label-weighted statistic. The linear specialization recovers residual-dependent singular-mode dynamics, while paired modes provide a dynamical mechanism for linked low-rank covariance components consistent with the rainbow-network picture. Experiments with fully connected and convolutional networks on binary CIFAR-10 show improved accuracy over forward Neural LoFi at matched widths.
ER  -

Abstract

Related

Cite