The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
Abstract
The Forward-Forward Algorithm (FFA) replaces backpropagation (BP) with layer-wise local contrastive objectives, eliminating the backward pass and the need to retain intermediate activations, yet suffers a persistent performance gap with BP that worsens with depth. This paper diagnoses two structural failure modes: an optimization floor arising from concurrent local updates; and a geometric collapse of layer representations driven by the local update mechanism. On the optimization side, we prove that the FFA loss satisfies the Polyak--Lojasiewicz inequality at each layer; however, simultaneous layer updates induce inter-layer representation-distribution drift, so each layer optimizes against a moving input distribution and incurs an error floor. On the representational side, the pairwise similarity kernel of layer representations contracts exponentially toward rank one as depth increases, collapsing the diversity of per-layer error signals. This collapse bounds FFA's effective learning capacity, which measures the diversity of gradient information across layers, independently of depth, whereas BP's chain-rule signal preserves per-layer diversity, yielding a capacity that scales with depth.
Related
- From forward-backward sweep to backpropagation for optimal control in tumor-immune dynamics.
- A samplewise backpropagation method for neural networks driven by fractional Brownian motion
- LocalProp: Neuro-Localized Memory-Efficient Backpropagation
- Truncated backpropagation through time induces a two-sided gradient turnpike property
- Beyond Backpropagation: Monte Carlo Method Can Train Deep Neural Networks
BibTeX
@misc{wu2026price,
title = {{The Price of Locality: Why Forward-Forward Underperforms Backpropagation?}},
author = {Zhaoxian Wu and Haichuan Liu and Tianyi Chen},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.33240},
eprint = {2609.33240},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.33240},
}
RIS
TY - UNPB
TI - The Price of Locality: Why Forward-Forward Underperforms Backpropagation?
AU - Zhaoxian Wu
AU - Haichuan Liu
AU - Tianyi Chen
PY - 2026
DA - 2026/09/27
PB - arXiv
DO - 10.48550/arxiv.2609.33240
UR - https://arxiv.org/abs/2609.33240
AB - The Forward-Forward Algorithm (FFA) replaces backpropagation (BP) with layer-wise local contrastive objectives, eliminating the backward pass and the need to retain intermediate activations, yet suffers a persistent performance gap with BP that worsens with depth. This paper diagnoses two structural failure modes: an optimization floor arising from concurrent local updates; and a geometric collapse of layer representations driven by the local update mechanism. On the optimization side, we prove that the FFA loss satisfies the Polyak--Lojasiewicz inequality at each layer; however, simultaneous layer updates induce inter-layer representation-distribution drift, so each layer optimizes against a moving input distribution and incurs an error floor. On the representational side, the pairwise similarity kernel of layer representations contracts exponentially toward rank one as depth increases, collapsing the diversity of per-layer error signals. This collapse bounds FFA's effective learning capacity, which measures the diversity of gradient information across layers, independently of depth, whereas BP's chain-rule signal preserves per-layer diversity, yielding a capacity that scales with depth.
ER -