Phase Transition from Random Features to Fully Trainable Networks
Abstract
Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent α>1 and leverage compatibility, we establish a squared L2 approximation rate 𝒪((κPlog(m/δ)/m)α) for two-layer networks of width m and trainable proportion P, where κP=(P/κ1*+(1−P)/κ0*)−1. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least 1−δ. The constants κ0* and κ1* quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension d, we verify these assumptions with α=(d+2)/(d−1) and κ0*=4√2τ(d+1)/d for initialization scale τ≥1, and κ1*=1 under ridge leverage sampling. For d=5 and τ=10, a trainable proportion of approximately 4.1% captures 90% of the reduction in the approximation bound from P=0 to P=1. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
Related
- Phase transition of random non-uniform hypergraphs
- A first order phase transition for lower tails in sparse random graphs
- A Sharp Phase Transition for Killed Branching Random Walks with Heavy-Tailed Displacements
- The localization transition for the directed polymer in a random environment is smooth
- Dynamic models with p parameters are identified by 2p+1 random features
BibTeX
@misc{he2026phase,
title = {{Phase Transition from Random Features to Fully Trainable Networks}},
author = {Xin He and Huan Lao and Xiliang Lu and Shijun Zhang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.31663},
eprint = {2610.31663},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.31663},
}
RIS
TY - UNPB
TI - Phase Transition from Random Features to Fully Trainable Networks
AU - Xin He
AU - Huan Lao
AU - Xiliang Lu
AU - Shijun Zhang
PY - 2026
DA - 2026/10/22
PB - arXiv
DO - 10.48550/arxiv.2610.31663
UR - https://arxiv.org/abs/2610.31663
AB - Constructing accurate and computationally tractable approximation spaces is a fundamental task in scientific computing. Partially trainable neural networks bridge fixed random features and fully adaptive representations. This paper quantifies a phase transition pattern in approximation bounds and observes analogous behavior in training errors: a rapid but smooth initial improvement followed by diminishing returns, with a sharper transition under greater initialization mismatch. Under polynomial spectral decay with exponent α>1 and leverage compatibility, we establish a squared L_2 approximation rate 𝒪((κ_Plog(m/δ)/m)^α) for two-layer networks of width m and trainable proportion P, where κ_P=(P/κ_1^*+(1−P)/κ_0^*)^−1. The bound holds uniformly over the unit ball of a reproducing kernel Hilbert space (RKHS) with probability at least 1−δ. The constants κ_0^* and κ_1^* quantify the fixed and trainable groups' sampling mismatches relative to the target RKHS. For spherical biased-ReLU features in dimension d, we verify these assumptions with α=(d+2)/(d−1) and κ_0^*=4√2τ(d+1)/d for initialization scale τ≥1, and κ_1^*=1 under ridge leverage sampling. For d=5 and τ=10, a trainable proportion of approximately 4.1% captures 90% of the reduction in the approximation bound from P=0 to P=1. A structured three-layer example illustrates how a few trainable first-layer neurons exploit fixed downstream blocks through nonlinear composition. Numerical experiments across diverse settings, including networks with up to 15 layers, exhibit similar phase transitions in training errors.
ER -