How does feature learning reshape the function space?
Abstract
Feature learning is widely regarded as the key mechanism distinguishing neural networks from fixed-kernel methods, yet its impact on the induced function space remains poorly understood. In this work, we precisely characterize how the function space spanned by the features of a two-layer neural network evolves during gradient descent training. We prove that, in the highdimensional proportional regime, after a large gradient step the post-update feature distribution is well approximated by a target-dependent spiked Gaussian covariance. This induces a data adaptive kernel that reshapes the function space and modifies its spectral structure. Our analysis reveals that feature learning can be interpreted as a distributional transformation in either parameter space or input space, equivalently as the introduction of a target-dependent kernel. In particular, it selectively amplifies eigenvalues aligned with the target direction and mixes leading eigenfunctions, coupling the top radial mode with a target-aligned quadratic harmonic. Overall, our results provide a precise function-space perspective on early-stage feature learning: rather than just rescaling a fixed kernel, gradient descent induces a data-adaptive deformation that preferentially enhances directions aligned with the signal in the data.
Related
- Diminishing Returns of Data in Sparse Autoencoders: Functional Behavior does not Guarantee Feature Reproducibility
- Instance-Specific Sharp Convergence Rate of the Add-λ Estimator via Feature Space Decomposition
- Sharp convergence rate of the estimation error of LASSO via the Feature Space Decomposition method
- Feature Space Adaptation for Effortless Gaussian Process Flows
- Residual Calibration via Local Feature-Space Refinement
BibTeX
@misc{lobo2026does,
title = {{How does feature learning reshape the function space?}},
author = {João Lobo and Bruno Loureiro and Long Tran-Than and Fanghui Liu},
howpublished = {HAL},
year = {2026},
month = may,
eprint = {2605.17718},
archivePrefix = {arXiv},
url = {https://hal.science/hal-05761651v1},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - How does feature learning reshape the function space?
AU - João Lobo
AU - Bruno Loureiro
AU - Long Tran-Than
AU - Fanghui Liu
PY - 2026
DA - 2026/05/18
PB - HAL
UR - https://hal.science/hal-05761651v1
AB - Feature learning is widely regarded as the key mechanism distinguishing neural networks from fixed-kernel methods, yet its impact on the induced function space remains poorly understood. In this work, we precisely characterize how the function space spanned by the features of a two-layer neural network evolves during gradient descent training. We prove that, in the highdimensional proportional regime, after a large gradient step the post-update feature distribution is well approximated by a target-dependent spiked Gaussian covariance. This induces a data adaptive kernel that reshapes the function space and modifies its spectral structure. Our analysis reveals that feature learning can be interpreted as a distributional transformation in either parameter space or input space, equivalently as the introduction of a target-dependent kernel. In particular, it selectively amplifies eigenvalues aligned with the target direction and mixes leading eigenfunctions, coupling the top radial mode with a target-aligned quadratic harmonic. Overall, our results provide a precise function-space perspective on early-stage feature learning: rather than just rescaling a fixed kernel, gradient descent induces a data-adaptive deformation that preferentially enhances directions aligned with the signal in the data.
ER -