Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines
Abstract
Recursive feature machines (RFMs) learn representations of data by alternating between fitting a predictor to a dataset and updating features of that predictor using the average gradient outer product (AGOP). Connections between AGOPs and feature learning in neural networks motivate linear RFMs as a simple setting for analyzing how representations evolve during training. Here, we study the dynamics and statistics of linear RFM in noisy multi-output regression with isotropic sub-Gaussian input data and targets generated by a low-rank teacher matrix of dimension d. We extend the known connection between linear RFM and iteratively reweighted least squares from the interpolating setting to ridge-regularized multi-output regression with noise. We show that the learned feature matrix remains close to its infinite-data ideal counterpart at every iteration. Namely, for n samples, we show the error in the feature matrix decays as O(√(d/n)) with high probability. Experiments on real-world text and single-cell gene-expression data illustrate the features learned by this simple linear model.
Related
- Cross-validated feature selection using least-squares support vector machines for binary classification
- Chapter 2: Geometry of the Fitness Surface and Trajectory Dynamics of Replicator Systems
- Exploiting UAV Attitude for Covert Communications: Dynamics-Consistent Trajectory-Attitude Co-Design
- The Geometry of Statistical Feature Learning in Mean-Field Langevin Dynamics
- Feature Information Dynamics in Diffusion
BibTeX
@misc{cheng2026exact,
title = {{Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines}},
author = {Andrew Cheng and Bobak T. Kiani and Yue M. Lu and Adityanarayanan Radhakrishnan},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.09196},
eprint = {2610.09196},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.09196},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Exact Dynamics and Finite-Sample Trajectory Recovery of Linear Recursive Feature Machines
AU - Andrew Cheng
AU - Bobak T. Kiani
AU - Yue M. Lu
AU - Adityanarayanan Radhakrishnan
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.09196
UR - https://arxiv.org/abs/2610.09196
AB - Recursive feature machines (RFMs) learn representations of data by alternating between fitting a predictor to a dataset and updating features of that predictor using the average gradient outer product (AGOP). Connections between AGOPs and feature learning in neural networks motivate linear RFMs as a simple setting for analyzing how representations evolve during training. Here, we study the dynamics and statistics of linear RFM in noisy multi-output regression with isotropic sub-Gaussian input data and targets generated by a low-rank teacher matrix of dimension d. We extend the known connection between linear RFM and iteratively reweighted least squares from the interpolating setting to ridge-regularized multi-output regression with noise. We show that the learned feature matrix remains close to its infinite-data ideal counterpart at every iteration. Namely, for n samples, we show the error in the feature matrix decays as O(√(d/n)) with high probability. Experiments on real-world text and single-cell gene-expression data illustrate the features learned by this simple linear model.
ER -