Speed Limit for Information Acquisition in Stochastic Learning Dynamics
Abstract
Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about latent variables in the data-generating process. The resulting inequality decomposes the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different latent variables are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for diagnosing when and how different aspects of the data-generating mechanism are acquired during stochastic learning.
Related
- GT-PSSM: Unified Probabilistic Framework for Stochastic Dynamics Modeling and Dependency Learning in Multivariate Time Series Anomaly Detection
- Global minimax risk and acquisition laws for heterogeneous information fusion
- Stochastic Knothe-Rosenblatt: Light-speed Calibration of Stochastic Local Volatility Models
- Algorithmic Information Dynamics of Learning: A Certified, Differentiable Complexity Controller for Grokking
- Dynamic Mission Risk Management via Adaptive Information Acquisition and Online Uncertainty Calibration
BibTeX
@misc{kobayashi2026speed,
title = {{Speed Limit for Information Acquisition in Stochastic Learning Dynamics}},
author = {Shuta Kobayashi and Andreas Dechant},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.08219},
eprint = {2609.08219},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.08219},
}
RIS
TY - UNPB
TI - Speed Limit for Information Acquisition in Stochastic Learning Dynamics
AU - Shuta Kobayashi
AU - Andreas Dechant
PY - 2026
DA - 2026/09/08
PB - arXiv
DO - 10.48550/arxiv.2609.08219
UR - https://arxiv.org/abs/2609.08219
AB - Neural networks acquire internal representations through learning. In this work, we formulate stochastic gradient descent (SGD) as a Markovian stochastic process and derive a Fisher-information flow speed limit that bounds the rate at which trainable parameters can acquire information about latent variables in the data-generating process. The resulting inequality decomposes the information flow into drift and noise contributions, thereby quantifying the roles of deterministic learning forces and SGD-induced fluctuations from an information-theoretic perspective. We verify the bound in analytically tractable basis-function linear regression, where the information budget predicted by the bound reproduces the ordering and characteristic time scales with which different latent variables are encoded in the learned parameters. These results establish Fisher-information speed limits as a quantitative framework for diagnosing when and how different aspects of the data-generating mechanism are acquired during stochastic learning.
ER -