Neural Scaling Laws of Transformer Operator Network
Abstract
Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for Hölder-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.
Related
- LGNO: A Local-Global Neural Operator for Hyperbolic Conservation Laws
- Structured Neural Network Models for Anisotropic Magnetic Constitutive Laws
- Neural Network-Assisted Refinement of Traditional Schemes for One-Dimensional Scalar Conservation Laws
- Well-posed by Design: Learning Constitutive Laws from Velocity Data using Convex Neural Network Potentials
- Derivation of the Wave Kinetic Equation: Full Range of Scaling Laws
BibTeX
@misc{yan2026neural,
title = {{Neural Scaling Laws of Transformer Operator Network}},
author = {Haoran Yan and Zhongjie Shi and Yuanzhe Xi and Peng Chen and Wenjing Liao},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.33533},
eprint = {2609.33533},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.33533},
}
RIS
TY - UNPB
TI - Neural Scaling Laws of Transformer Operator Network
AU - Haoran Yan
AU - Zhongjie Shi
AU - Yuanzhe Xi
AU - Peng Chen
AU - Wenjing Liao
PY - 2026
DA - 2026/09/27
PB - arXiv
DO - 10.48550/arxiv.2609.33533
UR - https://arxiv.org/abs/2609.33533
AB - Transformers have emerged as powerful architectures for learning solution operators of physical systems. Empirically the prediction error has been observed to decrease when the data size and model size increase, suggesting neural scaling behavior. Yet a theoretical understanding of such scaling laws for transformer-based operator learning remains limited. In this work, we develop a theoretical framework for characterizing the approximation and generalization errors of transformer-based operator learning. Our analysis builds on a local-to-global approximation principle that is naturally aligned with the softmax attention mechanism and yields discretization-invariant output functions. On approximation theory, we derive a universal approximation error of transformer-based operator learning for Hölder-regular operators. On generalization theory, we establish a power scaling law between the prediction error and the training data size. The rate of convergence represented by the scaling exponent explicitly reflects the dimensions of the input and output domains, the regularity of the underlying functions and operators, and crucially, the intrinsic dimension of the input function class. By exploiting this intrinsic low-dimensional structure, our analysis yields a power-law generalization rate for operator learning, in contrast to the logarithmic-type power-law rates appearing in existing analyses of operator learning with feedforward neural networks. Numerical experiments validate the predicted power-law scaling and confirm that the convergence rate varies systematically with the intrinsic dimension of the input function class.
ER -