RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization
Abstract
Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNN, a reparameterized neural network model with ReLU or tanh activations designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one-, two-, and four-dimensional function approximations, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning for an earthquake problem using real data, demonstrate that RepNN improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features. These results indicate that RepNN provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.
Related
- Spatio-Temporal Weighted Regression model with fractional-colored noise: Parameter estimation and consistency
- Practical Algebraic Parameter Estimation for Noisy Data via Gaussian Process Regression
- Zero-Complement Extrapolation: Parameter Recovery and Classification in Severely Imbalanced Logistic Regression under Rank-Deficient Regimes
- An automated ranking platform for machine learning regression models for meat spoilage prediction using multi-spectral imaging and metabolic profiling
- Optimal Tradeoffs Between Network Size and Parameter Magnitude in Neural Approximation and Minimax Regression
BibTeX
@misc{wang2026repnn,
title = {{RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization}},
author = {Yong Wang and Tao Zhou and Xuhui Meng},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2606.16575},
eprint = {2606.16575},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2606.16575},
}
RIS
TY - UNPB
TI - RepNN: Tackling spectral bias in deep neural networks for regression and PDE problems via parameter reparameterization
AU - Yong Wang
AU - Tao Zhou
AU - Xuhui Meng
PY - 2026
DA - 2026/09/28
PB - arXiv
DO - 10.48550/arxiv.2606.16575
UR - https://arxiv.org/abs/2606.16575
AB - Deep neural networks (DNNs) have achieved remarkable success in scientific computing, yet they often suffer from spectral bias in capturing oscillatory and multiscale behaviors. In this study, we investigate this limitation by examining the failure of shallow ReLU neural networks in fitting high-frequency functions. This observation identifies two important factors in resolving rapid oscillations: the initial slope scale and the distribution of partition points induced by the networks. Motivated by this analysis, we propose RepNN, a reparameterized neural network model with ReLU or tanh activations designed for high-frequency and multiscale problems. The key idea is to reparameterize the weights and biases in the first hidden layer, which enables effective control of the initial slope scale and provides an appropriate distribution of the initial partition points. Furthermore, treating the reparameterized weights and biases as trainable parameters allows the DNN to achieve adaptive frequency scaling during training. In addition, we derive quantitative estimates for the output and slope magnitudes of the reparameterized DNN to guide the initialization of the proposed method. Numerical experiments, including multiscale one-, two-, and four-dimensional function approximations, forward and inverse PDE problems in combination with physics-informed neural networks (PINNs), and operator learning for an earthquake problem using real data, demonstrate that RepNN improves the predicted accuracy of vanilla DNNs in capturing highly oscillatory features. These results indicate that RepNN provides an effective and flexible approach for overcoming spectral bias and applying DNNs to multiscale problems.
ER -