Replica Analysis of Overfitting in Vector Generalized Linear Models
Abstract
We show how the successful recent statistical physics based theory of overfitting in Generalized Linear Models (GLMs) can be generalized to a much larger family of regression models: the so-called Vector Generalized Linear Models (VGLMs), in which not a single linear combination of covariates drives the outcome, but where one has an arbitrary finite number of such linear combinations. The latter more general and complex models can describe more sophisticated statistical patterns in data, but are even more prone to overfitting than GLMs due to their significantly larger number of parameters; this often limits their use. We show that the generalization of the overfitting theory from GLMs to VGLMs causes many mathematical challenges that were absent from the earlier work, mostly related to the increased dimensional complexity of all order parameters in the theory, but that the new problem is once more solvable using the replica method. This opens up the field to many novel and welcome applications, especially in post-genome medical inference and data analysis, where overfitting is and remains a major barrier to progress. After developing the general VGLM overfitting theory, within the replica symmetric (RS) ansatz, we illustrate its merits and its correctness via application to several relevant VGLM regression models.
Related
- Generalized replica manifolds. Part I. Surgery and averaging
- Vector rogue wave patterns associated with generalized Hermite and Okamoto polynomials
- Generalized Forcing Method: Generation of Diverse Data for Training Linear Transport PDE Closure Models
- Linear Algebra of Generalized Contextuality in All Prepare-Transform-Measure Scenarios
- Limits of the Non-Linear Generalized Langevin Equation: Cross-Correlations, Irreversibility and Desynchronization
BibTeX
@misc{jr2026replica,
title = {{Replica Analysis of Overfitting in Vector Generalized Linear Models}},
author = {M Diaz-Ortiz Jr and ACC Coolen},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.36904},
eprint = {2610.36904},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.36904},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Replica Analysis of Overfitting in Vector Generalized Linear Models
AU - M Diaz-Ortiz Jr
AU - ACC Coolen
PY - 2026
DA - 2026/10/27
PB - arXiv
DO - 10.48550/arxiv.2610.36904
UR - https://arxiv.org/abs/2610.36904
AB - We show how the successful recent statistical physics based theory of overfitting in Generalized Linear Models (GLMs) can be generalized to a much larger family of regression models: the so-called Vector Generalized Linear Models (VGLMs), in which not a single linear combination of covariates drives the outcome, but where one has an arbitrary finite number of such linear combinations. The latter more general and complex models can describe more sophisticated statistical patterns in data, but are even more prone to overfitting than GLMs due to their significantly larger number of parameters; this often limits their use. We show that the generalization of the overfitting theory from GLMs to VGLMs causes many mathematical challenges that were absent from the earlier work, mostly related to the increased dimensional complexity of all order parameters in the theory, but that the new problem is once more solvable using the replica method. This opens up the field to many novel and welcome applications, especially in post-genome medical inference and data analysis, where overfitting is and remains a major barrier to progress. After developing the general VGLM overfitting theory, within the replica symmetric (RS) ansatz, we illustrate its merits and its correctness via application to several relevant VGLM regression models.
ER -