Solving the Ho--Nguyen Conjecture for Gaussian Mixtures: Applications to Mixtures of Experts
Abstract
Over-fitted Gaussian mixtures can contain many parameter configurations with nearly the same density, making the order of density cancellation central to their parameter-learning rates. Ho and Nguyen encoded these cancellations in a polynomial system and conjectured its exact order as a function of cluster size. The key to resolving the conjecture is a common positive-variance shift that turns the coefficient equations into Gaussian moment matching. For every cluster size Mhn≥2, we prove that the real, nonzero-weight system has order 2Mhn: a sharp sign-change bound forces rigidity at that order, while positive Gauss--Hermite quadrature constructs a non-trivial solution through order 2Mhn−1. Population-density calculations at 110-digit working precision recover the predicted leading term for two to six components; finite multistart searches recover the same observed powers for two to five components, with smaller finite-scale constants for three to five components. For parameter learning, the exact order gives an explicit pointwise maximum-likelihood exponent and an exact local minimax order on specified shrinking neighbourhoods of a fixed known mixing measure. The same coefficient viewpoint separates high-order cancellation from exact parameter symmetries in mixtures of experts. Synchronized replication preserves the conditional density while keeping the corresponding parameter losses positive in two broad standard-softmax classes, showing why statistical rate arguments must account for exact parameter symmetries.
Related
- Dimension-free comparison inequalities between f-divergences for Gaussian location mixtures
- Squarefree values of random polynomials are Gaussian mixtures
- Softmax Attention on Gaussian Mixtures: Linear When It Can, Selective When It Must
- Adversarially Trained Linear Transformers Are Optimal Robust In-Context Learners for Gaussian Mixtures
- Moment-Accurate Gaussian Mixtures for Constant-Step Stochastic Approximation
BibTeX
@misc{zhang2026solving,
title = {{Solving the Ho--Nguyen Conjecture for Gaussian Mixtures: Applications to Mixtures of Experts}},
author = {Yuyang Zhang and Qihuang Zhang},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.27265},
eprint = {2610.27265},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.27265},
}
RIS
TY - UNPB
TI - Solving the Ho--Nguyen Conjecture for Gaussian Mixtures: Applications to Mixtures of Experts
AU - Yuyang Zhang
AU - Qihuang Zhang
PY - 2026
DA - 2026/10/19
PB - arXiv
DO - 10.48550/arxiv.2610.27265
UR - https://arxiv.org/abs/2610.27265
AB - Over-fitted Gaussian mixtures can contain many parameter configurations with nearly the same density, making the order of density cancellation central to their parameter-learning rates. Ho and Nguyen encoded these cancellations in a polynomial system and conjectured its exact order as a function of cluster size. The key to resolving the conjecture is a common positive-variance shift that turns the coefficient equations into Gaussian moment matching. For every cluster size Mhn≥2, we prove that the real, nonzero-weight system has order 2Mhn: a sharp sign-change bound forces rigidity at that order, while positive Gauss--Hermite quadrature constructs a non-trivial solution through order 2Mhn−1. Population-density calculations at 110-digit working precision recover the predicted leading term for two to six components; finite multistart searches recover the same observed powers for two to five components, with smaller finite-scale constants for three to five components. For parameter learning, the exact order gives an explicit pointwise maximum-likelihood exponent and an exact local minimax order on specified shrinking neighbourhoods of a fixed known mixing measure. The same coefficient viewpoint separates high-order cancellation from exact parameter symmetries in mixtures of experts. Synchronized replication preserves the conditional density while keeping the corresponding parameter losses positive in two broad standard-softmax classes, showing why statistical rate arguments must account for exact parameter symmetries.
ER -