Connecting Score Matching, Maximum Likelihood, and Expectation-Maximization in Mixed Linear Regression
Abstract
We study variance-preserving diffusion of the response in mixed linear regression (MLR) with unknown mixing weights. Our analysis separates the statistical guarantees of score matching from the loss geometry and optimization signal at a fixed diffusion noise level. The KL divergence links the denoising score matching objective integrated over the diffusion path with the likelihood and a terminal discrepancy. Under mild regularity conditions and terminal schedule, the resulting estimator converges up to the ground truth parameters of MLR, and its scaled error converges to the Gaussian limit of the maximum-likelihood estimator. At a fixed scale of the diffusion noise level, we derive a decomposition linking the score matching loss to cross-entropy and Expectation-Maximization (EM) operators. This decomposition yields an EM-related low-noise gradient expansion with additional correction terms of latent variance. In the high-noise limit, we further characterize gradient descent on this limiting loss under isotropic covariance. Along fixed high signal-to-noise ratio rays, the score matching imbalance gradient and the latent-variance term tend to zero pointwise. Numerical experiments illustrate our theoretical findings and statistical guarantees.
Related
- Robust Covariance Estimation for Entry-wise Contaminated and Partially Observed Data via Expectation Maximization Algorithm
- Generalized Model Predictive Path Integral Control as Expectation--Maximization
- Demand Estimation with Unobserved Stockout Substitution: Exact Likelihood Maximization
- Online Non-Monotone DR-Submodular Maximization Matching the Offline $0.401$ Factor
- Unrestricted conditional maximum likelihood estimation for a Fréchet regression model
BibTeX
@misc{luo2026connecting,
title = {{Connecting Score Matching, Maximum Likelihood, and Expectation-Maximization in Mixed Linear Regression}},
author = {Zhankun Luo and Abolfazl Hashemi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.05688},
eprint = {2609.05688},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.05688},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Connecting Score Matching, Maximum Likelihood, and Expectation-Maximization in Mixed Linear Regression
AU - Zhankun Luo
AU - Abolfazl Hashemi
PY - 2026
DA - 2026/09/04
PB - arXiv
DO - 10.48550/arxiv.2609.05688
UR - https://arxiv.org/abs/2609.05688
AB - We study variance-preserving diffusion of the response in mixed linear regression (MLR) with unknown mixing weights. Our analysis separates the statistical guarantees of score matching from the loss geometry and optimization signal at a fixed diffusion noise level. The KL divergence links the denoising score matching objective integrated over the diffusion path with the likelihood and a terminal discrepancy. Under mild regularity conditions and terminal schedule, the resulting estimator converges up to the ground truth parameters of MLR, and its scaled error converges to the Gaussian limit of the maximum-likelihood estimator. At a fixed scale of the diffusion noise level, we derive a decomposition linking the score matching loss to cross-entropy and Expectation-Maximization (EM) operators. This decomposition yields an EM-related low-noise gradient expansion with additional correction terms of latent variance. In the high-noise limit, we further characterize gradient descent on this limiting loss under isotropic covariance. Along fixed high signal-to-noise ratio rays, the score matching imbalance gradient and the latent-variance term tend to zero pointwise. Numerical experiments illustrate our theoretical findings and statistical guarantees.
ER -