Optimization by Reparametrization: Time-Warped Mirror Flows with Singular Geometry
Abstract
Solving data-driven problems requires defining complex models and fitting them to data, neural networks being a motivating example. The fitting procedure can be seen as an optimization problem, which is often non-convex, and hence optimization guarantees are hard to derive. An opportunity is provided by viewing the model of interest as a redundant reparameterization--an overparameterization--of some simpler model for which optimization results are easier to achieve. In this paper, after formalizing the above idea, we revisit some recent results and derive new ones. In particular, we consider the gradient flow of some classes of linear overparameterizations and show that they correspond to suitable mirror flows on the original parameters. Our main contribution relates to the study of the latter, for which we establish well-posedness and convergence. Several specific instances are provided, including fully connected linear networks and networks with weight normalization. The results yield insight into the role of overparameterization in optimization and corresponding implicit regularization properties.
Related
- No-Regret Bayesian Optimization with Finite-Library Input-Warped Kernels
- Boundary transmission and excursion geometry in the Lorentz mirror model: a numerical study
- Convex optimization on moment polytopes: Hadamard mirror descent and efficient algorithms for quantum functionals and other tensor parameters
- Multiplicative Gradient Method for Log-Homogeneous Optimization: A Unified Mirror-Descent Framework
- The Geometry of Manifold Learning: Singular foliations via Deep Learning models
BibTeX
@misc{vega2026optimization,
title = {{Optimization by Reparametrization: Time-Warped Mirror Flows with Singular Geometry}},
author = {Cristian Vega and Cesare Molinari and Lorenzo Rosasco and Silvia Villa},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.07040},
eprint = {2610.07040},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.07040},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - Optimization by Reparametrization: Time-Warped Mirror Flows with Singular Geometry
AU - Cristian Vega
AU - Cesare Molinari
AU - Lorenzo Rosasco
AU - Silvia Villa
PY - 2026
DA - 2026/10/05
PB - arXiv
DO - 10.48550/arxiv.2610.07040
UR - https://arxiv.org/abs/2610.07040
AB - Solving data-driven problems requires defining complex models and fitting them to data, neural networks being a motivating example. The fitting procedure can be seen as an optimization problem, which is often non-convex, and hence optimization guarantees are hard to derive. An opportunity is provided by viewing the model of interest as a redundant reparameterization--an overparameterization--of some simpler model for which optimization results are easier to achieve. In this paper, after formalizing the above idea, we revisit some recent results and derive new ones. In particular, we consider the gradient flow of some classes of linear overparameterizations and show that they correspond to suitable mirror flows on the original parameters. Our main contribution relates to the study of the latter, for which we establish well-posedness and convergence. Several specific instances are provided, including fully connected linear networks and networks with weight normalization. The results yield insight into the role of overparameterization in optimization and corresponding implicit regularization properties.
ER -