How to Train Your Intercept: A Comparison of Intercept Update Strategies in Coordinate Descent
Abstract
Coordinate descent solvers for regularized generalized linear models (GLMs) differ in how they update the intercept. This seemingly routine choice can substantially affect convergence when the response is imbalanced. For direct coordinate descent on the original GLM loss, we compare a conservative step based on worst-case curvature, a safeguarded Newton step based on current curvature, and repeated Newton steps that solve the intercept subproblem. The conservative update can leave much of the interaction between the intercept and the coefficients unresolved, producing a slowdown when that interaction controls convergence. A single Newton step avoids this attenuation and, in our experiments, follows nearly the same outer trajectory as exact minimization while requiring fewer inner steps. Simulated and real-data experiments, including comparisons across six production solvers and a controlled intervention within skglm, support this explanation. We also show how the same mechanism appears in quadratic-approximation methods and extends to multinomial models with rare classes. We recommend a safeguarded Newton update for direct coordinate descent and full intercept optimization within frozen local quadratic surrogates.
Related
- Randomized average block coordinate descent method with heavy-ball momentum for tensor least squares problem under the t-product
- Globally Convergent Block Coordinate Descent Algorithms for Semi-Nonnegative Independent Component Analysis
- On randomized batch-sampling coordinate descent methods for solving linear least-squares problems
- Boundary Geometry and Global Rates of Cyclic Coordinate Descent for SVM Duals
- A Unified Coordinate Descent Framework for Penalized Longitudinal Log-Contrast Quantile Regression
BibTeX
@misc{larsson2026train,
title = {{How to Train Your Intercept: A Comparison of Intercept Update Strategies in Coordinate Descent}},
author = {Johan Larsson and Frederik Fabricius-Bjerre},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.18043},
eprint = {2610.18043},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.18043},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - How to Train Your Intercept: A Comparison of Intercept Update Strategies in Coordinate Descent
AU - Johan Larsson
AU - Frederik Fabricius-Bjerre
PY - 2026
DA - 2026/10/13
PB - arXiv
DO - 10.48550/arxiv.2610.18043
UR - https://arxiv.org/abs/2610.18043
AB - Coordinate descent solvers for regularized generalized linear models (GLMs) differ in how they update the intercept. This seemingly routine choice can substantially affect convergence when the response is imbalanced. For direct coordinate descent on the original GLM loss, we compare a conservative step based on worst-case curvature, a safeguarded Newton step based on current curvature, and repeated Newton steps that solve the intercept subproblem. The conservative update can leave much of the interaction between the intercept and the coefficients unresolved, producing a slowdown when that interaction controls convergence. A single Newton step avoids this attenuation and, in our experiments, follows nearly the same outer trajectory as exact minimization while requiring fewer inner steps. Simulated and real-data experiments, including comparisons across six production solvers and a controlled intervention within skglm, support this explanation. We also show how the same mechanism appears in quadratic-approximation methods and extends to multinomial models with rare classes. We recommend a safeguarded Newton update for direct coordinate descent and full intercept optimization within frozen local quadratic surrogates.
ER -