EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale
Abstract
EGGROLL (Sarkar et al., 2026) makes evolution strategies (ES) practical for LLMs by replacing dense Gaussian weight perturbations with low-rank Gaussian products, often of rank one. This choice is computationally attractive but geometrically severe: Each rank-one perturbation lies in a zero-volume subset of the ambient matrix space, despite having identity covariance. We characterize the EGGROLL update mean field at finite rank and nonzero perturbation radius as a resolvent applied to the gradient of the perturbation-smoothed objective. This transformation can make the mean field nonconservative and reverse the local stability of an optimum. EGGROLL nevertheless recovers the gradient exactly on quadratic objectives at every rank and radius. In finite populations, the additional sampling variance of rank-one perturbations relative to dense Gaussian ES decays inversely with matrix width under a local affine model, and is only 0.098% at width $4096$. Finally, we introduce LOO-ROLL, a leave-one-out estimator that replaces EGGROLL's two antithetic evaluations per direction by one. At equal evaluation cost, LOO-ROLL halves estimator MSE in transformer blocks. Across fourteen post-training settings up to 14B parameters, matched-time comparisons with EGGROLL yield eleven improvements in individual paired tests and no significant loss. At 1.7B, 8B, and 14B parameters, matched-time gains are $2.9$, $14.1$, and $7.9$ percentage points on GSM8K and $12.2$, $8.4$, and $8.1$ points on MATH-500.
Related
- Improving power without increasing sample size: Model-based and design-based strategies for controlling time-dependent nuisance variation in experiments
- Understanding Private Evolution as Learning-Augmented Clustering
- Jacobian-Free Unrolling: memory efficient unrolled network for 3D SPECT Reconstruction
- Deeply unrolled alternating projections for phase retrieval
- Unrolled gradients in disguise: bridging interpolation-based and Jacobian regularization for stable neural dynamics
BibTeX
@misc{kaya2026eggroll,
title = {{EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale}},
author = {Ege C. Kaya and Abolfazl Hashemi},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.10980},
eprint = {2609.10980},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.10980},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - EGGROLL, Unrolled: Understanding and Improving Low-Rank Evolution Strategies at Scale
AU - Ege C. Kaya
AU - Abolfazl Hashemi
PY - 2026
DA - 2026/09/25
PB - arXiv
DO - 10.48550/arxiv.2609.10980
UR - https://arxiv.org/abs/2609.10980
AB - EGGROLL (Sarkar et al., 2026) makes evolution strategies (ES) practical for LLMs by replacing dense Gaussian weight perturbations with low-rank Gaussian products, often of rank one. This choice is computationally attractive but geometrically severe: Each rank-one perturbation lies in a zero-volume subset of the ambient matrix space, despite having identity covariance. We characterize the EGGROLL update mean field at finite rank and nonzero perturbation radius as a resolvent applied to the gradient of the perturbation-smoothed objective. This transformation can make the mean field nonconservative and reverse the local stability of an optimum. EGGROLL nevertheless recovers the gradient exactly on quadratic objectives at every rank and radius. In finite populations, the additional sampling variance of rank-one perturbations relative to dense Gaussian ES decays inversely with matrix width under a local affine model, and is only 0.098% at width $4096$. Finally, we introduce LOO-ROLL, a leave-one-out estimator that replaces EGGROLL's two antithetic evaluations per direction by one. At equal evaluation cost, LOO-ROLL halves estimator MSE in transformer blocks. Across fourteen post-training settings up to 14B parameters, matched-time comparisons with EGGROLL yield eleven improvements in individual paired tests and no significant loss. At 1.7B, 8B, and 14B parameters, matched-time gains are $2.9$, $14.1$, and $7.9$ percentage points on GSM8K and $12.2$, $8.4$, and $8.1$ points on MATH-500.
ER -