AdamX: Cosine similarity meets gradient descent
Abstract
We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.
Related
- Radial–Angular Feature Shaping with an eXtra-Node and Prototype-Based Cosine Similarity Under Adversarial Perturbations
- Muon meets Tamed Langevin: Momentum Preconditioning beyond Convex and gradient-Lipschitz Potentials
- Broximal Gradient Descent: A Projection-Free Sister of Projected Gradient Descent
- PL conditions do not guarantee convergence of gradient descent-ascent dynamics
- Silver Rate Is (Almost) Optimal for Gradient Descent: The Strongly Convex Case
BibTeX
@misc{caldas2026adamx,
title = {{AdamX: Cosine similarity meets gradient descent}},
author = {Francisco Caldas and Ruben Belo and Cláudia Soares},
howpublished = {arXiv},
year = {2026},
month = sep,
doi = {10.48550/arxiv.2609.11867},
eprint = {2609.11867},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2609.11867},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - AdamX: Cosine similarity meets gradient descent
AU - Francisco Caldas
AU - Ruben Belo
AU - Cláudia Soares
PY - 2026
DA - 2026/09/10
PB - arXiv
DO - 10.48550/arxiv.2609.11867
UR - https://arxiv.org/abs/2609.11867
AB - We introduce AdamX, a first-order optimizer that incorporates cosine similarity as an adaptive mechanism for controlling update magnitudes. The proposed method is scalable, model-agnostic, and straightforward to integrate into existing training pipelines. We further introduce a variance rectification scheme that promotes smoother optimization during the early stages of training. Overall, we provide empirical evidence that AdamX achieves competitive convergence rates across a range of benchmark datasets and architectures. Performance is evaluated in terms of the number of epochs required to reach predefined performance thresholds under a fixed hyperparameter budget. Code and Experiments available at: https://github.com/FranciscoCaldas/adamX.
ER -