An Adaptive Optimizer without User-Specified Hyperparameters through Historical Projection and Self-Calibrated Scaling
- 1Graduate School of Information Sciences, Tohoku University
Abstract
Adaptive optimizers reduce sensitivity to coordinate-wise gradient scale, but they usually still require the user to choose an overall learning rate and often a schedule. We introduce YamAdam, an adaptive optimizer that requires no user-specified optimizer hyperparameters. YamAdam combines short-memory Adam moments with both decay factors fixed at 1/2, a projection that removes only the historical component opposing the current mini-batch gradient, a one-time calibration of a reference update radius, and subsequent scaling based on dimensionless directional statistics. We evaluate YamAdam without user-specified optimizer settings on six workloads spanning image classification, image reconstruction, tabular classification and regression, and language modeling. The comparison contains 30 fixed settings from 10 optimizer families, with every setting reported separately. Our primary measure, {Delta}TTT, is the comparator-minus-YamAdam difference in budgetnormalized time to five predefined targets, averaged equally across workloads. All 30 comparator settings produced a positive {Delta}TTT, ranging from +0.003526 to +0.210524. Across 540 setting-workload-seed comparisons, YamAdam was faster in 504, tied in 14, and slower in 22. Across 2,700 target observations, both methods reached 2,634 targets and only YamAdam reached 66; no target was reached only by the comparator or by neither method. These results show that an optimizer requiring no user-specified optimizer hyperparameters can attain predefined targets rapidly across heterogeneous workloads.
Related
- Ark-ADMET: endpoint-specific selection of molecular representations and hyperparameters for gradient-boosted ADMET models under the Therapeutics Data Commons benchmark protocol
- On the sensitivity to hyperparameters of a Bayesian method to infer selection signatures
- Simple but effective unsupervised classification for specified domain images: a case study on fungi images
- Novel hole quality indices and intelligent optimization of abrasive waterjet drilling of Al 7075-T6 using grey wolf optimizer
- The functional neuronal lipotype is specified by activity and PS synthesis using dietary and lysosomal inputs
BibTeX
@misc{yamada2026adaptive,
title = {{An Adaptive Optimizer without User-Specified Hyperparameters through Historical Projection and Self-Calibrated Scaling}},
author = {K. D. Yamada},
howpublished = {bioRxiv},
year = {2026},
month = oct,
doi = {10.1101/348557},
url = {https://www.biorxiv.org/content/10.1101/348557v5},
note = {CC BY-NC-ND},
}
RIS
TY - UNPB
TI - An Adaptive Optimizer without User-Specified Hyperparameters through Historical Projection and Self-Calibrated Scaling
AU - K. D. Yamada
PY - 2026
DA - 2026/10/28
PB - bioRxiv
DO - 10.1101/348557
UR - https://www.biorxiv.org/content/10.1101/348557v5
AB - Adaptive optimizers reduce sensitivity to coordinate-wise gradient scale, but they usually still require the user to choose an overall learning rate and often a schedule. We introduce YamAdam, an adaptive optimizer that requires no user-specified optimizer hyperparameters. YamAdam combines short-memory Adam moments with both decay factors fixed at 1/2, a projection that removes only the historical component opposing the current mini-batch gradient, a one-time calibration of a reference update radius, and subsequent scaling based on dimensionless directional statistics. We evaluate YamAdam without user-specified optimizer settings on six workloads spanning image classification, image reconstruction, tabular classification and regression, and language modeling. The comparison contains 30 fixed settings from 10 optimizer families, with every setting reported separately. Our primary measure, {Delta}TTT, is the comparator-minus-YamAdam difference in budgetnormalized time to five predefined targets, averaged equally across workloads. All 30 comparator settings produced a positive {Delta}TTT, ranging from +0.003526 to +0.210524. Across 540 setting-workload-seed comparisons, YamAdam was faster in 504, tied in 14, and slower in 22. Across 2,700 target observations, both methods reached 2,634 targets and only YamAdam reached 66; no target was reached only by the comparator or by neither method. These results show that an optimizer requiring no user-specified optimizer hyperparameters can attain predefined targets rapidly across heterogeneous workloads.
ER -