Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
Abstract
Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework.
Related
- MpSub: A Momentum p-Dimensional Subspace Trust-Region Method for Derivative-Free Fine-Tuning of Large Language Models
- A particle swarm optimization with a reduced-subspace trust-region safeguard for contact-rich controller tuning
- Distributed Grid Search for Hyperparameter Tuning
- Complexity analysis of trust-region methods under (α,L0,L1)-smoothness
- Bayesian Optimization with Fisher Information Geometry: Gradient Bounds and Trust-Region Methods
BibTeX
@misc{mcgee2026trust,
title = {{Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning}},
author = {Cristian McGee and El Houcine Bergou and Aritra Dutta},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02190},
eprint = {2610.02190},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02190},
}
RIS
TY - UNPB
TI - Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
AU - Cristian McGee
AU - El Houcine Bergou
AU - Aritra Dutta
PY - 2026
DA - 2026/10/01
PB - arXiv
DO - 10.48550/arxiv.2610.02190
UR - https://arxiv.org/abs/2610.02190
AB - Step-size selection remains a central challenge in large-scale neural network optimization; conservative steps slow convergence, while aggressive steps can destabilize it. We combine \textbf{Z}ero-and-\textbf{F}irst-\textbf{O}rder optimization~(ZFO) and propose a lightweight framework that decouples direction selection from step-size. ZFO uses a trusted first-order optimizer to determine the direction and performs zeroth-order evaluations only along this one-dimensional subspace to choose how far to move. Using the current {gradient information} and two additional objective function evaluations, ZFO instances construct a local model of the objective function along the proposed direction and select a curvature-aware step within a bounded search interval. This yields an adaptive step-selection mechanism that costs less than a full line search. We provide theoretical guarantees to show that shared-sample evaluations produce reliable finite-difference curvature estimates, that the induced local model selects a near-optimal step along the search interval, and that ZFO converges to a neighborhood of a stationary point. Across the evaluated settings, language models and datasets, ZFO frequently improves optimization and final performance relative to fixed-step first-order baselines, with the magnitude and preferred local model depending on the objective. Our code is publicly available at: https://github.com/nizswan/Zeroth-First-Order-Framework.
ER -