TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
Abstract
Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized 1→1 operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by 174× relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by 2.9× (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.
Related
- Decomposition of Sparse Integer Programs via Nonlinear Edge Encodings and Column-and-Row Generation
- Multiplicative vs. Additive State Coupling in Learned Optimizer Schedules
- Nash-Bargaining HalpernSGD via Limited-Memory MultiLRSGA: A Two-Phase Optimizer for Multi-Objective Learning
- Explicit counterexamples to Frank-Loss optimizer conjectures for critical Dirac and vector Sobolev inequalities
- Optimizer as Detector: Stochastic Gradient Descent for Latent Mixture Models
BibTeX
@misc{jiang2026taco,
title = {{TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning}},
author = {Jichao Jiang and Cristian McGee and El Houcine Bergou and HanQin Cai and Aritra Dutta},
howpublished = {arXiv},
year = {2026},
month = oct,
doi = {10.48550/arxiv.2610.02199},
eprint = {2610.02199},
archivePrefix = {arXiv},
url = {https://arxiv.org/abs/2610.02199},
note = {CC BY 4.0},
}
RIS
TY - UNPB
TI - TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
AU - Jichao Jiang
AU - Cristian McGee
AU - El Houcine Bergou
AU - HanQin Cai
AU - Aritra Dutta
PY - 2026
DA - 2026/10/06
PB - arXiv
DO - 10.48550/arxiv.2610.02199
UR - https://arxiv.org/abs/2610.02199
AB - Full-parameter fine-tuning of large language models (LLMs) incurs substantial optimizer state memory overhead, limiting the model sizes that fit on modern GPUs. Existing approaches either compress optimizer state, abandon first-order gradients, or change the update geometry while retaining dense state. The recently introduced Muon optimizer reduces optimizer memory through matrix-valued updates. Still, its geometry differs from AdamW and can lead to performance degradation when fine-tuning AdamW-pretrained models. To reduce optimizer memory without sacrificing accuracy or computational efficiency in LLM fine-tuning, we propose Ternary Absolute-max Column-wise One-sparse optimizer, or TACO, which follows Muon's operator-norm steepest-descent view but takes the geometric route further. TACO computes the exact steepest-descent direction under a dimension-normalized 1→1 operator norm by selecting the sign of the largest magnitude entry in each column of two-dimensional weight matrices. This retains first-order gradients while making optimizer state memory nearly negligible. Our practical TACO optimizer maintains only a small set of low precision gradient components per column, reducing persistent optimizer state by 174× relative to AdamW8bit (from 27.7 GB to 0.16 GB) and peak training memory by 2.9× (from 80.6 GB to 27.5 GB) on OPT-13B, while achieving comparable accuracy and runtime. TACO further enables full-parameter fine-tuning of 30-32B-parameter models on a single 80 GB H100 GPU across multiple model families and tasks.
ER -