TACO: Ternary Absolute-max Column-wise One-sparse Optimizer for LLM Fine-Tuning
What it is
TACO is a new optimizer for fine-tuning large language models that reduces memory usage by computing the exact steepest-descent direction under a dimension-normalized 1-to-1 operator norm. It achieves this by selecting the sign of the largest magnitude entry in each column of weight matrices, which allows for nearly negligible optimizer state memory while retaining first-order gradients.
The problem it solves
High optimizer state memory overhead in full-parameter fine-tuning of LLMs, which limits the model sizes that can fit on modern GPUs.
What you could use it for
- Full-parameter fine-tuning of 30-32B parameter LLMs on a single 80GB GPU
- Reducing peak training memory for large model fine-tuning tasks
- Fine-tuning AdamW-pretrained models without the performance degradation associated with Muon
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.