Trust the Direction, Search the Step: Zero-and-First-Order Methods for LLM Fine-Tuning
What it is
This paper proposes Zero-and-First-Order (ZFO) optimization, a method that decouples direction selection from step-size determination for LLM fine-tuning. It uses a standard first-order optimizer to define the search direction and performs zeroth-order evaluations along that 1D subspace to select an adaptive, curvature-aware step size.
The problem it solves
Conservative fixed step sizes slow down convergence, while aggressive step sizes can destabilize training; existing adaptive methods often require expensive full line searches or lack theoretical guarantees for finite-difference curvature estimates.
What you could use it for
- Fine-tuning large language models where fixed learning rates cause instability or slow convergence
- Optimizing neural networks where full line search is too computationally expensive
- Improving final model performance by adapting step size based on local objective function curvature
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.