DGA-Muon: Decoupled Geometry-Aligned Adaptive Scaling for Muon
What it is
This paper analyzes the adaptive scaling mechanism of the NorMuon optimizer, revealing that its adaptivity is largely an artifact of orthogonalization geometry rather than optimization signal. It proposes DGA-Muon, which decouples adaptive scaling from orthogonalization and aligns scaling dimensions with the matrix shape (row-wise for wide, column-wise for tall) to improve optimization performance.
The problem it solves
NorMuon's adaptive scaling is poorly understood and geometrically misaligned, leading to a paradox where more accurate orthogonalization weakens adaptivity.
What you could use it for
- Large-scale language model pretraining
- Improving the convergence of Muon-based optimizers
- Resolving the Orthogonalization-Adaptivity Paradox in adaptive optimizers
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.