Decentralized SGD under Heavy-Tailed Noise: Optimal Convergence Rates and the Role of Gradient Clipping
What it is
This paper analyzes the convergence properties of decentralized stochastic gradient descent (DSGD) when using gradient clipping under heavy-tailed noise. It demonstrates that clipped DSGD achieves order-optimal convergence rates for smooth non-convex functions and establishes a linear speed-up with the number of agents, distinguishing it from normalization which can fail to converge in decentralized settings.
The problem it solves
Existing decentralized optimization methods using gradient clipping yield suboptimal convergence rates, while normalization methods require additional mechanisms (like local momentum) to ensure convergence under heavy-tailed noise.
What you could use it for
- Designing robust decentralized optimization algorithms for machine learning models trained on distributed nodes with noisy data
- Selecting between gradient clipping and normalization for decentralized training when heavy-tailed gradients are expected
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.