Early Memory Selection for Balanced Adam
What it is
This paper proposes a method to automatically select the shared memory parameter (beta) for the Adam optimizer using a short pilot training run. It uses a local model to balance sampling variability against gradient averaging delay, deriving a cubic memory rule estimated from a few gradient probes.
The problem it solves
Manual tuning of Adam's beta parameters is time-consuming and often suboptimal, as fixed values like 0.95 do not account for the specific trade-off between noise and delay in different training contexts.
What you could use it for
- Automating hyperparameter tuning for Adam-based training runs
- Improving validation performance on vision and language workloads without manual grid search
Read the source ↗Back to the library →
Summary written by an AI model from the source; check the source before you rely on it.