It isolates the stochastic direction (the sign of the gradient) from the adaptive step size (the relative variance).
A more recent and highly regarded paper (2025) investigates what happens when Adam "wanders" around the manifold of minimizers. Splitting Adam
It argues that Adam's second moment actually causes word representations to become narrow and directional (anisotropic). It isolates the stochastic direction (the sign of
If you are coming from a statistics or rare-event simulation background, "ADAM" refers to . Splitting Adam
It's often applied to power grid reliability or particle transport. 3. Adam Reduces a Unique Form of Sharpness