Adam is the most widely used optimization algorithm in deep learning, the engine that adjusts a neural network’s millions or billions of internal settings during training so that the model gradually gets better at its task. Introduced in a 2015 paper by Diederik Kingma and Jimmy Ba, Adam combines two older ideas for adapting learning rates into one package that works surprisingly well across a huge range of problems with almost no manual tuning.1arXiv. Adam: A Method for Stochastic Optimization Its dominance is not without controversy, though: researchers have identified real convergence flaws, generalization weaknesses, and memory costs that have spawned an entire ecosystem of Adam variants and alternatives.
What Adam Actually Does
Training a neural network means repeatedly nudging its parameters in a direction that reduces errors. The simplest way to do this is plain stochastic gradient descent (SGD), which computes an estimate of the error slope and takes a fixed-size step downhill. The problem is that a single step size rarely works well for every parameter at once. Some parameters need large adjustments; others need tiny, careful ones. And the gradient estimates from small batches of data are noisy, so any single step might point in a misleading direction.
Adam tackles both problems. It keeps a running average of recent gradients (the direction of the slope) and a running average of recent squared gradients (roughly, how wildly the slope has been fluctuating). The first average acts like momentum, smoothing out the noise so training doesn’t zigzag. The second average lets Adam give each parameter its own effective step size: parameters with consistently large, volatile gradients get smaller steps, while parameters with small, stable gradients get bigger ones. This per-parameter adaptation is what makes Adam “adaptive” and why it converges faster than vanilla SGD on many tasks.
There’s one more wrinkle. Both running averages start at zero, which means they’re biased toward being too small at the beginning of training. Adam includes a bias-correction step that inflates the averages to compensate. This correction matters most during the first few hundred steps and fades away as training continues.
Why Adam Took Over
Adam arrived at the right moment. Deep learning was scaling up rapidly around 2014-2015, and practitioners needed an optimizer that didn’t require hours of hyperparameter tuning for every new architecture. Adam’s default settings (a learning rate of 0.001, and two decay rates of 0.9 and 0.999) worked reasonably well on image classifiers, language models, and generative networks alike. Earlier adaptive methods like Adagrad and RMSProp each solved part of the puzzle, but Adam bundled momentum and per-parameter scaling together in a way that felt complete. The original paper demonstrated that it “compares favorably to other stochastic optimization methods” in practice, and the community’s experience confirmed that.2arXiv. Adam: A Method for Stochastic Optimization
The practical upshot was that you could drop Adam into almost any training loop, leave the defaults alone, and get decent results. SGD, by contrast, often required careful tuning of the learning rate and a hand-crafted schedule for lowering it over time. For researchers trying a new idea every day, Adam’s “good enough out of the box” behavior was enormously valuable.
The Convergence Problem
For several years, Adam’s theoretical foundation went unquestioned. Then in 2018, Sashank Reddi and colleagues demonstrated something uncomfortable: there exist straightforward optimization problems where Adam simply fails to converge to the correct solution.3arXiv. On the Convergence of Adam and Beyond The issue lay in how Adam decays its memory of past squared gradients. In certain patterns where large and small gradients alternate, Adam’s second-moment estimate can drop in a way that inflates the effective step size at exactly the wrong time, causing the optimizer to oscillate instead of settling down.
Reddi et al. proposed AMSGrad as a fix, which prevents the second-moment estimate from ever decreasing. This resolved the theoretical convergence failure, but the practical benefits were less dramatic: on real-world tasks, AMSGrad and standard Adam often performed about the same.4IEEE Access. On the Convergence Proof of AMSGrad and a New Version The episode illustrated a recurring theme in optimizer research. The worst-case scenarios that break an algorithm on paper may almost never arise in the high-dimensional landscapes of real neural networks. Still, the discovery was important because it showed that Adam’s original convergence proof had a genuine gap, not just a conservative bound.
Adam vs. SGD and the Generalization Gap
Faster training doesn’t always mean a better final model. Researchers noticed that on certain tasks, especially image classification with convolutional networks, models trained with Adam converged quickly during training but performed worse on new, unseen data compared to models trained with plain SGD. Adaptive methods “tend to perform well in the initial portion of training but are outperformed by SGD at later stages,” as one study put it.5arXiv. Improving Generalization Performance by Switching from Adam to SGD
The leading explanation has to do with what kind of solutions each optimizer finds. Picture the error surface as a mountainous landscape. SGD, with its noisy, undamped steps, tends to bounce out of sharp, narrow valleys and settle in broader, flatter regions. Adam’s per-parameter scaling and momentum smoothing actually tame that noise, which makes it more likely to stay trapped in a sharp minimum. Flat minima tend to generalize better because small shifts in the data don’t change the error much, whereas sharp minima are brittle.6arXiv. Towards Theoretically Understanding Why SGD Generalizes Better Than ADAM in Deep Learning
This generalization gap is real, but it’s not universal. On natural language processing tasks and especially on Transformers, Adam consistently outperforms SGD. One reason is “block heterogeneity,” where different groups of parameters within a Transformer (attention layers, feed-forward layers, embedding layers) have very different gradient characteristics. Adam handles this naturally because it adapts to each parameter independently, while SGD applies the same step size everywhere and struggles when parameter groups need different treatment.7NeurIPS Proceedings. Understanding Adam vs SGD in Transformers On architectures without that heterogeneity, the two optimizers are much closer in performance.
The practical takeaway that emerged from this debate: use Adam (or its variant AdamW) for Transformers and language models; consider SGD with momentum and a tuned learning rate schedule for convolutional networks when you want the best possible test accuracy and have time to tune.
AdamW and the Weight Decay Fix
Weight decay is a standard technique for preventing overfitting, gently pushing all parameters toward zero so the model doesn’t memorize noise in the training data. For plain SGD, weight decay and a closely related technique called L2 regularization produce identical results. But for Adam, they do not. Loshchilov and Hutter showed in 2019 that most implementations of Adam were applying L2 regularization while calling it “weight decay,” and this mismatch was quietly degrading performance.8arXiv. Decoupled Weight Decay Regularization
The issue is that Adam’s per-parameter scaling interacts with L2 regularization in unexpected ways. Parameters that already have large adaptive scaling effectively get less regularization, which defeats the purpose. Loshchilov and Hutter’s fix, called AdamW, decouples the weight decay from the gradient-based update so that every parameter gets the same proportional decay regardless of its adaptive scaling. The modification is tiny in terms of code but meaningful in results, and AdamW has become the default for training large language models and vision Transformers. When people say they’re using “Adam” for training a modern model, they usually mean AdamW.
The Warmup Question
Early in training, Adam can behave erratically. The bias-correction mechanism partially addresses this, but there’s a deeper issue: with very few gradient samples, Adam’s estimate of per-parameter scaling has high variance. A few atypical gradients in the first batch or two can throw off the adaptive learning rates, causing unstable or excessively large updates. This is why many practitioners add a “warmup” phase where the learning rate starts near zero and ramps up over the first few hundred or thousand steps.
The RAdam variant proposed a principled solution by automatically adjusting the learning rate to compensate for this early variance, removing the need to hand-tune a warmup schedule.9arXiv. On the Variance of the Adaptive Learning Rate and Beyond However, a follow-up study challenged RAdam’s specific analysis, arguing that the instability comes from the magnitude of the update step rather than variance per se, and that a simple linear warmup already solves the problem adequately without needing a specialized algorithm.10Proceedings of the AAAI Conference on Artificial Intelligence. On the Adequacy of Untuned Warmup for Adaptive Optimization In practice, both approaches work, and most large-scale training runs use linear warmup with AdamW regardless.
A related subtlety affects the very first training steps. With Adam’s default bias correction, the effective update can actually be larger than the requested learning rate during the initial iterations because the denominator (the second-moment estimate) hasn’t accumulated enough information yet.11arXiv. AdamD: Improved bias-correction in Adam This is another reason warmup helps: it artificially limits the learning rate during the period when Adam’s internal estimates are least reliable.
Memory Costs at Scale
Adam stores two extra values for every trainable parameter: the running average of gradients (first moment) and the running average of squared gradients (second moment). This means Adam’s optimizer state takes up roughly twice as much memory as the model parameters themselves. For a model with a few million parameters, that’s trivial. For a model with tens of billions of parameters, it becomes a serious engineering constraint. Relative to basic SGD, Adam doubles the memory overhead, which has spurred significant research into lighter alternatives.12arXiv. MicroAdam: Accurate Adaptive Optimization with Low Space Overhead and Provable Convergence – Section: 3.2 Memory footprint analysis and comparison with other methods
Several approaches have emerged to cut this cost:
- Adafactor: Instead of storing a full second-moment matrix for weight matrices, it stores only per-row and per-column sums and reconstructs the per-parameter estimates from those. This produces similar results to Adam with far less memory.13arXiv. Adafactor: Adaptive Learning Rates with Sublinear Memory Cost
- 8-bit Adam: Compresses the optimizer states from 32-bit to 8-bit numbers using block-wise quantization, maintaining the performance of full-precision Adam at a fraction of the memory.14arXiv. 8-bit Optimizers via Block-wise Quantization
- MicroAdam: Compresses the gradient information before it enters the optimizer state, reducing the footprint further while preserving convergence guarantees.15NeurIPS Proceedings. MicroAdam
These memory-efficient variants matter most when training models that push up against GPU memory limits, which is essentially every frontier language model being trained today. The choice between them involves tradeoffs: Adafactor changes the optimization dynamics slightly, 8-bit Adam is a near-transparent drop-in, and MicroAdam takes a more aggressive compression approach. All three reflect the reality that Adam’s memory appetite is its biggest practical weakness.
Numerical Precision and Hardware
Modern training often uses 16-bit floating-point arithmetic (half precision) to speed up computation and reduce memory. Adam’s internal bookkeeping interacts poorly with this. The second-moment estimates involve squaring gradient values and then taking square roots, operations that can produce numbers too small or too large for 16-bit representation. This leads to numerical instability that can derail training entirely.16arXiv. Stabilizing Backpropagation in 16-bit Neural Training with Modified Adam Optimizer
The standard workaround is “mixed precision” training: the model’s forward and backward passes use 16-bit math for speed, but the optimizer states and the parameter update step stay in 32-bit. This adds some memory and a conversion step, but it prevents the catastrophic rounding errors that 16-bit Adam can produce. Most deep learning frameworks handle this automatically, though getting it right for very large distributed training runs still requires care.
Newer Optimizers Challenging Adam
The most interesting recent challenger is Lion (Evolved Sign Momentum), which was discovered through automated program search rather than human mathematical intuition. Lion only tracks momentum, not the second-moment estimates that Adam maintains, cutting memory use significantly. Its update rule uses the sign of the momentum, meaning every parameter gets an update of the same absolute size, scaled only by the learning rate and weight decay.17NeurIPS Proceedings. Discovering Optimization Algorithms via Program Search
On image classification, Lion improved accuracy on vision Transformers by up to 2 percentage points on ImageNet and saved up to five times the compute for pre-training on the larger JFT dataset. On diffusion models for image generation, it achieved better quality scores while cutting training compute by more than half. For language modeling and fine-tuning, it performed on par or better than Adam.18NeurIPS Proceedings. Discovering Optimization Algorithms via Program Search Lion also consistently achieves better GPU utilization efficiency than AdamW, likely because its simpler update rule translates to less computational overhead per step.19arXiv. Comparative Analysis of Lion and AdamW Optimizers for Cross-Encoder Reranking with MiniLM, GTE, and ModernBERT – Section: V-A GPU Efficiency Analysis
Another direction involves incorporating curvature information. AdaHessian uses approximate second-derivative information (the Hessian) to guide updates, which captures more about the shape of the loss landscape than the first-derivative information Adam relies on. It achieved strong results across NLP, computer vision, and recommendation tasks.20Proceedings of the AAAI Conference on Artificial Intelligence. ADAHESSIAN: An Adaptive Second Order Optimizer for Machine Learning The tradeoff is computational cost: estimating curvature is more expensive per step, which limits its appeal for the very largest models where each training step already takes seconds.
Choosing an Optimizer in Practice
If you’re training a Transformer-based model (language, vision-language, or a modern vision Transformer), AdamW with linear warmup is the default for good reason. It handles the heterogeneous parameter groups in Transformers well, and the decoupled weight decay keeps regularization behaving as expected. The standard recipe is a learning rate somewhere between 1e-4 and 3e-4, warmup for the first 1-5% of training steps, and then a cosine or linear decay schedule.
For convolutional networks where you have the budget to tune, SGD with momentum and a step or cosine learning rate schedule can still squeeze out better generalization. The gap has narrowed as the field has moved toward Transformers, but it remains relevant for practitioners working with ResNets and similar architectures.
Memory pressure changes the calculus. If you’re fine-tuning a model that barely fits on your GPU, switching from Adam to 8-bit Adam or Adafactor can free enough memory to avoid offloading or gradient checkpointing. Lion is worth trying if you want both memory savings and potentially faster convergence, though it’s less battle-tested than Adam in production pipelines and can be more sensitive to learning rate choice.
For research experiments where you’re iterating quickly and don’t want to spend time tuning, Adam’s original defaults remain remarkably robust. The optimizer’s greatest strength has always been that it just works, and a decade of scrutiny, patches, and competition hasn’t changed that fundamental appeal.
Why Adam’s Flaws Haven’t Killed It
The convergence failure, the generalization gap, the memory bloat, the numerical precision headaches: any one of these issues would seem damning enough to dethrone a default algorithm. Yet Adam and its close relatives remain the standard choice across most of deep learning. The reason is that each flaw has a targeted fix that patches the problem while preserving what makes Adam useful. Weight decay misbehaves? Use AdamW. Early instability? Add warmup. Memory too high? Use 8-bit states. Convergence theory broken? The pathological cases almost never arise in practice.
The research community has essentially treated Adam like a house that needs renovation rather than demolition. Each paper identifies a specific structural problem, proposes a repair, and the patched version carries forward. The result is that “Adam” in 2025 usually means something like “AdamW with linear warmup, mixed-precision optimizer states, and possibly a cosine learning rate schedule,” which is quite far from the algorithm described in the 2015 paper. But the core idea of adaptive per-parameter learning rates driven by running moment estimates has proven remarkably durable, and no single competitor has yet offered a compelling enough package to displace it across all the domains where it’s used.