How the RMSprop Optimizer Works in Deep Learning

RMSProp is an adaptive learning rate optimizer designed for training neural networks, and it remains one of the most influential algorithms in the field despite never being formally published in a peer-reviewed paper. Introduced by Geoffrey Hinton during a Coursera lecture series in 2012, it solved a specific and frustrating problem with an earlier optimizer called AdaGrad: the learning rate kept shrinking until the network essentially stopped learning. RMSProp’s fix was elegantly simple, and understanding it sheds light on why modern optimizers like Adam work the way they do.

The Problem That Prompted RMSProp

To understand why RMSProp exists, you need to know what went wrong with its predecessor. AdaGrad was a breakthrough because it gave each parameter in a neural network its own learning rate, adapting based on how large the gradients had been for that parameter in the past. It did this by accumulating the sum of all past squared gradients. Parameters that had seen large gradients would get smaller learning rates; parameters that had seen small gradients would get larger ones. This was especially useful when dealing with sparse data, where some features show up rarely and need bigger updates when they do appear.¹

The catch was that the accumulated sum of squared gradients only ever grew. It never forgot. Over many training steps, that accumulated value became so large that the effective learning rate shrank toward zero. At that point, the network was barely updating its weights at all, and training stalled. For tasks that required many passes through the data, this was a serious limitation. AdaGrad worked well in certain settings, but for deep networks trained over many epochs, it often ran out of steam long before the model had finished learning.

How RMSProp Fixes the Shrinking Learning Rate

RMSProp’s key innovation is replacing AdaGrad’s ever-growing sum with a moving average of recent squared gradients. Instead of remembering every gradient the network has ever seen, RMSProp uses an exponential decay that gradually forgets older values. This means the denominator used to scale the learning rate reflects recent gradient behavior rather than the entire training history.²

At each training step, RMSProp computes the gradient for each parameter, squares it, and blends it into a running average. The blending is controlled by a decay rate, commonly denoted as beta (β). A beta value close to 1 (the default is typically 0.9 or 0.99) means older gradient information fades slowly, producing a smoother and more stable average. A smaller beta puts more emphasis on the most recent gradient, making the optimizer more reactive to sudden changes in the loss landscape.

The parameter update itself divides the current gradient by the square root of this running average (hence “root mean square propagation”). When gradients have been consistently large for a particular parameter, its running average is large, and the effective step size shrinks. When gradients have been small, the running average is small, and the optimizer takes a relatively bigger step. This per-parameter adaptivity is what makes RMSProp navigate complex loss surfaces more effectively than a single global learning rate could.

Choosing the Right Hyperparameters

RMSProp has a small number of settings you need to choose, and in practice most people adjust only one or two of them. The most important are the base learning rate, the decay rate beta, and a tiny constant added to prevent division by zero (usually called epsilon).

Learning rate: This is the global step size that gets scaled by the adaptive mechanism. Common starting values are 0.001 or 0.0001. Unlike vanilla gradient descent, the adaptive scaling means RMSProp is somewhat forgiving of a learning rate that is slightly too large, but picking a wildly wrong value still causes problems.
Decay rate (β): Controls how quickly the running average forgets old gradients. The default of 0.9 works well for many tasks. Setting it higher (say 0.99) produces a longer memory and smoother behavior, which can help on noisy loss surfaces. Setting it lower makes the optimizer respond more aggressively to recent gradient changes.
Epsilon (ε): A small constant, typically around 1e-8, added to the denominator to avoid dividing by zero. You rarely need to change this, though bumping it up slightly can sometimes stabilize training in mixed-precision setups where numerical underflow is a risk.

One thing worth noting is that beta is not just a knob you turn for vague “smoothness.” It fundamentally changes how the optimizer weighs recent versus past gradient information. A beta of 0.9 means roughly the last 10 steps dominate the running average. A beta of 0.99 stretches that window to roughly the last 100 steps. For problems where the gradient landscape shifts quickly during training, a shorter window can keep the optimizer responsive. For problems where gradients are inherently noisy and you want to average out the noise, a longer window is safer.

The Convergence Question

For years, RMSProp was widely used in practice but poorly understood in theory. Practitioners trusted it because it worked, but the mathematical guarantees lagging behind the empirical results was a source of discomfort in the optimization community. Research has since caught up to some extent, with proofs that RMSProp can reach critical points (places where the gradient is zero or nearly zero) for smooth objectives, along with bounds on how quickly it gets there.³

That said, the story is not entirely clean. Research has shown that in certain simple convex settings, the standard version of RMSProp with a fixed beta can fail to converge to the optimal solution. This was a surprising result because convex problems are supposed to be the easy case. The issue stems from how the fixed decay rate interacts with the geometry of the problem in specific edge cases. One proposed fix is to make beta change over time rather than holding it constant. When beta is treated as a time-varying sequence, RMSProp can be shown to converge even for non-convex objectives.⁴

In practice, the non-convergence issue with fixed beta rarely causes problems for deep learning practitioners because the loss landscapes of neural networks are far from the pathological cases constructed in theoretical proofs. But if you are working on a problem and notice training oscillating without settling, the theoretical literature suggests experimenting with a decaying beta schedule rather than just lowering the learning rate.

How RMSProp Relates to Adam

Adam, which appeared in 2014, is often described as a combination of RMSProp and momentum. That description is essentially accurate. Adam maintains two running averages: one of the gradients themselves (first moment, similar to classical momentum) and one of the squared gradients (second moment, which is the RMSProp component). It then uses both to compute the parameter update, giving it the adaptive per-parameter scaling of RMSProp plus a momentum-like tendency to keep moving in directions that have been consistently beneficial.

Adam also includes a bias correction step that accounts for the fact that the running averages are initialized at zero and therefore underestimate the true moments early in training. RMSProp does not have this correction by default, which means RMSProp’s early updates can be slightly biased. In practice, this difference matters most during the first few hundred steps of training and becomes negligible afterward.

Whether Adam actually outperforms RMSProp is surprisingly context-dependent. Research that initially sparked concern about adaptive optimizers found that Adam and RMSProp sometimes generalize worse than well-tuned stochastic gradient descent (SGD) with momentum, particularly on image classification tasks.⁵ The gap has narrowed with better hyperparameter tuning and variants like AdamW (which decouples weight decay from the adaptive gradient update), but the finding that adaptive optimizers are not universally superior to SGD remains relevant. The optimizer that works best depends heavily on the architecture, the dataset, and how much time you are willing to spend tuning.

RMSProp’s Strengths With Recurrent Networks

One area where RMSProp has historically performed well is in training recurrent neural networks (RNNs), including LSTMs and GRUs. Recurrent networks process sequences step by step, and the gradients that flow backward through time can vary wildly in magnitude. Some time steps produce large gradients while others produce near-zero gradients, and this high variance makes fixed learning rates treacherous. RMSProp’s per-parameter adaptivity helps tame these swings, allowing the network to keep learning even when the gradient signal is uneven across different parts of the sequence.

Comparative studies evaluating RNNs, LSTMs, and GRUs under different optimization regimes, including RMSProp, Adam, SGD, and Nadam, have found that no single optimizer dominates across all architectures and datasets.⁶ RMSProp tends to be competitive with Adam on recurrent architectures, and in some configurations it trains faster or more stably. This is partly because the momentum component in Adam can amplify oscillations in the noisy gradient environments that recurrent networks produce, while RMSProp’s simpler adaptive scaling avoids that risk.

In reinforcement learning, RMSProp has had an even longer run as a default choice. Many foundational deep reinforcement learning papers, including the original Deep Q-Network work from DeepMind, used RMSProp. Reinforcement learning objectives are inherently non-stationary (the target the network is chasing keeps changing as the policy improves), and RMSProp’s short memory for gradient magnitudes meshes well with that shifting landscape. Adam works too, but the reinforcement learning community’s early adoption of RMSProp created a large body of tuned hyperparameters and established baselines built around it.

Where RMSProp Sits in the Modern Landscape

Adam and its variants, particularly AdamW, have become the default optimizer for most new deep learning projects, especially in natural language processing and large-scale pretraining. But RMSProp has not disappeared. It continues to be evaluated alongside Adam and SGD in newer architectures. Recent work examining how optimizer choice affects Vision Transformers for biometric tasks, for instance, still includes RMSProp as one of the primary candidates, because its behavior under different learning rate schedules can differ meaningfully from Adam’s.⁷

There is also ongoing theoretical work aimed at understanding adaptive optimizers more deeply. One line of research models RMSProp, AdaGrad, and Adam as continuous-time processes using integro-differential equations. The goal is to move beyond the discrete, step-by-step view of these algorithms and instead analyze them as continuous flows. Early results suggest strong agreement between these continuous models and the actual discrete implementations, which could eventually lead to better theoretical guarantees and more principled hyperparameter selection.⁸

For practitioners deciding which optimizer to use today, the honest answer is that RMSProp is a solid choice when you want adaptive learning rates without the added complexity of momentum, or when you are working in a domain (like reinforcement learning) where it has a long track record. If you are starting a new supervised learning project from scratch, Adam or AdamW is the safer default because the ecosystem of learning rate schedules, warmup strategies, and weight decay recipes has been more thoroughly explored for those optimizers. But “safer default” is not the same as “always better.” If Adam is giving you trouble on a particular problem, switching to RMSProp is one of the first things worth trying.

Continuous-Time Models and What They Reveal

One of the more interesting recent developments in understanding RMSProp is the push to describe it not as a sequence of discrete update steps but as a continuous process that evolves over time. The motivation is straightforward: discrete-step analysis can tell you what happens at each iteration, but continuous models let you use the well-developed tools of calculus and differential equations to reason about global behavior, stability, and long-term dynamics.

Researchers have proposed modeling RMSProp’s update rule as a first-order integro-differential equation, capturing both the gradient’s instantaneous value and the accumulated history stored in the running average. The resulting continuous-time model tracks the discrete algorithm closely, which means insights from the continuous version (such as conditions under which the trajectory is guaranteed to converge) can inform how you set hyperparameters in the actual discrete implementation.⁹

This kind of theoretical work may seem far removed from the day-to-day business of training models, but it feeds into practical tools over time. Better theoretical understanding of why a particular beta value or learning rate works in certain regimes, and fails in others, eventually shows up in better default settings, smarter learning rate schedulers, and optimization libraries that require less manual tuning. The fact that RMSProp, an algorithm introduced informally in a set of lecture slides over a decade ago, is still generating new theoretical insights speaks to how much remains unknown about even the most basic tools in deep learning.

Common Misconceptions About RMSProp

A few misunderstandings circulate about RMSProp that are worth clearing up. The first is that RMSProp is “just a worse version of Adam.” Since Adam incorporates RMSProp’s second-moment estimate and adds momentum on top, it might seem strictly superior. But the momentum component is not free. It introduces another hyperparameter, interacts with learning rate warmup in ways that can be surprising, and in some noisy optimization landscapes it actually hurts by amplifying oscillations. RMSProp’s relative simplicity is sometimes an advantage, not a limitation.

The second misconception is that RMSProp’s default hyperparameters are universally good. The common default of beta equal to 0.9 and learning rate equal to 0.001 is a reasonable starting point, but these values were established in an era when networks and datasets were much smaller than they are today. For modern architectures with millions or billions of parameters, you may need to adjust both. In particular, the learning rate often needs to be lower, and beta may benefit from being higher to smooth out the noisier gradient estimates that come from large-batch or distributed training.

A third misunderstanding is that adaptive optimizers like RMSProp eliminate the need for learning rate schedules. They do not. RMSProp adapts the relative scale of updates across different parameters, but the overall learning rate still controls the absolute magnitude. Using a learning rate schedule on top of RMSProp, such as reducing the learning rate partway through training or using a cosine annealing curve, almost always improves final performance. The adaptive mechanism and the schedule work on different aspects of the optimization and complement each other rather than being redundant.

Finally, there is the widespread belief that because RMSProp was never published in a formal paper, it is somehow less rigorous or trustworthy than alternatives. Its origins in a lecture series are unusual, but the algorithm itself has been analyzed extensively by the research community in the years since. The lack of a founding paper means there is no single canonical reference to cite, which is mildly inconvenient for academic writers, but it says nothing about the algorithm’s effectiveness or theoretical soundness. If anything, RMSProp’s widespread adoption despite having no marketing push from a prestigious publication is a testament to how well it works in practice.