What Is a Recurrent Neural Network and How Does It Work?

A recurrent neural network is a type of artificial neural network designed to process sequences by maintaining a form of memory. Unlike standard neural networks that treat every input as independent, an RNN feeds its own output back into itself at each step, allowing information from earlier in a sequence to influence what happens later. That looping mechanism is what makes RNNs naturally suited to data where order matters: speech, text, music, sensor readings over time, and much more.

How the Memory Loop Works

The defining feature of an RNN is its recurrence. At each step in a sequence, the network takes two inputs: the current data point and a “hidden state” that encodes what the network has seen so far. It combines these, produces an output, and passes along an updated hidden state to the next step. You can think of it as reading a book one word at a time while keeping a running mental summary. Each new word adjusts the summary, and the summary influences how you interpret the next word.

This makes RNNs fundamentally different from feedforward networks, which process all their input in one shot with no sense of order. A feedforward network looking at a sentence would see a bag of words. An RNN sees them arrive one after another and can recognize patterns that depend on sequence, like the fact that “not” before “good” reverses the meaning.

The catch is that this loop has to be “unrolled” for training. To teach the network, you effectively lay out copies of it across every time step in the input sequence and calculate how wrong it was at each step. The training signal then flows backward through all those copies, a process called backpropagation through time (BPTT). Several variations of BPTT exist, including truncated versions that only look back a limited number of steps to save computation.¹ The unrolling step is also what ties training cost directly to sequence length, because longer sequences mean more copies of the network stacked end to end.²

The Vanishing Gradient Problem

RNNs are notorious for struggling to learn long-term dependencies, and the primary culprit is what researchers call vanishing and exploding gradients.³ When BPTT sends the error signal backward through many time steps, that signal gets multiplied at each step by the same set of weights. If those weights tend to shrink the signal, it decays exponentially and effectively disappears before reaching the earlier steps, a state called vanishing gradients. If the weights amplify it, the signal grows uncontrollably, called exploding gradients.

In practical terms, vanishing gradients mean a basic RNN can struggle to connect a cause to an effect that are more than a handful of steps apart. If the word that determines the correct ending of a sentence appeared twenty words ago, a vanilla RNN may have lost that information entirely by the time it needs it. Exploding gradients are easier to handle in practice because you can simply clip the values once they exceed a threshold, but vanishing gradients require architectural changes to fix properly.

Gated Variants That Solved the Memory Problem

The most important architectural response to vanishing gradients came in the form of gating mechanisms. Two designs dominate: the Long Short-Term Memory network (LSTM) and the Gated Recurrent Unit (GRU).

An LSTM adds a dedicated “cell state” alongside the hidden state, along with three gates that control how information flows. One gate decides what new information to let in, another decides what old information to forget, and a third decides what to pass on to the output. This design lets important information travel across many time steps with minimal degradation, because the cell state acts as a kind of highway that gradients can flow through without vanishing.

GRUs simplify this by merging the input and forget gates into a single “update gate” and using a “reset gate” to control how much of the previous hidden state feeds into the new candidate state.⁴ The result is a network with fewer parameters that trains faster, while often performing comparably to LSTMs. In practice, GRUs tend to be preferred when computational resources are limited or when the sequences are not extremely long, while LSTMs remain the go-to for tasks requiring fine-grained control over what gets remembered and what gets discarded.

Where RNNs Earn Their Keep

The sequential nature of RNNs makes them a natural fit for several categories of real-world problems. While newer architectures have taken over some of these roles, RNNs and their gated variants remain actively used, especially in settings where their strengths outweigh the alternatives.

Speech and Language

Speech recognition was one of the early killer applications. RNNs trained with connectionist temporal classification (CTC) showed strong results in end-to-end speech recognition and handwritten character recognition, tasks where the alignment between input and output is not known in advance.⁵ Recurrent encoder-decoder architectures with attention mechanisms were also shown to translate speech in one language directly into text in another, skipping the intermediate transcription step entirely.⁶

Time Series and Sensor Data

Time series forecasting is another domain where RNNs have had lasting impact. Many real-world time series contain a mix of linear trends and nonlinear patterns, and RNNs are well suited to capturing the nonlinear component.⁷ GRU-based architectures have been used successfully for forecasting across multiple correlated time series, even in the presence of anomalies.⁸

A particularly interesting challenge in real-world data is missing values. Clinical health records, for example, are full of gaps: a patient’s blood pressure might be measured at irregular intervals, or certain lab tests might only be ordered sporadically. A GRU-based model called GRU-D was specifically designed to incorporate missing-data patterns (both which values are missing and how long they have been missing) directly into the network architecture, achieving strong performance on clinical prediction tasks using real hospital datasets.⁹

Music Composition

RNNs have also found a home in creative domains. The ability of recurrent networks to learn complex temporal patterns makes them well suited to music, where melody, rhythm, and harmony all unfold over time. One system, called Amadeus, used an LSTM network combined with reinforcement learning to compose polyphonic music, including intricate melodies, basic chords, and occasional contrapuntal sequences.¹⁰ A clever part of that approach was splitting polyphonic music into a small number of monophonic streams, which drastically reduced the complexity of the problem.

RNNs Versus Transformers

Starting around 2017, transformer architectures began displacing RNNs in many sequence-processing tasks, especially in natural language processing. Transformers use a self-attention mechanism that lets every element in a sequence attend to every other element simultaneously, rather than processing the sequence one step at a time. This parallel structure makes transformers much faster to train on modern hardware and gives them a stronger ability to capture relationships between distant parts of a sequence.¹¹

But the story is not as simple as “transformers replaced RNNs.” Transformers come with higher computational costs, particularly because their attention mechanism scales quadratically with sequence length. For very long sequences or tasks where you need to process data in real time as it streams in, the sequential nature of RNNs can be an advantage, not a limitation. LSTM and GRU architectures improved long-sequence modeling substantially through their gating mechanisms, and they remain competitive in settings where computational budgets are tight.¹²

The latest wave of architectures is blurring the boundaries further. Models like Mamba use selective state-space mechanisms that process sequences in linear time while still choosing what to remember or forget based on the current input, an approach that borrows ideas from both the recurrent and attention-based worlds.¹³ These newer models suggest that the field is converging toward hybrid designs rather than leaving recurrence behind entirely.

Running RNNs on Tiny Hardware

One area where RNNs have a clear practical edge is deployment on resource-constrained devices. If you want a smartwatch to recognize hand gestures, or a sensor node in a factory to detect anomalies in vibration data, you need a model that can run on a microcontroller with limited memory and no GPU.

Transformers, with their large parameter counts and attention computations, are difficult to squeeze into such environments. RNNs, particularly small GRU variants, are much more amenable to compression and quantization. A recent pipeline for deploying a compressed GRU variant called FastGRNN on ultra-constrained microcontrollers showed that the final model occupied just 566 bytes of weight storage and could sustain real-time inference at 50 Hz on an Arduino and an MSP430 microcontroller, with per-sample inference times well under 15 milliseconds.¹⁴ To put that in perspective, 566 bytes is smaller than this paragraph. The model used a lookup table for its activation functions, which provided a roughly 30-fold speedup on the multiplier-less MSP430 chip.

This kind of deployment is simply not feasible with most transformer architectures today. For on-device inference in wearables, IoT sensors, and embedded systems, lightweight recurrent models remain the practical choice.

Speeding Up Training on GPUs

Even when hardware is plentiful, the sequential nature of RNNs creates a training bottleneck. Because each time step depends on the output of the previous step, the computation cannot be trivially parallelized the way a transformer’s attention computation can. As RNNs grew larger and deeper, training times stretched into weeks or even months.

However, there is more parallelism available within an RNN than a naive implementation exploits. Research has shown that by exposing parallelism between operations within the network, such as running independent matrix operations simultaneously, you can achieve an order-of-magnitude speedup over a basic GPU implementation across a range of network sizes.¹⁵ These optimizations matter most when training large models, where even a modest per-step improvement compounds across millions of updates.

What Happens Inside an RNN

Neural networks in general are often criticized as “black boxes,” but RNNs offer some surprisingly interpretable internal behavior. Because the hidden state evolves step by step as the network reads through a sequence, researchers can track individual neurons (or “cells” in an LSTM) and observe what they respond to.

Experiments on character-level language models revealed that some individual cells specialize in tracking specific long-range dependencies. Certain cells activate when inside a quotation and deactivate when the closing quote appears. Others count the length of the current line and fire near the line boundary. Still others track bracket depth, incrementing when an opening bracket is read and decrementing when it is closed.¹⁶

This is remarkable because no one told the network about quotation marks or brackets. These cells emerged spontaneously from training on raw text. It suggests that RNNs can develop internal representations that correspond to meaningful structural features of their input, even without explicit supervision for those features. The specificity of these cells also helps explain why gated architectures like LSTMs outperform vanilla RNNs: the gating mechanism lets the network learn to hold onto a specific piece of information (like “we are inside a quote”) for as long as needed and then release it at exactly the right moment.

RNNs as Models of the Brain

Recurrent neural networks have found an unexpected second career in computational neuroscience, where they serve as models of how the brain’s own neural circuits might work. Biological neural networks are inherently recurrent: neurons connect to each other in loops, not in neat feedforward layers. This makes RNNs a natural choice for simulating brain regions.

One line of research uses RNNs to model the motor cortex, the brain region that controls voluntary movement. When researchers built an RNN that received simulated proprioceptive feedback (the body’s sense of its own position) and optimized it to control a virtual arm performing a reaching task, the simulated neural activity closely resembled real recordings from the motor cortex of animals. Further analysis showed that internal dynamics within the recurrent network dominated the neural activity patterns during movement, while proprioceptive feedback played a specific role in controlling when movements stopped. The feedback also improved the network’s robustness against noisy starting conditions.¹⁷

This is a genuinely different use of RNNs from the engineering applications discussed earlier. Here, the RNN is not a tool for making predictions; it is a scientific hypothesis about how a biological system works. The fact that a trained RNN reproduces the same patterns of activity seen in real brains lends credibility to the idea that the brain uses similar computational principles, and it gives neuroscientists a system they can manipulate in ways that would be impossible or unethical with living tissue.

Common Misconceptions

A few misunderstandings about RNNs are worth clearing up. The first is that RNNs are “obsolete” because transformers exist. Transformers have taken over many high-profile tasks, but RNNs remain the better choice in resource-constrained settings, in streaming applications where data arrives continuously, and in scientific modeling. The landscape is also shifting with hybrid and state-space models that incorporate recurrent ideas into newer frameworks.

The second is that LSTMs and GRUs “solved” the vanishing gradient problem completely. They dramatically improved the situation, but they are not immune to it. Very long sequences can still challenge gated architectures, and the title of a recent paper captures the sentiment well: vanishing and exploding gradients “are not the end of the story” when it comes to RNN training difficulties.¹⁸ Other factors, including the curvature of the loss landscape and the dynamics of optimization, also play a role in making long-term learning difficult.

A third misconception is that RNNs can only handle one type of data. In reality, they have been applied to text, speech, music, clinical health records, sensor streams, financial data, and even neuroscience simulations. The sequential processing framework is extremely general, and the variants available (bidirectional RNNs, deep stacked RNNs, attention-augmented RNNs) let you adapt the basic idea to a wide range of problems.

When to Choose an RNN Today

If you are deciding what architecture to use for a sequence problem in 2025, the choice depends on your constraints. Transformers are the default for large-scale language tasks where you have ample compute and your sequences fit comfortably in a context window. For time series forecasting, the picture is more mixed: RNNs, LSTMs, and GRUs remain competitive, and hybrid models that combine linear and nonlinear components are active research territory.¹⁹

For edge deployment on microcontrollers, RNNs are often the only feasible option. For real-time streaming tasks where you need to produce output as each new input arrives rather than waiting for the full sequence, the recurrent structure is a natural fit. And for scientific modeling, particularly in neuroscience, the recurrent architecture is chosen because it mirrors the system being studied, not because of any engineering convenience.

The broader trend in the field is not toward one architecture winning and the others disappearing. State-space models like Mamba borrow recurrent ideas while achieving transformer-like performance.²⁰ Some transformer variants add recurrence-like memory mechanisms. The core insight behind RNNs, that maintaining a running state while processing sequences is powerful, continues to influence every new generation of models even when the RNN label is not used.