The hyperbolic tangent function, usually written as tanh, is one of the oldest and most important activation functions in neural networks. It squashes any input into a smooth curve bounded between −1 and +1, centered on zero. While it was largely displaced by ReLU in deep feedforward networks during the early 2010s, tanh never really went away. It remains the default gate activation in recurrent networks, the go-to output activation in generative models, and has recently resurfaced in reinforcement learning and adversarial training research, where its smoothness turns out to be a genuine advantage rather than a relic of an earlier era.
What Tanh Actually Does
An activation function sits between layers of a neural network and decides how strongly a neuron should “fire” given its inputs. Without an activation function, stacking layers of neurons would just produce a big linear equation, no matter how many layers you add. The activation introduces the nonlinearity that lets the network learn complex patterns.
Tanh takes a neuron’s weighted sum of inputs and maps it onto the range from −1 to +1. Large positive inputs push the output close to +1, large negative inputs push it close to −1, and inputs near zero pass through roughly unchanged. The curve is S-shaped and perfectly symmetric around the origin. That symmetry is one of its most practical features: the function’s outputs are centered on zero, meaning the average output across a batch of data tends to hover near zero rather than being biased toward one side.
This zero-centering distinguishes tanh from the older sigmoid activation, which maps everything to the range 0 to 1. When a layer’s outputs are all positive (as with sigmoid), the gradients flowing back during training are forced to be either all positive or all negative for a given batch. That constraint can cause the optimization to zigzag inefficiently toward a good solution. Because tanh produces both positive and negative outputs, it avoids that problem and generally trains faster on comparable architectures.
The Vanishing Gradient Problem
Tanh’s biggest weakness is also baked into its shape. At the extremes of the S-curve, where outputs are near +1 or −1, the function flattens out. Mathematically, its slope approaches zero at these “saturated” regions. During training, the network learns by passing error signals backward through each layer, multiplying by the local slope at each step. When many layers in a row have slopes close to zero, these multiplied-together signals shrink to almost nothing. The early layers of the network receive vanishingly small updates and essentially stop learning.
This phenomenon, known as the vanishing gradient problem, is not unique to tanh. Any activation function whose derivative stays below one can cause gradients to shrink as they travel through many layers.1PubMed Central. Mitigating the Vanishing Gradient Problem Using a Pseudo-Normalizing Method Sigmoid suffers from it even more severely because its maximum slope is only 0.25, while tanh’s peaks at 1.0 right at the origin. Still, once networks started growing to dozens or hundreds of layers in the mid-2010s, even tanh’s comparatively better gradient flow was not enough. Researchers needed something that did not saturate at all for large inputs.
How ReLU Changed the Landscape
The Rectified Linear Unit, or ReLU, solved the saturation problem by being almost absurdly simple: it outputs zero for negative inputs and passes positive inputs through unchanged, with no upper bound. For positive values, ReLU’s slope is always exactly 1, so gradients never shrink from multiplication through that neuron. This allowed very deep networks to train successfully for the first time, and ReLU quickly became the default activation for convolutional and feedforward architectures.
But the comparison between tanh and ReLU is more nuanced than “ReLU won.” Empirical benchmarks show that while ReLU tends to achieve the highest accuracy in image classification and text classification, tanh outperforms ReLU on image reconstruction tasks, producing higher-fidelity outputs as measured by signal-to-noise ratio.2arXiv. Deep Learning using Rectified Linear Units (ReLU) Both functions show stable convergence during training, unlike some other activations that can behave erratically.
ReLU has its own well-known problem. Because it outputs exactly zero for all negative inputs, neurons can get stuck in a state where they never activate. Once a neuron’s weights shift so that it receives only negative inputs, it produces zero output, receives zero gradient, and never recovers. These “dead neurons” are wasted capacity. Research in reinforcement learning found that roughly 60% of neurons in a representation layer were dead when using ReLU, compared to about 40% with tanh or sigmoid.3arXiv. Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL That is a significant difference in how much of the network is actually participating in the computation.
Where Tanh Remains the Standard Choice
Despite losing its default status in feedforward and convolutional networks, tanh holds firm in several important niches where its specific properties are exactly what the architecture needs.
Recurrent Networks and Sequence Modeling
Long Short-Term Memory (LSTM) networks and Gated Recurrent Units (GRUs) use tanh in two distinct roles. The “cell state” update in an LSTM uses tanh to produce candidate values between −1 and +1, while the gate mechanisms use sigmoid to produce values between 0 and 1 that act as multipliers. The combination lets the network decide both what new information to add (tanh’s range) and how much of it to let through (sigmoid’s range). Replacing tanh with ReLU in these roles tends to cause instability because ReLU’s unbounded positive output can cause cell states to grow without limit over long sequences.
Generative Adversarial Networks
In GANs, the generator network needs to produce outputs that match the range of the training data. When images are normalized to the range [−1, 1], tanh is the natural choice for the generator’s final layer because it maps directly onto that range. This is standard practice in GAN architectures. Research on analog hardware implementations of GANs has specifically highlighted that the generator’s last layer uses tanh instead of ReLU, with the bounded output ensuring compatibility with the physical voltage limits of the hardware.4Nature. Memristive GAN in Analog
Reinforcement Learning
Reinforcement learning has seen something of a tanh renaissance. In many RL settings, action spaces are bounded (a robot joint can only turn so far, a game controller has finite ranges), and tanh naturally constrains the policy network’s output to a fixed interval. Beyond just the output layer, recent work has found that using tanh throughout the network can reduce the dead neuron problem compared to ReLU. When combined with architectural innovations like latent assistance networks, tanh-based agents showed fewer dead neurons than either standalone tanh or sigmoid, which researchers attributed to the ability of multiplied tanh activations to resist long-term saturation.5arXiv. Latent Assistance Networks: Rediscovering Hyperbolic Tangents in RL
The Smoothness Advantage
One property of tanh that seemed like a footnote for years has turned out to be surprisingly important: it is smooth everywhere. Its curve has no corners, kinks, or discontinuities. ReLU, by contrast, has a sharp corner at zero where the function abruptly switches from flat to linear. That corner means ReLU’s derivative is undefined at exactly zero and jumps discontinuously from 0 to 1.
This matters most in adversarial training, where a network is deliberately exposed to worst-case inputs designed to fool it. Research on adversarial robustness found that ReLU significantly weakens adversarial training because of its non-smooth nature. The sharp corner makes it harder for the training process to find the hardest adversarial examples and compute reliable gradient updates. Replacing ReLU with smooth approximations, including tanh-family functions, produced networks that were more robust to adversarial attacks.6arXiv. Smooth Adversarial Training
The intuition is that a smooth activation function creates a smooth loss landscape, which gradient-based methods can navigate more reliably. When the landscape has sharp ridges (introduced by ReLU’s corner), the optimizer can miss important features of the terrain. For standard training on clean data, this rarely matters much. But when the training process needs to explore worst-case scenarios, smoothness provides a genuine edge.
Tanh as a Building Block for Modern Activations
Even where tanh is not used directly, it shows up as a component inside newer activation functions. The Mish activation, which gained popularity in computer vision around 2019, is defined as the input multiplied by the tanh of the softplus of that input.7arXiv. Mish: A Self Regularized Non-Monotonic Activation Function Mish inherits tanh’s smoothness but avoids full saturation at large positive inputs, giving it some of the best properties of both tanh and ReLU. It has been adopted in frameworks like YOLO for object detection.
Other tanh-derived functions include LiSHT, which multiplies the input by tanh to create a function that is non-monotonic (it dips slightly below zero before rising). The general pattern is that researchers keep returning to tanh as a smooth, bounded, zero-centered foundation and then modifying it to fix the saturation problem. The function’s mathematical properties make it a natural ingredient in activation function design, even when it is not the final product.
Connections to Physics and Biology
The tanh function was not invented for neural networks. It is a standard function in mathematics and physics, and its appearance in deep learning has deep roots in both statistical mechanics and neuroscience.
In statistical physics, the Ising model describes a lattice of interacting “spins” that can point up or down. When you calculate the average state of one spin given its neighbors, the result is a tanh function of the weighted influence from those neighbors.8arXiv. Correspondence Between Ising Machines and Neural Networks This is not merely a superficial resemblance. The mathematical structure of a neural network layer with tanh activation is formally analogous to mean-field theory in the Ising model: both systems compute a nonlinear function of a weighted sum of neighboring states. This connection has been known since the early days of neural network theory, and recent work continues to explore how it might be exploited to build neural network hardware based on physical Ising machines.
On the biological side, researchers have investigated why sigmoidal activation functions (including tanh) work so well by connecting them to how real neurons fire. Biological neurons exhibit a phase transition between quiescent and firing states: below a threshold they are silent, above it they spike. When you average over the stochastic behavior of many ion channels, the resulting firing-rate curve takes on a sigmoidal shape. One formal analysis showed that an artificial neuron with a sigmoidal activation is, mathematically, a mean-field approximation of this biological membrane dynamics.9arXiv. Formalising the Use of the Activation Function in Neural Inference The artificial neuron does not mimic a single spike; it represents the average firing behavior of a biological neuron, and tanh captures that average remarkably well.
Practical Guidance for Choosing Tanh
If you are building or fine-tuning a neural network and wondering whether tanh belongs somewhere in your architecture, here are the situations where it tends to earn its place:
- Bounded output layers: When your network’s output should fall within a fixed range, tanh maps naturally to [−1, 1]. This is common in GAN generators, policy networks in RL, and any regression task where the target is normalized to that interval.
- Recurrent architectures: LSTM and GRU cells use tanh internally by design. Swapping it out rarely helps and often hurts stability.
- Adversarial robustness: If you need your model to withstand adversarial attacks, replacing ReLU with tanh or a tanh-based smooth function in some or all layers can improve robustness during adversarial training.
- Shallow networks: In networks with only a few hidden layers, the vanishing gradient problem is much less severe, and tanh’s zero-centered output and smooth gradients can be an advantage over ReLU.
- Normalization pipelines: When your data is already centered and scaled to [−1, 1], tanh as a hidden-layer activation can preserve that centering through the network, reducing the need for batch normalization in some cases.
Conversely, in deep convolutional networks for image classification or large transformer-based language models, tanh in the hidden layers is generally a poor choice. The vanishing gradient problem is real, and these architectures train better with ReLU, GELU, or SiLU as their primary hidden-layer activation.
Running Tanh on Specialized Hardware
One underappreciated challenge with tanh is computational cost. Unlike ReLU, which is a single comparison (is the input positive?), tanh involves exponential calculations. On a GPU running a large model, this difference is usually negligible because the memory bandwidth, not the arithmetic, is the bottleneck. But on edge devices, microcontrollers, and custom accelerator chips, computing tanh efficiently becomes a real engineering problem.
Hardware designers have developed several approaches to make tanh fast on resource-constrained devices. Lookup tables store precomputed tanh values for a grid of inputs and interpolate between them. Piecewise linear approximations replace the smooth curve with a series of straight line segments that are cheap to compute. Polynomial approximations, such as those based on Chebyshev polynomials, achieve high accuracy with just a few multiplications and additions. The trade-off is always between accuracy and power consumption: a more faithful approximation uses more chip area and energy, while a rougher one introduces small errors that may or may not affect the final model’s performance. For applications like always-on voice recognition on a smartwatch or real-time control in a drone, getting tanh right in hardware is a nontrivial design decision.
The bounded output of tanh is actually an advantage here. Because the result is always between −1 and +1, hardware designers know exactly what numerical range to allocate, which simplifies fixed-point arithmetic. ReLU’s unbounded positive output requires more careful handling of dynamic range, though its simpler computation partly compensates for that.
Common Misconceptions
The most widespread misunderstanding about tanh is that it is obsolete, replaced by ReLU and its variants. The reality is more like a division of labor. Tanh is no longer the general-purpose default, but calling it obsolete ignores its continued role in recurrent networks, generative models, reinforcement learning, adversarial training, and as an ingredient in newer activation functions like Mish.
Another frequent confusion involves tanh and sigmoid being treated as fundamentally different functions. They are actually linear rescalings of each other: tanh of x equals two times sigmoid of 2x, minus one. The key practical difference is the output range (−1 to +1 versus 0 to 1), not some deep mathematical distinction. Choosing between them often comes down to whether your downstream computation benefits from zero-centered values or strictly positive ones.
A subtler misconception is that tanh “kills gradients” in the same way sigmoid does. While both saturate, tanh’s maximum derivative is four times larger than sigmoid’s. In networks with moderate depth, this difference is meaningful. The vanishing gradient problem with tanh typically emerges only in networks with many layers in sequence or when inputs consistently drive the function into saturation. With proper weight initialization and data normalization, tanh trains quite well in networks that are not excessively deep.