The derivative of cross entropy loss, when paired with the softmax function, reduces to a strikingly simple expression: for each class, it equals the model’s predicted probability minus the true label. If a model assigns a 0.7 probability to the correct class, the gradient for that output is 0.7 − 1 = −0.3, nudging the prediction upward toward 1. This elegance is not a coincidence but a product of how the logarithm in cross entropy cancels neatly against the exponential in softmax, and it is one of the main reasons cross entropy became the default training objective for classification tasks across machine learning.
Why the Gradient’s Simplicity Matters for Training
Every time a neural network trains on an example, it computes how wrong its prediction was and then adjusts its internal weights in proportion to that wrongness. The derivative of the loss function is the engine of that adjustment. A messy or poorly behaved derivative can make training slow, unstable, or prone to getting stuck. The cross entropy gradient avoids these problems because it scales naturally with the size of the error: when the model is badly wrong (predicting 0.1 for a class that should be 1), the gradient is large (−0.9), creating a strong corrective push. When the model is nearly right (predicting 0.95), the gradient is small (−0.05), producing only a gentle nudge. This proportional behavior means the model self-regulates its learning speed without needing elaborate external adjustments.
Compare this to squared error loss, where the gradient involves multiplying the error by the derivative of softmax directly. That product can become vanishingly small when the model is very wrong but confident in the wrong answer, a situation sometimes called gradient saturation. Cross entropy largely sidesteps this problem because the logarithm amplifies the penalty for confident wrong predictions, keeping the gradient large enough to drive learning even in those difficult cases.
How the Logarithm and Exponential Cancel
To understand where the clean gradient comes from without wading into pages of calculus, think of it this way. The softmax function converts a set of raw scores (logits) into probabilities using exponentials. Cross entropy then takes the negative logarithm of the predicted probability for the correct class. When you differentiate the composition of these two operations, the exponential from softmax and the logarithm from cross entropy are inverse operations, so they largely cancel each other out. What survives is just the difference between the predicted probability and the target, that “predicted minus actual” expression.
This cancellation is specific to the log-exponential pairing. If you swap in a different loss function while keeping softmax, or swap in a different activation while keeping cross entropy, the derivative generally becomes more complicated. The pairing is not arbitrary: it traces back to the mathematical family of exponential distributions, where the logarithm is the natural “matching” function. This is part of why cross entropy occupies a privileged position among loss functions rather than being just one option among many.
The Link to Maximum Likelihood
Minimizing cross entropy turns out to be mathematically identical to maximizing the likelihood of the observed data under the model’s predicted distribution.1arXiv. Quantum Cross Entropy and Maximum Likelihood Principle This equivalence means that when you train a classifier with cross entropy, you are implicitly finding the set of model parameters that makes the training labels most probable according to the model. The derivative inherits this interpretation: each gradient step moves the parameters in the direction that increases the probability the model assigns to the correct class for each training example.
This connection is reassuring because maximum likelihood estimation has deep theoretical foundations. It is known to be statistically efficient under certain conditions, meaning it extracts as much information from the data as possible. But the equivalence also explains a weakness: maximum likelihood can overfit, and cross entropy’s gradient will happily drive the model to assign probability 1.0 to every training label, even noisy or mislabeled ones. Several of the modifications discussed below exist precisely to counteract this tendency.
Focal Loss and Reshaping the Gradient for Imbalanced Data
Standard cross entropy treats every misclassification equally in terms of gradient shape. In medical imaging or object detection, where one class might outnumber another by 100 to 1, this becomes a problem. The model can achieve low average loss by simply predicting the majority class almost everywhere, and the gradient from the rare class gets swamped.
Focal loss addresses this by adding a modulating factor to the standard cross entropy gradient. The modification multiplies the loss by a term that shrinks toward zero as the model’s predicted probability for the correct class increases.2PubMed Central. Unified Focal loss: Generalising Dice and cross entropy-based losses to handle class imbalanced medical image segmentation In practical terms, examples the model already classifies confidently contribute very little gradient, while hard or misclassified examples contribute much more. The derivative of focal loss is no longer just “predicted minus actual” but includes additional terms from differentiating the modulating factor, making the gradient more complex but better targeted. The tunable parameter gamma controls how aggressively easy examples are down-weighted: at gamma equal to zero, focal loss is identical to standard cross entropy.
This gradient reshaping has proven particularly useful in tasks like segmenting tumors from medical scans, where the lesion might occupy a tiny fraction of the image. Without focal loss, the gradient from the vast background region dominates training, and the model struggles to learn the rare foreground class.
Label Smoothing Changes What the Gradient Pushes Toward
Standard cross entropy uses “hard” labels: the correct class is labeled 1, and every other class is labeled 0. The gradient therefore pushes the model to assign all probability mass to a single class, which can produce overconfident predictions that generalize poorly to new data. Label smoothing modifies the target distribution by redistributing a small amount of probability from the correct class to all other classes, so instead of targeting 1.0 for the right class and 0.0 for everything else, the model might target 0.9 and roughly 0.01 spread across the rest.
The derivative’s form stays the same (predicted minus target), but the target values change, and this has a meaningful effect on what the gradient does during training. Research examining this through the lens of neural collapse has found that label smoothing produces features that are more evenly separated across classes, promoting a geometric structure where class representations form a maximally spread-out arrangement.3arXiv. Cross Entropy versus Label Smoothing: A Neural Collapse Perspective The study found that compared to standard cross entropy, label smoothing resulted in stronger between-class separation at comparable levels of within-class compactness, which the authors linked to better generalization. In contrast, vanilla cross entropy can push within-class features to collapse too tightly around training examples, overfitting to particular data points rather than learning broadly useful representations.
From a gradient perspective, the key shift is that label smoothing prevents the gradient from ever fully vanishing for the correct class. With hard labels, once the model predicts 1.0, the gradient is zero and no further adjustment happens. With smoothed labels, the target is below 1.0, so the model still receives a gentle gradient signal even when it is very confident, discouraging it from pushing logits to extreme values.
When Noisy Labels Corrupt the Gradient
Because the cross entropy gradient treats every label as ground truth, mislabeled training examples inject corrupted gradient signals. A label that says “cat” on an image of a dog generates a gradient pushing the model to associate dog-like features with the cat class. Neural networks have enough capacity to memorize these contradictions, so training loss can decrease even as the model learns the wrong associations.
Mean absolute error (MAE) has been proposed as a noise-robust alternative because its gradient does not depend on how confident the model is, making it less susceptible to memorizing outlier labels. However, this same property makes MAE harder to optimize with deep networks on complex datasets because the gradient provides less useful signal about how to adjust.4PubMed Central. Generalized Cross Entropy Loss for Training Deep Neural Networks with Noisy Labels The generalized cross entropy loss bridges this gap by parameterizing a family of loss functions that interpolate between MAE and standard cross entropy. By tuning a single parameter, you can trade off noise robustness against optimization efficiency. The derivative of this generalized form incorporates a power term that controls how much the gradient is influenced by confident predictions, letting you dial back the tendency to memorize noisy labels without completely abandoning the optimization advantages of the logarithmic gradient.
The practical takeaway is that the “predicted minus actual” gradient of standard cross entropy is optimal when labels are clean but becomes a liability when they are not. In real-world datasets where some fraction of labels are always wrong, modifying the gradient’s behavior near high-confidence predictions is often the most effective intervention.
Calibration and the Overconfidence Problem
A well-calibrated model’s predicted probabilities reflect actual accuracy: if it says “90% chance this is a cat” across many images, roughly 90% of those predictions should be correct. Modern neural networks trained with cross entropy are famously miscalibrated, tending to be overconfident. The cross entropy gradient’s structure contributes to this because it always pushes toward higher confidence in the training label, with no penalty for overshooting.
Temperature scaling is the most common post-hoc fix. After training, all logits are divided by a single learned temperature parameter before applying softmax, which softens the probability distribution. This generally improves average calibration across a dataset, but it applies the same correction to every prediction, reducing confidence on correct predictions just as much as on incorrect ones.5AAAI Publications. Sample-Dependent Adaptive Temperature Scaling for Improved Calibration Sample-dependent approaches try to address this limitation by learning different temperature adjustments for different inputs, so that high confidence is preserved when the model is actually right and reduced when it is likely wrong.
From the perspective of the derivative, calibration problems arise because the gradient during training cares only about pushing the correct-class probability higher, not about whether the resulting probability distribution is well-calibrated. Cross entropy is minimized when all probability mass sits on the correct class, regardless of how uncertain the underlying data actually is. The gradient has no mechanism for saying “0.85 is confident enough for this ambiguous example, stop pushing.” Every modification that improves calibration, whether label smoothing, temperature scaling, or mixup training, works by altering the effective gradient to limit this runaway confidence.
Cross Entropy in Knowledge Distillation
Knowledge distillation trains a smaller “student” model to mimic the output distribution of a larger “teacher” model rather than learning directly from hard labels. The classic approach adds a distillation loss (the cross entropy between the student’s softened predictions and the teacher’s softened predictions) on top of the standard cross entropy against the true labels. Decomposing the distillation loss reveals something surprising: it can be expressed as a combination of a standard cross entropy term and an additional term that has the same mathematical form as cross entropy but operates on different probability distributions.6arXiv. Rethinking Knowledge Distillation via Cross-Entropy
The gradient that the student model receives during distillation is therefore a blend of two cross entropy gradients: one pulling it toward the hard labels and another pulling it toward the teacher’s soft probability distribution. The research found a subtle problem with this arrangement. The extra term forces the student’s relative probabilities to match the teacher’s absolute probabilities, and the sum of probabilities across classes differs between the two distributions, creating an optimization mismatch. The proposed fix reformulates the loss so that both components operate on compatible probability scales, producing a cleaner gradient signal for the student.
This matters practically because the gradient’s quality in distillation directly determines how much knowledge transfers from the large model to the small one. A mismatched gradient means the student spends optimization effort reconciling incompatible probability scales rather than learning useful structure from the teacher’s predictions.
Weight Norms, Logit Scale, and How the Gradient Interacts with Network Internals
The cross entropy gradient does not operate in isolation. It flows backward through the entire network via the chain rule, and the behavior of each layer shapes how the gradient signal transforms along the way. Recent work on the phenomenon of “grokking,” where networks suddenly generalize long after memorizing training data, has highlighted an underappreciated aspect of this interaction. Under cross entropy loss, the weight norm of the network controls the scale of the logits fed into softmax, which in turn controls how saturated the softmax distribution becomes.7arXiv. What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy
The finding is that the weight norm acts as an upstream handle on softmax saturation, and it is this saturation that mediates the transition from memorization to generalization. When logits are large, softmax outputs are nearly binary (close to 0 or 1), and the cross entropy gradient becomes very small for most classes because the predicted distribution is already peaked. When logits are smaller, the distribution is softer, the gradient flows more evenly across classes, and the network is more responsive to structural patterns in the data. The research showed this effect is specific to cross entropy: under mean squared error loss, the logit scale is effectively pinned, and the weight norm influences training through a different mechanism entirely.
This has practical implications for regularization choices. Weight decay, which penalizes large weight norms, does not just prevent overfitting in some vague sense. Under cross entropy, it specifically controls how peaked the softmax distribution is, which determines the gradient landscape the optimizer navigates. Too little weight decay allows the logits to blow up, saturating softmax and shrinking gradients to near zero. Too much weight decay keeps the distribution so soft that the model struggles to commit to any classification.
Binary Versus Multi-Class Forms
Most discussions focus on the multi-class case with softmax, but the binary cross entropy variant used in two-class problems and multi-label tasks has its own gradient behavior worth understanding. Binary cross entropy treats each output independently using the sigmoid function instead of softmax. The derivative for each output is still “predicted minus actual,” but because sigmoid operates independently per output (rather than enforcing that all outputs sum to 1 as softmax does), the gradients for different outputs are decoupled.
This decoupling is exactly what you want in multi-label classification, where an image might correctly be tagged as both “beach” and “sunset.” Softmax-based cross entropy would force the model to choose between the two labels since increasing confidence in one necessarily decreases confidence in the other. Binary cross entropy’s gradient has no such coupling, allowing each label to be driven toward its target independently. The cost is that you lose the relative structure softmax provides: with binary cross entropy, the model has no built-in mechanism for expressing that two classes are mutually exclusive.
The choice between binary and softmax cross entropy therefore comes down to the structure of the label space, and each carries a different gradient geometry. In practice, using the wrong one is a common source of training failures that produces confusing symptoms: a model trained with softmax cross entropy on a multi-label task will show gradients that fight each other, and accuracy will plateau well below what the architecture is capable of.
Numerical Stability in Practice
The mathematical elegance of the cross entropy derivative assumes exact arithmetic, but real implementations run on floating-point numbers with limited precision. When a predicted probability approaches 0, the logarithm in cross entropy approaches negative infinity, producing enormous gradient values that can destabilize training. When a predicted probability approaches 1, the logarithm approaches 0 and the gradient vanishes, but the computation of log(1 − p) for the incorrect classes can underflow.
This is why deep learning frameworks compute softmax and cross entropy together in a single fused operation rather than computing softmax first and then taking the log. The fused computation, often called “log-softmax” or “softmax with cross entropy,” works in the log domain throughout, avoiding the intermediate step where probabilities can become dangerously close to 0 or 1. The gradient of this fused operation is numerically identical to “predicted minus actual” but avoids the catastrophic precision loss that would occur if the two steps were computed separately. If you ever implement cross entropy from scratch, this is the single most important implementation detail: never compute softmax and then take the log of the result. Compute the log-sum-exp directly.
Some research has specifically audited for numerical artifacts in cross entropy training. Work on grokking behavior in transformers, for example, included a float64 softmax-collapse audit to confirm that the observed training dynamics were genuine phenomena rather than artifacts of numerical precision.8arXiv. What Does the Weight Norm Control in Grokking? Logit-Scale Mediation under Cross-Entropy The fact that researchers felt the need to include such checks underscores how sensitive cross entropy training can be to precision issues, particularly in experiments where logit scales grow large.
Self-Supervised Learning and Contrastive Losses
Cross entropy’s gradient also appears in disguise within self-supervised learning frameworks, where models learn representations without labeled data. In contrastive learning methods, the InfoNCE loss used by popular frameworks is structurally a cross entropy computed over augmented views of the same data point. The “correct class” is the matching augmented pair, and the “incorrect classes” are all other data points in the batch. Analysis of these methods has shown that during each gradient update, the weights at each layer are adjusted by an operator that amplifies features varying across data samples while surviving averages over data augmentations.9arXiv. Understanding Self-supervised Learning with Dual Deep Networks
The cross-entropy-like structure of InfoNCE means the gradient behaves similarly to supervised classification: it pushes matching pairs closer together in representation space and non-matching pairs apart, with the strength of the push proportional to how confused the model is. The difference is that the “classes” are not fixed semantic categories but are dynamically constructed from data augmentations in each batch. Understanding that contrastive losses inherit the gradient properties of cross entropy helps explain why many of the same training tricks (temperature scaling of logits, large batch sizes to provide more “negative classes,” careful learning rate schedules) transfer directly from supervised classification to self-supervised pretraining.