The logistic regression cost function, usually called log loss or binary cross-entropy, measures how far a model’s predicted probabilities land from the actual outcomes. For a single training example, it equals the negative log of the probability the model assigns to the correct class. When the model confidently predicts the right answer, the cost is near zero; when it confidently predicts the wrong answer, the cost shoots toward infinity. That steep penalty for confident mistakes is the defining feature of this cost function and the reason it works so well for classification problems.
What Log Loss Actually Computes
Imagine you have a binary outcome: yes or no, spam or not spam, click or no click. Logistic regression outputs a number between 0 and 1, which you interpret as the probability that the outcome is “yes.” The cost function then asks a simple question: given the true label, how surprised should we be by the probability the model assigned?
If the true label is 1 (yes), the cost for that example is −log(p), where p is the predicted probability. If the true label is 0 (no), the cost is −log(1 − p). In both cases, a perfect prediction yields a cost of zero, and a terrible prediction yields a cost that climbs without bound. The total cost function averages this quantity over all training examples, giving you a single number that captures how well the model fits the data overall.
This formulation comes directly from the idea of maximum likelihood estimation. Logistic regression assumes each data point is drawn from a Bernoulli distribution whose parameter depends on the input features. Maximizing the likelihood of observing the actual labels under that assumption is mathematically equivalent to minimizing the average log loss. So the cost function is not an arbitrary scoring rule someone invented; it falls out naturally from the probabilistic model that logistic regression assumes.
Why Not Just Use Squared Error
If you have used linear regression, your first instinct might be to measure error the same way: take the difference between the prediction and the true label, square it, and average. You can do this with logistic regression, and the math will not catch fire. But it creates a problem that makes training painfully slow or outright unreliable.
The issue is the shape of the error surface. Logistic regression passes its linear combination of features through a sigmoid function, which squashes everything into the 0-to-1 range. When the sigmoid output is near 0 or near 1, the curve is almost flat. If you pair that flat sigmoid with a squared-error cost, the gradient of the cost with respect to the model’s weights becomes tiny in exactly the region where the model is making confident but wrong predictions. The model barely updates its weights even though it is badly wrong, a situation sometimes called the vanishing gradient problem in this context.
Log loss sidesteps this entirely. Because it takes the negative log of the predicted probability, the cost grows sharply as the prediction moves toward the wrong extreme. The gradient stays large when the model is confidently wrong, which is precisely when you want the biggest updates. The result is faster, more stable training.
Convexity and Why It Matters for Optimization
One of the most useful properties of the logistic regression cost function is that it is convex with respect to the model’s weights. A convex function has a single valley: no matter where you start, if you follow the downhill direction you will eventually reach the global minimum. There are no local minima to get trapped in and no saddle points to stall on.
This has been formally established by showing that the second derivative of the cross-entropy loss with respect to the weights is always positive, confirming that optimization methods built around convexity will converge to the best solution.1arXiv. A proof of convergence of multi-class logistic regression network In practice, this means standard gradient descent, along with fancier variants, will reliably find the optimal weights for a logistic regression model. You do not need the kind of careful initialization or learning-rate scheduling that deeper neural networks demand.
Because of this convexity, researchers have also developed specialized second-order optimization methods for logistic regression that use curvature information to converge faster, particularly when new data arrives sequentially.2SIAM Journal on Control and Optimization. An Efficient Stochastic Newton Algorithm for Parameter Estimation in Logistic Regressions These methods exploit the smooth, bowl-shaped loss surface to take larger, more informed steps toward the minimum compared to plain gradient descent.
Convexity is a property of the logistic regression cost function specifically, not of cross-entropy loss in general. Once you swap out the linear model for a neural network with hidden layers, the same cross-entropy formula is no longer convex with respect to the network’s weights. The loss surface gains ridges, valleys, and saddle points. That is why logistic regression occupies a sweet spot: you get a probabilistic classifier with a cost function that is mathematically guaranteed to have a clean optimization landscape.
When the Cost Function Misbehaves
Convexity guarantees a unique minimum exists, but it does not guarantee the minimum is at a finite, well-behaved set of weight values. In certain data configurations, the optimal weights that minimize log loss are literally infinite. This happens when one or more features perfectly separate the two classes, a situation called complete separation.
If a feature perfectly predicts the outcome, the model can keep pushing its corresponding weight higher and higher, driving the predicted probability closer and closer to 0 or 1 for every training example. The log loss keeps decreasing but never actually reaches zero because the sigmoid never quite touches 0 or 1. The optimization algorithm chases this vanishing target, producing weight estimates that grow without bound.3Oxford Academic (American Journal of Epidemiology). Separation in Logistic Regression: Causes, Consequences, and Control
In practice, separation tends to show up when you have a small dataset with many features, or when a categorical variable happens to perfectly split the outcome in your sample. The warning signs are absurdly large coefficient estimates, enormous standard errors, and software that either fails to converge or spits out a warning message. The model has not learned anything useful; it has memorized a quirk of the training data.
Several fixes exist. The most common is regularization: adding a penalty term to the cost function that discourages large weights. L2 regularization (also called ridge) adds a term proportional to the sum of squared weights, effectively placing a ceiling on how large any weight can grow. This modified cost function is still convex, still has a unique minimum, but that minimum now sits at finite values even when the data are perfectly separated. Penalized maximum likelihood and Bayesian approaches with informative priors accomplish much the same thing. The key insight is that pure log loss trusts the data completely, and when the data contain a pattern that is too clean, that trust becomes a liability.
Adjusting the Cost Function for Imbalanced Classes
Standard log loss treats every training example equally. A correct prediction on a positive case and a correct prediction on a negative case contribute equally to the total cost. This is fine when the classes are roughly balanced, but it creates a subtle problem when one class vastly outnumbers the other. If 99% of your examples are negative, a model that predicts “negative” for everything achieves a very low average log loss. The tiny number of positive examples barely dents the total cost, so the model has little incentive to learn what makes them different.
The most straightforward fix is weighted cross-entropy: multiply the loss for each example by a class-specific weight, so that mistakes on the rare class cost more. If positive cases are 100 times rarer than negatives, you might weight each positive-case loss by a factor of 100. This forces the model to pay attention to the minority class, but it can also make training noisier because each rare example now has outsized influence on the gradient.
A more refined approach is focal loss, which reshapes the cost function so that well-classified examples contribute less to the total loss regardless of their class. It adds a modulating factor that shrinks the loss for examples the model is already getting right and preserves the full penalty for examples the model finds difficult. Focal loss was originally developed for object detection, where the vast majority of candidate regions in an image contain background rather than objects of interest. It down-weights the easy negatives so that training focuses on the sparse set of hard examples.4IEEE Transactions on Pattern Analysis and Machine Intelligence. Focal Loss for Dense Object Detection The concept has since migrated to other classification tasks where class imbalance or easy-versus-hard example imbalance is a concern.
Both weighted cross-entropy and focal loss modify the cost function while preserving its core structure. You are still computing a log-probability-based penalty; you are just reweighting which examples matter most during optimization. The gradients change shape, but the underlying idea of penalizing confident wrong answers remains intact.
What the Cost Function Gives You Beyond Classification
Many classifiers output a score, but logistic regression’s cost function specifically trains the model to output well-calibrated probabilities. Because log loss is a proper scoring rule, minimizing it encourages the model to output probabilities that match the true frequencies in the data. If the model says “there is a 70% chance this email is spam,” then among all emails it scores at 70%, roughly 70% should actually be spam. This calibration is not just a nice theoretical property; it is essential whenever the predicted probability itself matters, as in medical risk scores, insurance pricing, or any decision where you need to weigh the cost of acting against the cost of waiting.
Other classifiers like support vector machines or random forests output scores that rank examples correctly but are not calibrated as probabilities. A common post-hoc fix is to run those raw scores through logistic calibration, essentially fitting a logistic regression on the scores. But research has shown this logistic calibration step can actually make things worse for classifiers whose score distributions are heavily skewed, because the logistic curve family does not include the identity function and can uncalibrate a classifier that was already well-calibrated.5Project Euclid. Beyond sigmoids: How to obtain well-calibrated probabilities from binary classifiers with beta calibration The point for the cost function discussion is that logistic regression’s native log loss already bakes calibration into training, which is one reason the model remains popular even in an era of far more complex alternatives.
Regularization as a Cost Function Modification
In practice, almost nobody trains logistic regression by minimizing raw log loss. Nearly every implementation adds a regularization term that penalizes large weights. This modified cost function is the sum of the average log loss and a penalty term, and the choice of penalty shapes the model’s behavior in important ways.
L2 regularization adds a penalty proportional to the sum of squared weights. It discourages any single weight from growing too large but does not push weights all the way to zero. The resulting model uses all features but with dampened coefficients. This is the default in most software libraries for logistic regression.
L1 regularization adds a penalty proportional to the sum of absolute weight values. It tends to push some weights exactly to zero, effectively selecting a subset of features. If you suspect many of your input features are irrelevant, L1 regularization can produce a simpler, more interpretable model. The tradeoff is that the cost function is no longer smooth at zero, which requires slightly different optimization algorithms.
Elastic net combines both penalties, giving you a tunable mix of feature selection and coefficient shrinkage. In all three cases, a hyperparameter controls how strongly the penalty influences the total cost. Setting it too low returns you to near-raw log loss with all its separation risks; setting it too high forces the model to underfit by shrinking coefficients more than the data warrant. Cross-validation is the standard way to choose a good value.
From the cost function’s perspective, regularization changes the shape of the optimization landscape. The pure log loss surface is a smooth bowl. Adding L2 regularization sharpens the bowl, pulling the minimum closer to the origin in weight space. Adding L1 regularization introduces creases along the axes, which is why weights can land exactly at zero. Understanding these geometric effects helps explain why certain regularization choices work better for certain problems, even though the log loss portion of the cost remains unchanged.
Multi-Class Extensions
Binary logistic regression handles two classes, but many real problems have more. The standard extension is multinomial logistic regression, sometimes called softmax regression, which generalizes the sigmoid to the softmax function and the binary cross-entropy to categorical cross-entropy. Instead of a single probability, the model outputs a probability for each class, and the cost function penalizes the negative log of the probability assigned to the correct class.
The good news is that categorical cross-entropy inherits the convexity of binary cross-entropy. The cost function for multi-class logistic regression is still convex with respect to the weights, so gradient-based optimization still finds the global minimum.6arXiv. A proof of convergence of multi-class logistic regression network Separation can still occur, and regularization is still the standard remedy.
One practical wrinkle with multi-class problems is that the number of weights scales with the number of classes. A model with 100 features and 10 classes has roughly 1,000 weights to optimize. The cost function’s gradient is still straightforward to compute, but the higher-dimensional weight space means training takes longer and the risk of overfitting grows. Regularization becomes even more important in this regime.
How Log Loss Relates to Cross-Entropy in Deep Learning
If you have spent time in the deep learning world, you have seen “cross-entropy loss” everywhere. The cross-entropy used to train a neural network’s final classification layer is the same mathematical formula as logistic regression’s log loss. The difference is scope: in logistic regression, the features feed directly into a linear function followed by a sigmoid, and the entire cost surface is convex. In a deep network, the features pass through multiple nonlinear layers before reaching that final sigmoid or softmax, and the cost surface is wildly non-convex.
This distinction matters because many of the clean guarantees that make logistic regression’s cost function so appealing do not carry over. Gradient descent on a deep network’s cross-entropy loss can get stuck in poor local regions. Weight initialization, learning rate schedules, batch normalization, and other engineering tricks become necessary to navigate the rough landscape. The cost function formula is identical, but its interaction with the model’s architecture changes everything about how optimization behaves.
Understanding this connection also clarifies why logistic regression is often described as a single-layer neural network. It is: one layer of weights, one sigmoid activation, cross-entropy loss. Every concept about the cost function discussed so far, including convexity, separation, regularization, and calibration, applies to that single-layer case. Stacking more layers preserves the loss formula but abandons the convexity, which is the fundamental tradeoff of going deeper.
Numerical Stability in Practice
Computing −log(p) when p is very close to zero produces enormous numbers. Computing it when p is exactly zero produces infinity. Since computers work with finite-precision floating-point arithmetic, naive implementations of log loss can overflow or produce NaN (not a number) values that crash training.
Every mature machine learning library handles this by computing the log loss in a numerically stable way, typically by working with the raw linear outputs (called logits) rather than the probabilities themselves. Instead of first computing the sigmoid and then taking the log, the library combines both steps into a single formula that avoids the dangerous intermediate values. In TensorFlow, for example, this is the purpose of the sigmoid_cross_entropy_with_logits function, and in PyTorch it is handled by BCEWithLogitsLoss.
If you are implementing logistic regression from scratch for a homework assignment or a custom application, the most common rookie mistake is computing the sigmoid first and then taking the log. This works fine for predictions in the comfortable range of 0.1 to 0.9, but it will produce garbage for extreme predictions. Clipping the probabilities to a small range like [1e-7, 1 − 1e-7] before taking the log is a workable patch, but using the log-sum-exp trick or a built-in numerically stable function is the proper solution.
This is a pure implementation concern; it changes nothing about the mathematics. The cost function is the same regardless of whether you compute it naively or carefully. But in practice, numerical instability is one of the most common sources of mysterious training failures, especially for people writing their own optimization loops rather than relying on established libraries.