What Is Residual Variance? Definition and Examples

Residual variance is the leftover scatter in your data that a model’s predictions don’t account for. It is often called unexplained variance, and it is the part of the variation that your model’s predictor variables cannot explain.1Statology. What is Residual Variance? (Definition & Example) That makes it a scorecard for a model: the less that is left unexplained, the better the model fits.2Medium. What Is Residual Variance? Definition and Examples

What a Residual Is and Why the Leftovers Get Measured

A residual is the difference between an observed value and the value the model predicted for it.2Medium. What Is Residual Variance? Definition and Examples Residual variance is simply the variance of those differences across all your data points.2Medium. What Is Residual Variance? Definition and Examples The reason this works is that the total variance of the outcome splits into a part explained by the regression and a part left unexplained.3Statistics How To. Residual Variance (Unexplained / Error) Unexplained variance is what remains when you subtract the regression’s share from the total.3Statistics How To. Residual Variance (Unexplained / Error)

Each residual is the vertical distance from a point to the line. Points above the line have positive residuals and points below have negative ones. Squaring the gaps stops the positives and negatives from cancelling each other out. It also means big gaps dominate, so one far-off point can raise the residual variance a lot.

Residuals are easy to confuse with errors. The error is the gap from the true, unknowable relationship. The residual is the gap from your fitted model, which you can actually compute. Residual variance is your estimate of the error’s variance, which textbooks write as sigma squared.

In an ordinary least squares fit that includes an intercept, the residuals sum to zero by construction, so averaging the raw gaps would always say the model is perfect. The variance is therefore about the size of the misses, not their direction.

A smaller value indicates a better fit and a larger one a poorer fit.1Statology. What is Residual Variance? (Definition & Example)2Medium. What Is Residual Variance? Definition and Examples But the number is in squared units of the outcome, so its size only means something next to the outcome’s own spread. If you predict house prices in dollars, the residual variance is in squared dollars, which has no everyday meaning. Taking the square root gives a figure in dollars that you can picture as a typical miss.

How to Calculate Residual Variance

Residual variance equals the sum of squared observed-minus-predicted gaps divided by n minus p minus 1, where n is the number of observations and p the number of predictors.2Medium. What Is Residual Variance? Definition and Examples Squared gaps appear because the regression itself is built on the sum of squared differences between predicted and observed points.1Statology. What is Residual Variance? (Definition & Example) The divisor n minus p minus 1 is called the degrees of freedom. It is smaller than n because fitting each predictor uses up some of the data’s freedom.

Each estimated coefficient, including the intercept, pulls the fitted line closer to the data and removes one independent piece of information. Dividing by the smaller count keeps the estimate from being too optimistic about how small the true scatter is.

Software does this for you. In R, a model with 24 observations and one predictor gives residual variance as the sum of squared residuals divided by 22.4Lancaster University. What Is Residual Variance? Definition and Examples The 22 is 24 observations, minus one predictor, minus one for the intercept. Most regression output reports the square root instead, labelled residual standard error, which is in the outcome’s own units.

A small example shows the steps. Suppose a one-predictor model predicts house price from floor area, and you have actual and predicted prices for five houses, in thousands of dollars. The actual prices are 200, 250, 300, 350 and 400, and the predictions are 210, 240, 310, 340 and 410. In a spreadsheet, that is a column for actual, a column for predicted and a column for the gap.

Subtracting predicted from actual gives residuals of -10, 10, -10, 10 and -10. Squaring each gives 100, and five of them add up to a sum of squared residuals of 500. With n of 5 and one predictor, the divisor is 5 minus 1 minus 1, which is 3, so the residual variance is about 166.7. Its square root is about 12.9, so the model’s typical miss is roughly 12.9 thousand dollars.

Is a High or Low Residual Variance Good

Within the same data, lower is better, but no universal cutoff exists. The ratio of residual variation to total variation gives the percentage of variation the predictors cannot explain.1Statology. What is Residual Variance? (Definition & Example) So the same raw value can be small for wildly variable outcomes and large for tightly clustered ones.

Adding predictors isn’t automatically an improvement. Adding a predictor almost always lowers the sum of squared residuals at least a little, even if the new variable is unrelated to the outcome. The divisor shrinks with each predictor too, so residual variance does not fall as steadily as the raw sum does. AIC and BIC go further and charge a penalty for each extra parameter, so a model has to improve its fit enough to justify its added complexity.

A tiny value can even be a warning. On a small data set, a model with many predictors can shrink its residual variance almost to zero by fitting the quirks of those particular points. Such a model will usually predict new data poorly, which is what overfitting means. A good check is whether the model still performs well on data it was not fitted to.

How Residual Variance Relates to R-Squared

R-squared, shown in nearly every regression output, is the share of variation in the outcome that the predictors explain, given as a percentage.1Statology. What is Residual Variance? (Definition & Example) Explained and unexplained variance together make up the total,3Statistics How To. Residual Variance (Unexplained / Error) so one minus R-squared is roughly the share left unexplained, the residual share.

With an R-squared of 0.80, the predictors explain about 80% of the variation and roughly 20% remains in the residuals. If the leftover share doubled to 40%, R-squared would fall to 0.60.

Two models can both have an R-squared of 0.80 yet very different residual variances if their outcomes are on different scales. A model predicting salaries in dollars will have a huge raw residual variance next to one predicting test scores out of 100, even with identical fit quality. Raw residual variance is not comparable across outcomes measured on different scales, while R-squared, being a share, does not depend on the units.

When Residuals Spread Unevenly

Standard linear regression assumes residuals are normally distributed with the same variance for every observation.5PubMed Central. Homoscedasticity: an overlooked critical assumption for linear regression A single residual variance number only fully describes the model when that holds. This even spread of residuals is called homoscedasticity, a term you’ll meet in assumption checklists.6Statistics Solutions. Assumptions of Linear Regression Analysis Regression also assumes residuals are independent of each other, not autocorrelated.6Statistics Solutions. Assumptions of Linear Regression Analysis

The first check is to plot residuals against predicted values and look for patterns.7Tilburg Science Hub. Testing and Correcting for Heteroskedasticity A healthy plot is a random band around zero with no trend.8Wikipedia. Errors and residuals If the residuals fan out, the spread is uneven, which is called heteroscedasticity.8Wikipedia. Errors and residuals A curved trend instead suggests the model’s shape is wrong, for example a curve fitted with a straight line.8Wikipedia. Errors and residuals Formal tests such as Breusch-Pagan and White can confirm what the plot suggests.9Decision Making: Applications in Management and Engineering. Violation of the assumption of homoscedasticity and detection of heteroscedasticity

Take household spending predicted from income. Low earners tend to spend similar amounts, so their points sit close to the line. High earners differ more, with some saving much and others spending lavishly, so their points scatter widely. The residual plot then widens like a cone as predicted spending rises.

A single residual variance averages over the tight region and the wide region, so it overstates error for low incomes and understates it for high ones. Prediction intervals built on that one number will be too wide in some places and too narrow in others. Coefficient estimates usually remain unbiased, but the standard errors and tests based on them can be unreliable.

What to Do About Uneven Residual Variance

One fix is to transform the outcome, most often by taking its logarithm.10Penn State Eberly College of Science. 10.1 – Nonconstant Variance and Weighted Least Squares If a log is too strong, use a square root, and if it is too weak, use a reciprocal.10Penn State Eberly College of Science. 10.1 – Nonconstant Variance and Weighted Least Squares Another fix is weighted least squares, which lets the variance differ by predictor values and weights observations accordingly.10Penn State Eberly College of Science. 10.1 – Nonconstant Variance and Weighted Least Squares After either fix, replot the residuals to confirm the fan has flattened.

A log suits outcomes such as prices or incomes, where differences grow in proportion to the level rather than by fixed amounts. On the log scale, a fan produced by proportional variation often flattens into an even band. The log needs every outcome value to be positive, so it can’t be used directly on data with zeros or negatives.

Transforming changes interpretation. Coefficients now describe changes in the log of the outcome, which are roughly percentage changes. Predictions come out on the log scale and must be converted back before you can quote them in dollars or units. The residual variance is in log units too, so it can’t be compared directly with a model fitted to the raw outcome.

Weighting keeps the original scale but needs a sensible guess at how spread changes. It gives noisy observations less say in the fit.

Residual Variance in Forecasting and Other Fields

Residual variance sets the width of prediction intervals. Assuming normal errors, a 95% prediction interval is the forecast plus or minus 1.96 times the estimated standard deviation.11OTexts. Forecasting: Principles and Practice A larger residual variance therefore means wider, less precise intervals. A sales forecast of 500 units with a standard deviation of 20 gives a margin of 1.96 times 20, about 39 units, so the interval runs roughly 461 to 539. Halving the standard deviation to 10 halves the margin to about 20 and so halves the interval width.

Intervals also widen the further ahead you forecast, because uncertainty grows.11OTexts. Forecasting: Principles and Practice There is more room for conditions to change and for errors to compound, so a forecast one month out can carry a much wider interval than one for next week from the same model.

In genetics, models split trait variation into genetic effects and a residual environmental part.12PubMed Central. Heritability—a paradox of quantitative genetics The residual there represents influences the genes in the model don’t capture. In machine learning, once a model can’t improve further, the leftover error estimates the variability the inputs simply can’t predict.13arXiv. Diagnosing the Conditional-Mean Barrier in Scientific Machine-Learning Surrogates