The right regression model depends on two things: what your outcome variable looks like and how your data behaves. A continuous outcome like weight or revenue points you toward linear regression. A yes/no outcome like “purchased” or “diagnosed” points you toward logistic regression. Count data, survival times, and nested data structures each have their own models. The key is matching your model to the shape of your data rather than forcing your data into a familiar model.
Start With Your Outcome Variable
Your outcome variable, the thing you’re trying to predict or explain, narrows the field immediately. If it’s continuous (temperature, salary, test scores), you’re in linear regression territory. If it’s binary (yes/no, pass/fail), you need logistic regression. If it’s a count of events (number of hospital visits, number of customer complaints), you need Poisson or negative binomial regression. If it’s the time until something happens (days until equipment failure, months until remission), you need survival analysis like Cox regression.
Getting this first decision wrong is the most common mistake. No amount of tuning will fix a linear regression model that’s trying to predict a binary outcome.
Continuous Outcomes: Linear Regression and Its Variants
Standard linear regression (ordinary least squares, or OLS) works when four assumptions hold. First, the relationship between your predictors and outcome is actually linear. Second, the residuals (the gaps between your predictions and the real values) are normally distributed. Third, those residuals have roughly equal spread across all predicted values, a property called homoscedasticity. Fourth, the residuals are independent of each other, meaning there’s no hidden pattern connecting them.
You don’t need your raw variables to be normally distributed. What matters is that the residuals are normal. That’s a distinction many introductory courses gloss over, but it changes how you diagnose problems. Plot your residuals after fitting the model, not your raw data before.
When the relationship between your predictors and outcome curves rather than following a straight line, you have two main options: polynomial regression or splines. Polynomial regression adds squared or cubed terms to capture curves, but higher-degree polynomials tend to overfit, chasing noise in your data rather than real patterns. Splines split the data into segments and fit smooth curves locally, which produces more stable and accurate fits. Comparative testing shows splines generally outperform polynomial models in terms of prediction error when both use the same degree of complexity. If your scatter plot shows a clear curve, try splines first.
Checking for Multicollinearity
When two or more predictors are highly correlated with each other, your regression coefficients become unstable. Small changes in the data can swing them wildly. The standard diagnostic is the Variance Inflation Factor (VIF). A VIF above 10 signals serious multicollinearity. Some researchers use a stricter cutoff of 5, which flags predictors that deserve closer inspection even if they haven’t reached crisis levels. If you find high VIF values, you can drop redundant predictors, combine them, or switch to a regularized model.
Checking for Autocorrelation
If your data has a natural ordering, like time series or spatial measurements, your residuals may be correlated with each other. The Durbin-Watson test produces a value between 0 and 4. A value near 2 means no autocorrelation. Values drifting toward 0 suggest positive autocorrelation (neighboring residuals tend to be similar), while values near 4 suggest negative autocorrelation (neighboring residuals tend to alternate). If you detect autocorrelation, standard OLS will underestimate your standard errors, making results look more significant than they are. Time series models or generalized least squares are better choices in that situation.
Binary and Categorical Outcomes: Logistic Regression
When your outcome has two categories, binary logistic regression is the standard tool. It estimates the probability of one outcome versus the other, and it doesn’t require the assumptions about residuals that linear regression does.
When your outcome has more than two unordered categories (say, choosing between three brands), multinomial logistic regression handles the comparison across all groups simultaneously. When those categories have a natural order (mild, moderate, severe), ordinal logistic regression is the better choice because it uses that ranking information. However, ordinal logistic regression requires what’s called the parallel regression assumption: the effect of each predictor must be consistent across the category boundaries. If that assumption fails (which you can test), fall back to multinomial logistic regression, which makes no such assumption.
Count Data: Poisson vs. Negative Binomial
If your outcome is a count of discrete events, Poisson regression is the starting point. It assumes that the average count equals the variance of the counts. In practice, real-world count data almost always has more spread than the average would predict. This is called overdispersion.
When overdispersion is present, meaning the variance exceeds the mean, negative binomial regression is the better fit. It adds an extra parameter to capture that additional spread. You can think of it as a more flexible version of Poisson regression. A simple way to check: fit a Poisson model, then compare the residual deviance to the degrees of freedom. If the deviance is much larger, your data is overdispersed and you should switch to negative binomial.
Time-to-Event Data: Cox Regression
When your outcome is the time until an event occurs, and some subjects haven’t experienced the event yet (censored data), Cox proportional hazards regression is the go-to model. It estimates how different predictors speed up or slow down the rate of the event without assuming any particular shape for the baseline survival curve.
The critical assumption is proportional hazards: the relative effect of each predictor stays constant over time. If smoking doubles the hazard of a particular event, it must double it at year one and at year ten. You need to verify this before trusting the results. Plotting Schoenfeld residuals over time is the standard check. If the proportional hazards assumption fails for a particular variable, you can stratify by that variable or allow its effect to change over time.
Nested or Repeated Data: Mixed-Effects Models
When your observations aren’t independent because of the data’s structure, like students nested within schools, patients nested within hospitals, or repeated measurements on the same person, standard regression will give you misleading results. Mixed-effects models (also called multilevel or hierarchical models) account for this by separating the variation at each level.
The first question is whether your research question actually requires a multilevel approach. If your outcome exists at the highest level (one value per school, not per student), regular regression works fine because those values are already independent. But if you’re predicting student-level outcomes and students are grouped within schools, ignoring that structure inflates your confidence in the results.
Mixed-effects models also require larger samples than standard regression. While a minimum of 25 observations can work for simple fixed-effects regression, mixed-effects models with random effects typically need substantially more to produce reliable estimates. The exact number depends on how many groups you have and how many observations per group.
Too Many Predictors: Regularized Regression
When you have many predictors relative to your sample size, or your predictors are highly correlated, standard OLS becomes unreliable. Regularized regression methods add a penalty for model complexity, which stabilizes the estimates.
Ridge regression works best when you believe most predictors contribute something, even if their individual effects are small. It shrinks all coefficients toward zero but never eliminates any entirely, which makes it effective when you have many correlated predictors and don’t need to identify a sparse subset.
Lasso regression works best when you suspect only a handful of predictors truly matter. It can shrink coefficients all the way to zero, effectively removing irrelevant variables from the model. This makes it a powerful feature selection tool. However, Lasso struggles with highly correlated predictors. It will arbitrarily pick one from a group of correlated variables and ignore the rest, and it can’t select more variables than your sample size.
Elastic net combines both penalties, getting you the variable selection ability of Lasso with the stability of Ridge when predictors are correlated. It tends to select groups of correlated features together rather than arbitrarily picking one. If you’re unsure whether to use Ridge or Lasso, elastic net is often the safest starting point.
Comparing Models With Information Criteria
Once you’ve narrowed your choices, you often need to compare two or more candidate models on the same dataset. AIC (Akaike Information Criterion) and BIC (Bayesian Information Criterion) both balance how well a model fits against how complex it is.
AIC penalizes each additional parameter by 2 units. It favors models that predict well, even if they’re slightly more complex. BIC penalizes more heavily as sample size grows, so it tends to prefer simpler models. In both cases, lower values are better, and the scores only make sense as comparisons between models fit to the same data.
One caution with AIC: an uninformative predictor that absorbs some random noise can sometimes improve AIC by a tiny amount, sneaking into your “best” model without contributing real explanatory power. Don’t rely on AIC rankings alone. If a variable improves AIC by less than 2 units, it’s not meaningfully contributing.
A Practical Decision Sequence
- Continuous outcome, linear relationships, assumptions met: standard linear regression (OLS).
- Continuous outcome, curved relationships: spline regression or polynomial regression.
- Continuous outcome, many correlated predictors: Ridge, Lasso, or elastic net.
- Binary outcome (yes/no): binary logistic regression.
- Categorical outcome, no natural order: multinomial logistic regression.
- Categorical outcome, ordered levels: ordinal logistic regression (if the parallel assumption holds).
- Count outcome, variance equals mean: Poisson regression.
- Count outcome, variance exceeds mean: negative binomial regression.
- Time-to-event with censoring: Cox proportional hazards regression.
- Grouped or repeated observations: mixed-effects (multilevel) regression.
No single checklist replaces actually examining your data. Plot your variables, fit a candidate model, then check the residuals. The diagnostics after fitting often matter more than the assumptions you check before. A model that looks right on paper can fall apart when the residuals reveal patterns you didn’t expect, and the fix is usually straightforward: switch to the model type that handles whatever your residuals are telling you.