Bagging, short for bootstrap aggregating, is an ensemble technique in machine learning that builds many versions of the same model on slightly different random samples of the training data, then combines their predictions to produce a single, more stable result. Introduced by Leo Breiman in 1996, the idea is deceptively simple: instead of trusting one model’s answer, you train dozens or hundreds of models in parallel and let them vote. The technique underpins some of the most widely used algorithms in applied data science, most famously random forests, and remains a go-to strategy when a model’s predictions swing too wildly depending on which data it happens to see.
How Bagging Actually Works
The process starts with your original training dataset. From that dataset, you draw many new samples of the same size, each created by randomly selecting data points with replacement. “With replacement” is the key detail: each time you pick a data point, you toss it back into the pool, so some points show up more than once in a given sample while others get left out entirely. In a typical bootstrap sample, roughly 63% of the unique original data points appear at least once, and the remaining 37% or so are absent. You then train a separate model on each of these bootstrap samples. When it comes time to make a prediction on new data, every model produces its own answer, and the final output is determined by majority vote for classification problems or by averaging for regression problems.
The genius of this arrangement is that each model sees a slightly different slice of reality. If a particular data point is an outlier or noise, it will dominate some models but be absent from others. When you combine all the predictions, the influence of that noisy point gets diluted. The net effect is a reduction in variance, the tendency of a model to give very different answers when trained on slightly different data, without meaningfully increasing bias, the tendency to systematically miss the right answer. This variance reduction is where bagging earns its keep, and it works best on models that are already good at capturing complex patterns but are prone to overfitting, like deep decision trees.
Why Decision Trees and Bagging Fit Together
Decision trees are the classic partner for bagging because they are “high-variance” learners. A single decision tree can change dramatically if you add or remove even a handful of training examples. That instability is a weakness when you rely on one tree alone, but it becomes a strength in an ensemble: the more the individual models disagree with each other because of random data differences, the more their errors cancel out when averaged. If you bagged a stable, low-variance model like a simple linear regression, you would see almost no improvement, because every bootstrap sample would produce nearly the same model.
Random forests, the most popular bagging-based algorithm, take this idea one step further. In addition to training each tree on a different bootstrap sample, random forests also restrict each split in a tree to consider only a random subset of the available features. This extra layer of randomness pushes the individual trees even further apart, increasing the diversity of the ensemble and making the combined prediction more robust. The distinction matters: bagging is the general strategy of train-on-bootstrap-samples-then-aggregate, while random forests are a specific implementation that adds feature randomization on top.
The Out-of-Bag Shortcut for Estimating Error
Because each bootstrap sample leaves out roughly a third of the data, those leftover points serve as a built-in test set for the model trained on that sample. For any given data point in the original dataset, you can collect predictions from only the models whose bootstrap sample did not include it, then see how often those predictions were correct. This gives you the out-of-bag (OOB) error estimate, a convenient way to gauge performance without setting aside a separate validation set.
The OOB error is widely used in practice, especially with random forests, because it lets you use every available data point for both training and evaluation simultaneously. Research has shown, however, that the OOB error can be slightly pessimistic, meaning it overestimates how much the model will err on new data. The reason is that each individual OOB prediction comes from a model built on fewer effective training points than the full ensemble. Cross-validation, by contrast, evaluates a model built on a larger portion of the data, so its error estimate can be somewhat more optimistic. In a study examining this gap, the authors noted that the OOB error’s advantage is that the complete original sample is used both for building the classifier and for error estimation, which is why it remains popular despite the slight bias.1PubMed Central. On the overestimation of random forest’s out-of-bag error For most practical purposes, OOB error is close enough to be trusted as a quick diagnostic, but if precise performance benchmarks matter, cross-validation is worth the extra effort.
Subagging and the Question of Sample Size
Standard bagging draws bootstrap samples the same size as the original dataset. An alternative called subagging (short for subsample aggregating) draws smaller samples without replacement instead. The appeal is computational: if each model trains on half the data rather than all of it, you can build the ensemble faster, which matters when the dataset is large.
Recent theoretical work on subagging has produced some practical insights. For any given tree complexity, subagging consistently improves on a single tree, and the improvement tends to be larger when the individual trees are grown deep with many splits.2arXiv. When does Subagging Work? The same research also showed that subagging’s bias matches that of a single tree grown on a subsample of the same size, which in turn is similar to the bias of a full-data tree when both use the same number of splits.3arXiv. When does Subagging Work? There is a catch, though: a single tree grown at the right size can actually outperform a subagged ensemble whose individual trees are not sized well. The practical lesson is that you cannot simply grow the biggest trees possible and trust averaging to fix everything. Tuning the depth of the individual learners still matters.
Separately, research into the optimal sampling fraction for bootstrap resampling has found that no single universal fraction works best across all situations, but for standard error estimation, sampling fractions above about 37.5% of the data tend to land in a good region.4Revista Mundi Engenharia, Tecnologia e Gestão (ISSN: 2525-4782). Determination of Optimum Sampling Fraction for Bootstrap Resampling This aligns with the intuition that you need each sample to capture enough of the data’s structure to produce a competent model, while still leaving room for meaningful variation across samples.
How Many Models Do You Need?
One of the first questions people ask about bagging is how many base models to include. More is generally better up to a point, because each additional model adds another vote that helps smooth out noise. But the gains diminish: going from 10 trees to 50 usually delivers a noticeable improvement, while going from 500 to 1,000 rarely moves the needle. The computational cost, on the other hand, keeps climbing linearly.
Interestingly, research into ensemble pruning has shown that you can sometimes get better performance from a carefully selected subset of models than from the full ensemble. One approach uses boosting-style ordering to rank the bagged models by their contribution, then keeps only the subset that minimizes error. When researchers plotted error against the number of classifiers included in such an ordered ensemble, both training and test error curves showed a minimum for partially aggregated subensembles, meaning the best results came from a pruned ensemble rather than the full one.5Pattern Recognition Letters. Using boosting to prune bagging ensembles In practice, most people still use a fixed large number like 100 or 500 trees and accept the slight computational overhead, but pruning is an option when efficiency or deployment size is a concern.
Bagging Versus Boosting
Bagging and boosting are the two pillars of ensemble learning, and they attack the problem differently. Bagging trains models independently and in parallel, each on a random sample, then combines them democratically. Boosting trains models sequentially, where each new model focuses specifically on the mistakes the previous ones made. The result is that boosting tends to reduce bias as well as variance, making it a more aggressive learner.
In practice, boosting algorithms like gradient boosting and XGBoost often achieve higher raw accuracy than bagging, especially on structured tabular data. An experimental comparison of the two approaches on decision trees for ranking data confirmed that boosting outperformed bagging, consistent with known results from classification.6SpringerLink. Comparing Boosting and Bagging for Decision Trees of Rankings So why would anyone choose bagging? Several reasons. Bagging is far less prone to overfitting on noisy data, because it does not obsessively try to correct every past error. It is also embarrassingly parallel, meaning you can train all the models at the same time on different machines with no coordination needed, while boosting’s sequential nature makes it harder to speed up. And bagging is more tolerant of hyperparameter choices: a bagged ensemble of unpruned trees is a decent default that works well across a wide range of problems without much fiddling, whereas boosting requires careful tuning of learning rates and tree depths to avoid overfitting.
The Interpretability Trade-Off
A single decision tree is easy to understand. You can draw it, follow the splits, and explain to a non-technical person exactly why a particular prediction was made. Bag a hundred of those trees together and you gain accuracy, but you lose that transparency. The ensemble becomes a black box: each tree votes, but no single tree tells the whole story, and the trees may disagree about which features matter most.
This tension between accuracy and explainability is a real concern in fields like medicine or criminal justice, where stakeholders need to understand why a model made a particular decision. Researchers have explored compromises. One approach, called PCTBagging, partially consolidates the trees in the ensemble so that they share a common structure up to a certain depth and diverge only lower down. At a low consolidation level, this technique achieves accuracy similar to full bagging while retaining much of the interpretable structure of a single tree.7Information Sciences. PCTBagging: From inner ensembles to ensembles. A trade-off between discriminating capacity and interpretability Outside of purpose-built methods like this, practitioners often rely on post-hoc tools such as feature importance rankings and partial dependence plots to peer inside the ensemble. These are approximations rather than true explanations, but they are better than nothing when you need to explain to a regulator or a patient why the model said what it said.
Handling Imbalanced Data
Standard bagging draws bootstrap samples that mirror the class distribution of the original data. If your dataset is 95% healthy patients and 5% cancer patients, each bootstrap sample will also be about 95-5, and the models will learn to predict “healthy” almost all the time because that is the easiest way to minimize overall error. This is a problem whenever the rare class is the one you actually care about detecting.
A family of modified bagging methods addresses this by manipulating the sampling process. Roughly Balanced Bagging, for instance, adjusts each bootstrap sample so that the classes are more evenly represented, giving the models enough examples of the minority class to learn its patterns.8Statistical Analysis and Data Mining: The ASA Data Science Journal. Roughly balanced bagging for imbalanced data Other variants like SMOTEBagging combine resampling with synthetic data generation. These approaches are essential in fields like fraud detection, rare disease diagnosis, and manufacturing defect identification, where the events you most want to catch are the ones that barely appear in the training data.
Bagging for Time Series Forecasting
Applying bagging to time series is less straightforward than applying it to cross-sectional data. In a typical classification or regression task, individual data points are assumed to be independent, so randomly sampling them with replacement makes sense. Time series data has a natural ordering and dependencies: today’s value depends on yesterday’s, so shuffling the data randomly would destroy the information the model needs.
One approach that has proven effective decomposes a time series into its trend, seasonal pattern, and remainder component, then applies bootstrapping only to the remainder. The bootstrapped remainders are recombined with the original trend and seasonality to produce many synthetic versions of the series, and separate forecasting models are fitted to each. The final forecast is the average across all of these models. Research into this method found that bagging genuinely improves the performance of exponential smoothing forecasts, and the improvement stems from the diversity injected through the bootstrapped residuals.9European Journal of Operational Research. Exploring the sources of uncertainty: Why does bagging for time series forecasting work? This decompose-then-bag strategy respects the structure of the data while still gaining the variance-reduction benefits of aggregation.
Where Bagging Shows Up in Practice
Bagging’s most visible real-world impact comes through random forests, which are used across virtually every applied domain. In genomics and personalized medicine, random forest classifiers have been used to predict disease phenotypes from genetic variants, identify novel CRISPR arrays in bacteria, predict adverse drug responses by integrating multiple large-scale biological datasets, and discover biomarkers for cancer diagnosis.10PubMed Central. Machine learning in postgenomic biology and personalized medicine These are problems with high-dimensional data, often thousands of genetic features measured in a relatively modest number of patients, where random forests’ built-in feature selection and resistance to overfitting give them a natural advantage.
In cancer research specifically, ensemble methods including bagging have been combined with oversampling techniques to predict early cancer diagnoses from exome sequencing data, with one study reporting accuracy reaching about 83% on a held-out test set when bagging was used with multiple types of base classifiers.11PubMed Central. Implementation of ensemble machine learning algorithms on exome datasets for predicting early diagnosis of cancers In psychiatry, a bagging ensemble with feature selection outperformed other algorithms in predicting functional outcomes of schizophrenia using specific genetic biomarkers, suggesting that the technique can extract meaningful signal even from complex psychiatric genetics data.12Scientific Reports. Prediction of functional outcomes of schizophrenia with genetic biomarkers using a bagging ensemble machine learning method with feature selection
Beyond biomedicine, bagging appears in agriculture, finance, environmental science, and engineering. A study comparing ensemble techniques for predicting subclinical mastitis in dairy cows evaluated bagging alongside boosting, stacking, and super-learner ensembles using records from thousands of Holstein-Friesian cows across multiple herds.13PubMed Central. Performance comparison of machine learning models used for predicting subclinical mastitis in dairy cows: Bagging, boosting, stacking, and super-learner ensembles versus single machine learning models That such a straightforward technique shows up in contexts ranging from genome editing to dairy farming speaks to its versatility: bagging does not require domain-specific customization to deliver solid baseline performance.
Common Misconceptions
One widespread misunderstanding is that bagging always improves any model. It does not. Bagging reduces variance, so it helps models that are already unstable, meaning they change a lot with small changes in training data. A stable model like logistic regression or a shallow decision stump will barely benefit, because the bootstrap samples produce nearly identical models each time. If variance is not your problem, bagging is not your solution.
Another misconception is that more trees always leads to better results. While adding trees never hurts accuracy in theory (the error plateaus rather than rising), the law of diminishing returns kicks in fast. And as the pruning research showed, a carefully selected subset of the ensemble can sometimes outperform the full collection. Blindly scaling up the number of trees is not a substitute for thinking about whether the individual trees are well-configured.
A subtler error is assuming that because bagging averages over many models, you do not need to worry about the quality of the individual ones. The subagging research directly challenges this: growing individual trees to their maximum depth and trusting the ensemble to smooth things out is common practice, but a single tree grown at the optimal size can beat a subagged ensemble of poorly sized trees.14arXiv. When does Subagging Work? The ensemble amplifies the strengths of its members but cannot fully compensate for poorly chosen building blocks.
When to Reach for Bagging
Bagging is at its most useful when you have a moderately sized dataset, an unstable base learner like a decision tree, and you want a method that works well out of the box without extensive tuning. It is a particularly good choice when the data is noisy, because the averaging process smooths out the noise rather than chasing it. It is also the right call when you need to train the ensemble quickly and have access to parallel computing, since every model can be trained simultaneously.
If your primary concern is squeezing out every last fraction of a percent in accuracy and you have the time and expertise to tune hyperparameters carefully, boosting methods are typically the stronger choice. If interpretability is paramount and you cannot afford any black-box behavior, a single well-tuned model might serve you better than any ensemble. And if your data has strong temporal dependencies, you will need the decompose-then-bag strategy described earlier rather than naive bootstrapping. Bagging is not always the best tool, but it is rarely a bad one, and the fact that it requires so little configuration to work well makes it one of the most practical ensemble techniques available.