Sampling with replacement is a method of drawing items from a group where each item is returned before the next one is picked. This means any item can be selected more than once, and every draw has exactly the same odds as the one before it. It’s a foundational concept in statistics and machine learning, and understanding it clarifies why many common statistical formulas work the way they do.
How Sampling With Replacement Works
The process follows three steps, repeated as many times as you need:
- Draw: Randomly select one item from the population.
- Record: Note which item you picked.
- Return: Put the item back into the population before drawing again.
Imagine a jar with 10 colored beads. You reach in, pull out a blue bead, write “blue” on your notepad, then drop the bead back into the jar. When you reach in again, all 10 beads are still there. You have a 1-in-10 chance of picking that same blue bead again, and a 1-in-10 chance of picking any other specific bead. The probabilities never shift.
This is distinct from sampling without replacement, where you set aside each item after drawing it. In that case, the pool shrinks with every draw, and the probability of selecting any remaining item changes each time.
Why Returning Items Matters for Probability
Putting items back keeps each draw independent. The outcome of your first pick has zero effect on the probabilities of your second pick, third pick, or any pick after that. Independence is a powerful property because it simplifies the math behind statistical inference considerably.
When draws are independent, the standard error of the sample average is simply the population’s standard deviation divided by the square root of your sample size. That clean formula only holds because each observation carries the same probability structure. Without replacement, you need an additional correction factor (called the finite population correction) that accounts for the shrinking pool.
Independence also connects directly to the binomial distribution, one of the most widely used probability models. The binomial answers questions like “what is the probability of getting exactly 4 successes in 10 trials?” But it only applies when the probability of success stays the same on every trial. If you’re drawing from a physical population, that condition requires replacement. Picture drawing balls from a group of three (one red, two black). The chance of picking red is 1 in 3 on every draw, but only if you put the ball back each time.
The 10% Rule: When It Barely Matters
In practice, many real surveys and experiments sample without replacement. You survey 500 people out of a city of 200,000, and you don’t ask the same person twice. Technically, this violates the independence assumption. But the effect is so small it’s negligible.
This is formalized as the 10% rule: if your sample is no larger than 10% of the population, you can treat the draws as approximately independent and use the same statistical tools that assume replacement. A sample of 500 from 200,000 is 0.25% of the population, well under the threshold. The probabilities shift so slightly between draws that the difference is rounding error.
The 10% rule is why introductory statistics courses teach formulas based on independence (and therefore on replacement) even though most data collection happens without it. For large populations, the distinction is academic.
A Simple Example With Names in a Hat
Say you write the names of five students on slips of paper: Andy, Karl, Tyler, Becca, and Jessica. You want to draw a sample of two names with replacement. On the first draw, you pull out Tyler. You put Tyler’s name back in the hat. On the second draw, you pull out Tyler again. Your final sample is {Tyler, Tyler}.
That result is impossible without replacement, since Tyler’s slip would already be gone. With replacement, every combination is on the table, including duplicates. This might seem odd for a student survey, but it’s exactly the behavior that certain statistical techniques depend on.
Where Sampling With Replacement Is Used
The most prominent application is bootstrapping. Bootstrapping is a technique for estimating properties of a population (like a confidence interval or standard error) when you only have one sample to work with. It works by repeatedly resampling from your existing data with replacement, creating hundreds or thousands of new “bootstrap samples,” each the same size as your original dataset. Some data points appear multiple times in a given bootstrap sample; others don’t appear at all. By calculating your statistic of interest on each bootstrap sample and looking at the spread of results, you get a reliable estimate of uncertainty without needing to collect new data.
Replacement is what makes bootstrapping possible. If you sampled without replacement from a dataset of 100 observations and drew 100 observations, you’d get the exact same dataset every time. With replacement, each resample is a different mix, and that variation is what lets you estimate how much your results would fluctuate across different samples from the real population.
Machine learning uses the same principle in several algorithms. Random forests, for example, train many individual decision trees, each on a different bootstrap sample of the training data. This technique, called bagging (short for bootstrap aggregating), reduces the tendency of models to overfit by introducing controlled randomness. Boosting algorithms use similar resampling strategies. In all these cases, sampling with replacement is the mechanism that generates diversity across models trained on the same underlying dataset.
With vs. Without Replacement at a Glance
- Probability across draws: With replacement, it stays constant. Without replacement, it shifts after each draw.
- Independence: Draws are independent with replacement. Without replacement, they’re dependent (though approximately independent when the sample is small relative to the population).
- Duplicates: Possible with replacement, impossible without.
- Standard error formula: With replacement, it’s the standard deviation divided by the square root of the sample size. Without replacement from a smaller population, you multiply by a correction factor that shrinks the estimate slightly.
- Typical use: Sampling with replacement is standard in resampling methods like bootstrapping and bagging. Sampling without replacement is more common in surveys and experiments where contacting the same person twice would be wasteful or confusing.
The choice between the two comes down to what you need from the math. When constant probability and true independence matter for your analysis, replacement is the way to go. When you’re collecting data from real people or physical items and your population is large, sampling without replacement is more natural and functionally equivalent.