A designed experiment is a structured test where a researcher deliberately changes one or more input variables, called factors, and measures how those changes affect an outcome, called the response variable. Unlike simply observing what happens in the real world, a designed experiment puts the researcher in control: they decide which factors to manipulate, how extreme to make those changes, and which subjects receive which treatment. This deliberate control is what separates experimental design from other forms of statistical study and is what allows researchers to draw cause-and-effect conclusions rather than just noting correlations.
How It Differs From an Observational Study
The distinction matters because it determines what kind of conclusions you can draw. In an observational study, a researcher watches and records what happens without interfering. A cohort study tracking coffee drinkers over ten years, for example, can reveal associations between coffee consumption and heart disease, but it can’t prove coffee caused anything. The people who drink more coffee may also exercise less, sleep poorly, or have other habits that muddy the picture. These hidden influences are called confounding variables.
In a designed experiment, the researcher assigns treatments randomly. That randomization is the key ingredient. It ensures that confounding variables, whether known or unknown, are spread roughly evenly across all groups. If you randomly assign 200 people to either drink coffee or a placebo every morning for six months, the smokers, the athletes, and the poor sleepers all get distributed across both groups by chance. Any difference in outcomes can then be attributed to the coffee itself, not to some lurking third variable. This ability to establish causation is the primary reason designed experiments sit at the top of the evidence hierarchy in fields like medicine and engineering.
The Building Blocks: Factors, Levels, and Response
Every designed experiment has three core components. Factors are the input variables the researcher manipulates. In a study testing whether fertilizer type and watering frequency affect plant growth, both fertilizer type and watering frequency are factors. Levels are the specific values each factor takes. If you’re testing two fertilizers at high and low watering frequencies, each factor has two levels. Good practice is to set these at realistic extremes to capture the full range of effects. The response variable is the outcome you measure, such as plant height after eight weeks.
Choosing the right factors and levels is where domain expertise meets statistics. Too many factors make the experiment unwieldy. Too few levels can miss important patterns, like a response that peaks at a middle value you never tested. The art of experimental design is balancing thoroughness with practicality.
Three Principles That Make Experiments Valid
The statistical framework for designed experiments rests on three principles first formalized by the statistician R.A. Fisher in the early twentieth century: randomization, replication, and blocking.
Randomization is the formal, probability-based process of assigning treatments to subjects. It protects against selection bias, systematic bias, and unanticipated trends that could create false differences between groups. Without randomization, a researcher might unconsciously assign healthier patients to the new drug or place stronger plants closer to the window. Random assignment eliminates that possibility.
Replication means having enough independent observations in each treatment group to estimate the natural variation in your results. A single plant in each group tells you almost nothing, because you can’t separate the treatment effect from random noise. Multiple plants per group let you quantify how much variation is just background scatter and how much is a real treatment effect. The number of replications you need depends on how large an effect you expect to find and how much natural variability exists in your measurements. Smaller expected effects and noisier data both demand larger sample sizes. As a rough benchmark, doubling the standard deviation of your measurements quadruples the required sample size.
Blocking is the primary method for controlling known sources of unwanted variation. A blocking factor is a “nuisance” variable you know could influence results but aren’t interested in studying. If you’re testing three fertilizers across multiple greenhouses, each greenhouse is a block. By ensuring every fertilizer appears in every greenhouse, you can statistically separate greenhouse-to-greenhouse differences from the fertilizer effect. Blocking doesn’t require more subjects; it simply reorganizes the experiment to be more efficient.
Common Types of Experimental Designs
Completely Randomized Design
The simplest structure. Each subject gets one treatment assigned entirely at random, so subjects receiving different treatments are intermingled throughout the research environment. If you have 30 mice and three drugs, you randomly assign 10 mice to each drug. Analysis uses a one-way comparison of group means. This design works well when subjects are fairly uniform and the research environment is stable, but it can be inefficient when there’s a lot of natural variation between subjects because all that variation ends up in the error term.
Randomized Block Design
This design splits the experiment into independent blocks, each containing one subject per treatment, assigned in random order. Blocks can be separated by time, location, or any other nuisance variable. For example, if you run your three-drug experiment across multiple days, each day becomes a block containing one mouse per drug. This approach often provides substantially better control of both individual-to-individual and environmental variation. In one documented comparison, blocking provided extra statistical power equivalent to using about 40% more subjects, a major gain when subjects are expensive or difficult to obtain. Blocks can also be set up over a period of time to suit the researcher’s schedule, making this design both statistically and logistically appealing.
Factorial Design
When you want to study two or more factors simultaneously, a factorial design tests every combination of factor levels. A 2×2 factorial with two drugs (present or absent) and two therapies (present or absent) creates four groups. The power of this approach is that it reveals interaction effects: situations where the impact of one factor depends on the level of another. A study might find that a drug works well on its own and therapy works well on its own, but combining them doesn’t double the benefit, or perhaps it does more than double it. Without a factorial design, you’d never detect that interaction. The analysis produces a main effect for each factor plus an interaction effect between them.
Latin Square Design
When you need to control for two nuisance variables at once, a Latin square arranges treatments in a grid so that each treatment appears exactly once in every row and every column. The number of rows, columns, and treatments must all be equal. If you’re testing four fertilizers and want to block for both field position (four rows) and soil moisture (four columns), a Latin square handles both simultaneously. The tradeoff is an assumption that the treatment and the two blocking factors don’t interact with each other. Despite being underused, Latin squares are highly efficient designs for the right situation.
Planning for Statistical Power
A common pitfall in experimental design is running too few subjects. If your sample size is too small, real effects can go undetected simply because you didn’t collect enough data to distinguish them from random noise. Statistical power is the probability that your experiment will detect a real effect when one exists, and researchers typically aim for 80% power at a 5% significance level.
Three quantities drive the sample size calculation. First, the effect size you want to detect: smaller effects need more data. Second, the variability in your measurements: noisier data need more observations. Third, the power level you choose. At 80% power and 5% significance, detecting an effect of 0.56 standard deviations requires about 20 subjects per group. Detecting a subtler effect of 0.35 standard deviations pushes that to about 50 subjects per group. Running a power analysis before you begin the experiment is one of the most important steps in the planning process, because adding more subjects after you’ve started can introduce its own biases.
Steps to Conduct a Designed Experiment
The National Institute of Standards and Technology outlines seven steps for running a designed experiment:
- Set clear objectives. Define what you’re trying to learn and what a meaningful result looks like.
- Select process variables. Identify which factors to manipulate and which to hold constant or block.
- Choose an experimental design. Match the design type to your number of factors, available subjects, and blocking needs.
- Execute the design. Run the experiment following the randomization plan exactly.
- Check assumptions. Verify that the data behave as the statistical model requires (roughly equal variance across groups, roughly normal distribution of errors).
- Analyze and interpret results. Use the appropriate statistical test for your design type.
- Use or present the results. This step often leads to follow-up experiments that refine the initial findings.
Controlling Bias and Confounding
Beyond randomization, researchers use several strategies to keep confounding variables from distorting results. Restriction limits enrollment to a specific group (only adults aged 30 to 50, for instance), which eliminates age as a confounding factor but narrows how broadly the results apply. Matching pairs subjects with similar characteristics and places one from each pair in each treatment group, ensuring the groups are balanced on important variables from the start.
When confounding can’t be fully eliminated by design, statistical adjustments step in during analysis. Stratification divides the data into subgroups and examines treatment effects within each subgroup. Regression methods can mathematically adjust for confounding variables that were measured during the study. These tools are powerful, but they only work for confounders you thought to measure. Randomization remains the only method that protects against confounders you didn’t anticipate, which is why it’s considered the single most important feature of a well-designed experiment.