Experiment Design
Experiment design is the plan that structures a product test before it runs: the hypothesis being tested, the method, who takes part, what is measured, how long it lasts and what result counts as success or failure. A good design makes the outcome interpretable, so the team can trust what the result says and knows in advance which decision each result will trigger.
How Experiment Design Works
A complete experiment design answers a short set of questions, written down before the test starts:
- Hypothesis. What do we believe, and what would prove it wrong? The hypothesis names the change, the audience and the expected effect.
- Method. Which test produces the evidence we need at the lowest cost? An interview, a landing page, a prototype test, a concierge pilot and an A/B test produce evidence of very different strength.
- Participants and assignment. Who takes part, how many, and how are they chosen? For causal claims, participants are randomly split into a control group and a treatment group. The randomized entity, often the user, is the unit of randomization.
- Metrics. One primary metric decides the outcome. In controlled experiments Ron Kohavi and colleagues call this the Overall Evaluation Criterion (OEC). Guardrail metrics check that nothing important gets worse.
- Success criteria. The threshold that counts as a pass, set in advance. For statistical tests, this includes the significance level and the smallest effect worth detecting.
- Sample size and duration. Enough participants and time to detect that effect with reasonable statistical power, and to cover normal cycles such as weekdays and weekends.
- Decision rule. What the team will do if the result passes, fails or is inconclusive.
Strategyzer's Test Card captures the core of this in four lines: we believe that, to verify that we will, and measure, we are right if.
Why Experiment Design Matters
Most failed experiments fail in the design, not the execution. Without a set success criterion, teams reinterpret disappointing results as wins. Without enough participants, a real effect looks like noise and a chance fluctuation looks like a win. Without a control group, a seasonal rise in sign-ups gets credited to a new feature. Stopping a test the moment it looks good, or scanning many metrics for any that moved, makes false positives much more likely.
A design written in advance also speeds decisions. When the decision rule is agreed up front, the result review is short: the team checks the outcome against the criteria and acts.
Experiment Design Example
A team wants to test whether a setup checklist improves activation for a project management tool.
- Hypothesis: New workspace admins who see a five-item setup checklist will invite teammates sooner.
- Method: Randomized controlled test on new sign-ups.
- Assignment: Each new workspace is randomly assigned to checklist or no checklist, at the workspace level, because teammates share one workspace.
- Primary metric: Share of workspaces with at least two invited members within seven days.
- Guardrails: Trial-to-paid conversion and support tickets per workspace.
- Success criterion: A statistically significant increase of at least three percentage points.
- Duration: Three weeks, based on a sample size calculation.
- Decision rule: Ship if it passes; if it is inconclusive, interview ten admins before redesigning.
Experiment Design vs. Product Experiment
A product experiment is the test itself as a learning activity: the thing the team runs to reduce uncertainty. Experiment design is the blueprint for that test. The same experiment idea, such as testing a checklist, can be designed well or badly, and the design determines whether the result can be trusted. Statistical concepts such as statistical significance belong to the design, because the threshold has to be chosen before the data arrives.