Check If a Product Change Works by Comparing Randomly Split Groups (A/B Test)
Use it to learn if a product change really helps. It asks what you will decide and measure, then plans the groups, number of people, test length, and harm checks, and reports the effect and how sure it is.
To use this AI rule
## Decide Whether a Comparison Test Can Answer Your Question
Begin with the decision the evidence must inform.
A controlled online experiment is useful when a change can be assigned, an outcome can be measured, and exposure can be managed safely.
Confirm that:
- the intervention is sufficiently stable during the test;
- eligible units can be assigned without unacceptable interference;
- the outcome can occur within an observable period;
- the expected learning is worth the traffic, time, and risk;
- the organization can act on each plausible result.
Do not use an experiment to compensate for an undefined problem or unreliable instrumentation.
Use qualitative research, operational analysis, simulation, or observational methods when the question concerns why behavior occurs, rare harm, long-term effects, or a change that cannot be assigned.
## Write a Prediction That Says What You Will Do with Each Result
State the proposed change, affected population, expected behavioral mechanism, primary outcome, direction of effect, and decision consequence before implementation.
Ground the hypothesis in prior evidence rather than testing arbitrary surface changes.
A useful hypothesis identifies:
- the current problem and supporting observation;
- the intervention and what remains unchanged;
- the expected response and its mechanism;
- the primary metric and analysis window;
- the smallest effect worth acting on;
- the action for positive, neutral, and harmful results.
Distinguish exploration from confirmation.
If the team cannot state what result would alter the decision, refine the question before consuming experimental traffic.
## Choose Who Is Included and How They Are Split into Groups
Specify the experimental unit, the population eligible for enrollment, and the moment assignment occurs.
The unit may be a person, account, device, session, location, or other entity, but the choice must match how the intervention and outcomes propagate.
Document:
- inclusion and exclusion rules;
- allocation proportions;
- randomization or matching procedure;
- persistence of assignment across visits;
- handling of new and returning units;
- known interactions among units.
Prevent one unit from receiving incompatible variants unless the design explicitly models repeated exposure.
When members of a group influence one another, assign at the group level or account for interference in design and analysis.
## Build Test Versions That Change Only What You Want to Check
Change only what the hypothesis requires and keep unrelated behavior consistent.
The comparison should isolate a meaningful intervention rather than bundle several untraceable modifications.
For every variant, record:
- the exact user-visible and system-level difference;
- implementation owner and version;
- expected exposure conditions;
- accessibility, privacy, and operational risks;
- fallback behavior if delivery fails;
- checks proving that the intended variant was served.
Multiple simultaneous changes can estimate the effect of a package but cannot identify which component caused it.
Use a factorial or multivariate design only when the sample, analysis plan, and decision need justify the added complexity.
## Pick the Main Number to Judge and the Numbers That Warn of Harm
Choose one primary decision metric that represents the hypothesized behavior and can be interpreted at the experimental-unit level.
Define its numerator, denominator, unit, attribution window, exclusions, and treatment of repeated events.
Add guardrail metrics for effects the primary metric could conceal:
- retention or repeat use;
- task completion and error;
- latency and reliability;
- refunds, complaints, or support demand;
- revenue, cost, or margin;
- privacy, fairness, or safety indicators.
Do not redefine success after observing the data.
Label diagnostic metrics as secondary, and state whether they support explanation, monitoring, or future hypothesis generation.
## Set How Many People You Need and When to Stop, Before Starting
Estimate the required sample from the baseline rate or variance, minimum detectable effect, statistical error criteria, allocation, and planned analysis method.
Choose the minimum effect from practical value and implementation cost, not from the sample that happens to be convenient.
Before launch, record:
- expected eligible traffic;
- target sample by variant;
- minimum runtime needed to cover known cycles;
- maximum runtime or expiration condition;
- fixed-horizon or sequential analysis method;
- rules for stopping because of harm or technical failure.
Statistical significance is not a substitute for adequate exposure or representative timing.
Do not repeatedly inspect an ordinary fixed-horizon test and stop at the first favorable threshold crossing.
## Check That Group Splits and Data Tracking Work Before Reading Results
Verify the assignment, exposure, event collection, and metric computation before interpreting outcomes.
Run the full pipeline with test accounts or synthetic events, then compare recorded data with expected behavior.
Check:
- allocation matches the configured proportions;
- identifiers remain stable and variants do not leak;
- exposure is recorded before attributable outcomes;
- events are neither duplicated nor silently dropped;
- clocks, time zones, and windows are aligned;
- metric queries reproduce known examples.
An identical-variant comparison can reveal allocation, instrumentation, or analysis defects, but it does not prove that every future intervention is valid.
Define acceptable diagnostic tolerances before reviewing the result.
## Start with a Small Share of Users and Watch for Harm
Begin with limited exposure when the intervention could affect revenue, reliability, trust, or participant welfare.
Increase allocation only after delivery and guardrail checks remain within predefined limits.
Monitor operational signals such as:
- assignment and exposure counts;
- error rates and latency;
- missing or malformed events;
- severe guardrail movement;
- support or incident reports;
- imbalance in important pre-experiment attributes.
Separate emergency monitoring from outcome analysis.
Stopping for verified harm or broken delivery protects participants and data quality; stopping because an early estimate is unattractive changes the statistical contract.
## Keep the Test Unchanged While It Runs
Keep variant logic, eligibility, metrics, and analysis code stable for the planned test unless safety or correctness requires intervention.
Record every operational event that could alter exposure or measurement.
Avoid:
- overlapping experiments with unmodeled interactions;
- campaigns or releases that affect only one variant;
- manual reassignment of selected units;
- excluding observations based on post-assignment behavior;
- extending or shortening the test to obtain a desired result.
If a material change occurs, determine whether to restart, amend the analysis plan, or report the result as compromised.
Do not silently combine data collected under incompatible definitions.
## Report How Big the Effect Is and How Sure You Can Be
Analyze units according to their assigned condition unless a prespecified design requires another estimand.
Report effect size and uncertainty, not only a threshold label.
The analysis should include:
- sample and exposure counts by variant;
- data-quality and balance checks;
- primary and guardrail estimates;
- absolute and relative effects where meaningful;
- confidence or credible intervals;
- missing-data and sensitivity analysis;
- deviations from the planned method.
Separate statistical significance from practical importance.
A precise trivial effect may not justify implementation, while an uncertain estimate may justify more evidence rather than a claim of no effect.
## Avoid Chance Findings When Slicing Results into Small Groups
Treat subgroup and secondary-metric findings cautiously, especially when they were selected after results were visible.
Every additional comparison increases the chance of an apparently notable result arising by chance.
Before claiming a segment effect:
- define the segment independently of post-treatment behavior;
- confirm adequate sample and overlap;
- test the interaction rather than comparing separate significance labels;
- apply the planned multiplicity control;
- seek replication when the finding drives a consequential decision.
Use exploratory cuts to generate hypotheses, not to rewrite the original test.
Report all material comparisons considered so readers can assess selection risk.
## Decide Whether to Ship the Change and Keep a Record
Combine the experimental estimate with implementation cost, qualitative evidence, operational constraints, ethical considerations, and the likely duration of the effect.
Choose explicitly whether to ship, reject, revise, repeat, or investigate further.
The final record should preserve:
- the original hypothesis and decision rule;
- population, variants, dates, and sample;
- metric definitions and analysis method;
- effect estimates and uncertainty;
- guardrail outcomes and incidents;
- limitations, exclusions, and deviations;
- the decision, owner, and follow-up measure.
Do not claim that one experiment explains why behavior changed or proves the effect will persist in another population or context.
Archive the reproducible analysis and monitor the implemented change so later evidence can confirm, qualify, or reverse the decision.