---
name: ab-testing
description: Design and run statistically valid A/B tests and build a growth experimentation program: hypotheses, sample sizes, variant design, analysis, ICE-prioritized backlogs. Use when the user mentions split tests, experiments, statistical significance, "which version is better," or experiment velocity. Not for generating page-level optimization ideas: use cro. Not for tracking implementation: use analytics. Not for writing variant copy: use copywriting.
license: MIT
metadata:
  author: TechTide AI (Alex Cinovoj)
  provenance: rewritten from patterns in a marketing skills pack (MIT)
  category: Marketing & Growth
---

# A/B Testing

A test without a pre-committed hypothesis, sample size, and primary metric is not an experiment, it is a coin flip with dashboards. Lock the method before traffic flows, then trust it.

## Before designing anything

If a product-marketing context file exists in the repo (`.agents/product-marketing.md` or `.claude/product-marketing.md`), read it first and only ask what it does not answer. Then establish three facts:

1. What are you trying to improve, and what change is on the table?
2. Baseline conversion rate and traffic volume?
3. Constraints: tooling, timeline, engineering budget?

## Test design workflow

1. **Write the hypothesis.** Format: "Because [observation/data], we believe [change] will cause [expected outcome] for [audience]. We'll know when [metric moves X]." No hypothesis, no test.
2. **Pick one variable.** Single meaningful change per test, bold enough to detect. Multivariate only with very high traffic.
3. **Choose metrics.** Primary: one metric, tied to the hypothesis, the metric that calls the test. Secondary: explain how it worked. Guardrails: things that must not get worse (support tickets, refunds); a significantly negative guardrail stops the test.
4. **Calculate sample size before launch.** Quick anchors, per variant, 95% confidence: 3% baseline needs ~47k for a 10% lift, ~12k for 20%, ~2k for 50%; 10% baseline needs ~12k / ~3k / ~550. Full tables, duration math, and calculator links: read references/sample-size.md when traffic or lift falls outside these anchors.
5. **Set allocation.** 50/50 default. 90/10 when a bad variant is expensive. Ramp when there is technical risk. Users must see the same variant on return.
6. **Implement.** Client-side (fast, can flicker) or server-side (no flicker, needs dev work). QA every variant, verify tracking fires before launch.
7. **Run without touching.** Monitor for breakage and guardrails only. No variant edits, no new traffic sources, no early calls.
8. **Analyze.** In order: reached sample size? Statistically significant (p < 0.05, check confidence intervals)? Effect size meaningful against the minimum detectable effect? Secondary metrics consistent? Guardrails clean? Segment differences (mobile/desktop, new/returning)?
9. **Decide and document.** Winner: implement. Loser: keep control, extract the lesson. No difference: more traffic or a bolder variant. Every test gets a written record; template in references/program.md.

## The peeking rule

Checking results early and stopping at the first significant blip manufactures false positives. Pre-commit to the sample size. Look for breakage during the run, not for winners.

### Red flags (verbatim excuses, all invalid)

- "It's already significant at day 3, let's ship it."
- "We're clearly losing, kill it early." (Only guardrails justify early stops.)
- "Let's just add one small tweak to the variant mid-run."
- "We'll extend the test until it turns significant."

Any procedure that ends with a winner declared before the pre-committed sample size is peeking, whatever it is called.

## Running a program, not just tests

One test is a data point; a pipeline is a compounding asset. The loop: generate hypotheses from analytics drop-offs, customer research, competitor moves, support tickets, and past losers; score each 1-10 on Impact, Confidence, Ease and run the highest ICE average first; conclude tests on a fixed cadence; promote winners into a written playbook of reusable patterns. Healthy velocity for most teams: 4-8 launches per month, 20-30% win rate, 20+ hypotheses in the backlog. Cadence details, playbook entry template, and velocity metrics: read references/program.md when setting up or reviewing a program.

## Verification

Before launch, run this check: the hypothesis document contains a falsifiable prediction, a named primary metric, a computed per-variant sample size, and a QA sign-off for each variant. Expect all four in writing. If any is missing, the test does not launch. After analysis, expect the decision record to state sample achieved, p-value or interval, and the decision; a "winner" without those three numbers goes back for analysis.

## Completion checklist

- [ ] Hypothesis in the Because/We believe/We'll know format
- [ ] One variable changed
- [ ] Primary, secondary, and guardrail metrics named
- [ ] Sample size computed and committed before launch
- [ ] Variants QA'd and tracking verified
- [ ] No peeking-based decisions anywhere in the run
- [ ] Result documented with numbers and a reusable lesson

Any box unchecked: not done. Fix or say so.

## Good vs bad

- Bad: "Changing the button color might increase clicks. We'll watch it for a week or two."
- Good: "Because heatmaps show users miss the CTA, we believe a larger high-contrast button will lift CTA click-through 15%+ for new visitors. Baseline 3%, so 12k visitors per variant; we call it at 24k total on click-through to signup start."

## Footguns

- Testing a change too small to detect wastes weeks; the sample size math will warn you, run it.
- Cherry-picking a winning segment after an inconclusive overall result is p-hacking with extra steps. Segments generate the next hypothesis, not this decision.
- Uneven exposure across days of the week biases results. Run whole weeks.
- A sustained win rate above ~40% usually means hypotheses are too timid, not that the team is brilliant.