ab-testing

A/B Testing

A test without a pre-committed hypothesis, sample size, and primary metric is not an experiment, it is a coin flip with dashboards. Lock the method before traffic flows, then trust it.

Before designing anything

If a product-marketing context file exists in the repo (.agents/product-marketing.md or .claude/product-marketing.md), read it first and only ask what it does not answer. Then establish three facts:

1. What are you trying to improve, and what change is on the table?
2. Baseline conversion rate and traffic volume?
3. Constraints: tooling, timeline, engineering budget?

Test design workflow

1. Write the hypothesis. Format: "Because [observation/data], we believe [change] will cause [expected outcome] for [audience]. We'll know when [metric moves X]." No hypothesis, no test.
2. Pick one variable. Single meaningful change per test, bold enough to detect. Multivariate only with very high traffic.
3. Choose metrics. Primary: one metric, tied to the hypothesis, the metric that calls the test. Secondary: explain how it worked. Guardrails: things that must not get worse (support tickets, refunds); a significantly negative guardrail stops the test.
4. Calculate sample size before launch. Quick anchors, per variant, 95% confidence: 3% baseline needs ~47k for a 10% lift, ~12k for 20%, ~2k for 50%; 10% baseline needs ~12k / ~3k / ~550. Full tables, duration math, and calculator links: read references/sample-size.md when traffic or lift falls outside these anchors.
5. Set allocation. 50/50 default. 90/10 when a bad variant is expensive. Ramp when there is technical risk. Users must see the same variant on return.
6. Implement. Client-side (fast, can flicker) or server-side (no flicker, needs dev work). QA every variant, verify tracking fires before launch.
7. Run without touching. Monitor for breakage and guardrails only. No variant edits, no new traffic sources, no early calls.
8. Analyze. In order: reached sample size? Statistically significant (p < 0.05, check confidence intervals)? Effect size meaningful against the minimum detectable effect? Secondary metrics consistent? Guardrails clean? Segment differences (mobile/desktop, new/returning)?
9. Decide and document. Winner: implement. Loser: keep control, extract the lesson. No difference: more traffic or a bolder variant. Every test gets a written record; template in references/program.md.

The peeking rule

Checking results early and stopping at the first significant blip manufactures false positives. Pre-commit to the sample size. Look for breakage during the run, not for winners.

Red flags (verbatim excuses, all invalid)

• "It's already significant at day 3, let's ship it."
• "We're clearly losing, kill it early." (Only guardrails justify early stops.)
• "Let's just add one small tweak to the variant mid-run."
• "We'll extend the test until it turns significant."

Any procedure that ends with a winner declared before the pre-committed sample size is peeking, whatever it is called.

Running a program, not just tests

One test is a data point; a pipeline is a compounding asset. The loop: generate hypotheses from analytics drop-offs, customer research, competitor moves, support tickets, and past losers; score each 1-10 on Impact, Confidence, Ease and run the highest ICE average first; conclude tests on a fixed cadence; promote winners into a written playbook of reusable patterns. Healthy velocity for most teams: 4-8 launches per month, 20-30% win rate, 20+ hypotheses in the backlog. Cadence details, playbook entry template, and velocity metrics: read references/program.md when setting up or reviewing a program.

Verification

Before launch, run this check: the hypothesis document contains a falsifiable prediction, a named primary metric, a computed per-variant sample size, and a QA sign-off for each variant. Expect all four in writing. If any is missing, the test does not launch. After analysis, expect the decision record to state sample achieved, p-value or interval, and the decision; a "winner" without those three numbers goes back for analysis.

Completion checklist

• [ ] Hypothesis in the Because/We believe/We'll know format
• [ ] One variable changed
• [ ] Primary, secondary, and guardrail metrics named
• [ ] Sample size computed and committed before launch
• [ ] Variants QA'd and tracking verified
• [ ] No peeking-based decisions anywhere in the run
• [ ] Result documented with numbers and a reusable lesson

Any box unchecked: not done. Fix or say so.

Good vs bad

• Bad: "Changing the button color might increase clicks. We'll watch it for a week or two."
• Good: "Because heatmaps show users miss the CTA, we believe a larger high-contrast button will lift CTA click-through 15%+ for new visitors. Baseline 3%, so 12k visitors per variant; we call it at 24k total on click-through to signup start."

Footguns

• Testing a change too small to detect wastes weeks; the sample size math will warn you, run it.
• Cherry-picking a winning segment after an inconclusive overall result is p-hacking with extra steps. Segments generate the next hypothesis, not this decision.
• Uneven exposure across days of the week biases results. Run whole weeks.
• A sustained win rate above ~40% usually means hypotheses are too timid, not that the team is brilliant.