How to A/B test marketing campaigns
What you'll learn
- →Why most small businesses A/B test wrong
- →Sample size 101 (calculators included)
- →What to test (and what not to)
- →Reading results without fooling yourself
Before you start
- □A list or audience large enough for splits
- □A testing tool or manual split
Most A/B tests at small scale are noise. Done right, they're the closest thing to free intelligence. Here's how to do them honestly.
The steps
- Step 01
Pick one element to test
Subject line OR CTA OR image — never multiple at once. Multivariate needs huge samples.
- Step 02
Calculate sample size first
Use a free calculator (Evan Miller, Optimizely). Most subject-line tests need 1,000+ per arm to detect a 10% lift.
- Step 03
Run for a full cycle
Email tests: minimum 24 hours. Ad tests: at least 7 days to account for day-of-week effects.
- Step 04
Don't peek and stop early
Early 'winners' often regress. Wait for the calculated sample.
- Step 05
Calculate statistical significance
Aim for 95% confidence. Below that = inconclusive, not 'winner'.
- Step 06
Implement winners and retest
Test → implement → re-baseline → test again. Compounding lifts beat one-off wins.
- Step 07
Log every test
What, when, result. After 12 months you'll have a playbook.
Common questions
+How small a list can A/B test?
Below 500 per arm, results are usually noise. Test bigger things less often.
+Best free calculator?
Evan Miller's A/B test calculator.
+What if I have no significant result?
Often correct — they're equal. Pick the simpler version and move on.
+Can I test pricing?
Carefully — and never on customers who'd compare prices.
+How long between tests?
Run one at a time. Overlapping tests confound each other.
What to do next
Social Perks turns happy customers into a marketing team. Free 14-day trial. No card required.
Start freeMore guides in Analytics & growth
A spreadsheet model for tracking marketing dollars in and revenue out — without enterprise tooling.
Three layered methods that together capture 80%+ of true source attribution.
Three ways to calculate LTV, when to use each, and how to use it to set your max CAC.
Move past 'increase awareness' to SMART goals tied to revenue, with a quarterly review rhythm.