Email A/B Testing: What to Test and How to Act on Results

Most email A/B testing is done wrong: too many variables at once, insufficient sample sizes, and no clear protocol for acting on results. Here's the systematic approach that actually improves reply rates and open rates over time.

A/B Testing Fundamentals

Two rules that most teams violate: test one variable at a time, and run tests until statistical significance is reached. Testing subject line AND body copy simultaneously produces uninterpretable results. Calling a test after 20 sends produces unreliable conclusions. Discipline here is the difference between learning and noise.

The Testing Priority Order

1. Subject line (highest leverage — determines open rate)
2. Opening line (determines whether they keep reading)
3. Email length (3 sentences vs 6 sentences)
4. CTA format (question vs calendar link vs soft ask)
5. Send time (Tuesday 9am vs Thursday 2pm)
6. Body copy angle (pain-led vs proof-led vs peer-referenced)

Statistical Significance in Outbound

For outbound email sequences, aim for 100+ sends per variant before evaluating. At typical reply rates (5-15%), you need this volume to distinguish signal from noise. With 50 sends per variant, a 2% difference in reply rate could easily be random.

Acting on Results

When a variant wins: document it, update your master template, and retire the losing variant. Don't continue running both "to see if it continues." The winner is your new control. Then test the next variable. This compound improvement approach — consistent 10-15% gains from each test — produces dramatic performance improvement over 6-12 months.

The outbound teams that consistently hit 15%+ reply rates got there through systematic testing and iteration — not a single great template. Build the testing discipline and reply rates compound month over month.