top of page
6 days ago
16 min read

A/B Testing for Ecommerce Email and SMS: From First Test to Reliable Decisions


By Trayan Hristov · Thrivelia · Published 15 November 2026


Thrivelia ecommerce A/B testing playbook: choose the question, compare fairly and read the result, including contribution and uncertainty.

An A/B test can produce more clicks, a higher conversion rate and a worse commercial result. It can also produce a promising number from too few people to tell you much.


When I plan ecommerce email and SMS tests, I start with the decision. What would we change if B worked? How much improvement would make that change worthwhile? What could get worse?


This guide covers the full process: choosing a question, estimating the audience, setting up campaigns and flows, testing signup forms and offers, measuring SMS incrementality, and deciding what to roll out. The numerical examples are hypothetical. They are teaching examples, not Thrivelia client results.


For the platform controls themselves, keep our Klaviyo A/B testing guide alongside this one. Here, the emphasis is on making a result useful enough to act on.


First, decide which question the test answers


A message test compares two subject lines, layouts or arguments. A journey test compares timing, sequence length or an entire approach to helping someone buy. An incrementality test compares sending a marketing intervention with withholding that intervention.


These answer different questions. A stronger welcome email can beat the old email without proving that the whole welcome programme creates the revenue claimed in an attribution dashboard. Likewise, email plus SMS can outperform email alone without proving that every SMS message earns its cost.


Question

Fair comparison

Useful primary outcome

Does fit guidance improve this email?

Same eligible audience and timing; change the guidance

Purchasers per assigned person within a defined window

Should checkout recovery contain two messages or four?

Random assignment before either sequence starts

Contribution per assigned entrant over the full sequence and follow-up window

Does the popup help the store?

Popup versus no popup among randomly assigned eligible visitors

Store purchase rate or contribution per assigned visitor

Does this SMS layer add value?

Email as usual versus the same email programme plus SMS

Additional contribution per assigned eligible person after messaging costs


I normally start with two versions. Change one factor when the goal is to understand that factor. A larger redesign is also a valid test, but its result applies to the whole package. It will not tell you which individual change caused the difference.


Write the test brief before building the variants


A useful hypothesis names the customer problem and the mechanism. “Try a shorter email” is a production instruction. “First-time footwear buyers hesitate over sizing; putting fit guidance above the product grid should help more of them buy” explains why a change might work.


Here is a brief a strategist, builder and analyst can all use. The targets below are examples to replace with your own baseline, margins and traffic.


Field

Example decision

Decision

Replace the current first welcome email if fit guidance produces a worthwhile improvement

Population

Newly confirmed email subscribers in one market, excluding existing customers

Assignment

One customer identity, assigned once, 50/50, before the first welcome message

A and B

Current email versus the same email with a fit guide above the product grid

Primary metric

Percentage of assigned people making at least one purchase within seven days of assignment

Worthwhile effect

Baseline 2.0%; plan to detect an increase to 2.4%

Guardrails

Contribution per person, refunds, unsubscribes and spam complaints, with limits agreed before launch

Duration

Planned sample plus a minimum calendar span; wait seven days after the final entrant before the main readout

Analysis and owner

Fixed-sample comparison, named analyst, saved cohort export and a dated decision


Record exclusions before launch. A staff account, bot or existing customer should not disappear from one arm after you see the outcome. Save the content, offers, eligibility rules and screenshots so the next person can reconstruct the test.


Choose the metric that matches the decision


Open rate can diagnose a subject-line result, but privacy-related opens make it a poor stand-in for buying intent. Clicks help explain what happened. Neither proves that an email improved purchases or profit.


For a conversion question, count unique purchasers divided by assigned eligible people. Specify whether repeat orders count as additional orders or whether the outcome is simply purchased versus did not purchase. Those are different metrics with different analysis requirements.


For an offer, cadence or channel question, I prefer a contribution measure when reliable cost data is available. Start with revenue after discounts, subtract refunds and agreed variable costs, then subtract incremental channel costs. Keep tax, shipping revenue, returns and currency treatment consistent. Do not subtract a discount twice if it is already reflected in paid revenue.


Calculate contribution per assigned person, including zero-purchase people. Aggregating customer outcomes this way keeps the commercial question connected to the people who could have been affected.


Use channel attribution as a diagnostic view. For causal comparisons, compare store outcomes for the randomised groups over the same follow-up window, even when an order is attributed to another channel. An SMS click can change where revenue is credited without creating an extra purchase.


Guardrails protect against a narrow win. Agree what deterioration in contribution, complaints, opt-outs, refund rate or customer experience would make you reject a variant. Rare events need careful interpretation: zero complaints in a small sample does not establish that complaint risk is zero.


Estimate the sample before choosing the deadline


Minimum detectable effect means the smallest improvement you want the test to have a reasonable chance of detecting. It should reflect a useful business change. Choosing an enormous effect just to make a calculator return a smaller audience does not make a subtle improvement easier to measure.


The approximate planning figures below use two independent groups, equal allocation, a binary purchase outcome, a two-sided 5% significance level and 80% power. They use the standard normal approximation for two proportions, rounded for planning. The statsmodels two-proportion power documentation describes the underlying approach.


Baseline to target

Absolute and relative improvement

Approximate people per version

2.0% to 2.4%

+0.4 percentage points; +20% relative

21,100

2.0% to 3.0%

+1.0 percentage point; +50% relative

3,830

5.0% to 6.0%

+1.0 percentage point; +20% relative

8,160


These are examples, not universal minimums. Different allocation, repeated observations, clustering or a different statistical method changes the calculation. Revenue and contribution need a power calculation based on their variability; a purchase-rate calculation cannot establish that a profit test is adequately powered. The Experimentation Platform power calculator distinguishes binary and continuous outcomes.


At 2,000 new eligible people a day across both arms, roughly 42,200 assignments take about 22 days to accrue. A seven-day outcome window then requires another seven days for the last entrants. At 300 eligible people a day, the same sample takes about 141 days to accrue. A seven-day follow-up takes that to about 148 days.


Those are idealised estimates. They assume stable eligibility and traffic. A test spanning several promotions, seasons or stock changes may no longer answer the original question cleanly. Set both the sample target and a sensible calendar limit before launch. If the target is infeasible, change the question or accept a less precise result.


Read the uncertainty around the lift


Suppose A produces 40 purchasers from 2,000 people and B produces 48 from 2,000. Purchase rate rises from 2.0% to 2.4%. That is a 20% relative lift, but only a 0.4 percentage-point absolute difference.


Under a simple fixed-sample normal approximation, the 95% confidence interval for that difference runs from approximately -0.51 to +1.31 percentage points. The data are compatible with a loss as well as a worthwhile gain. Calling B a proven winner would overstate the evidence.


Illustrative purchase-rate difference of 0.4 percentage points: the approximate 95% interval is -0.51 to +1.31 points with 2,000 people per arm, and +0.13 to +0.67 with 22,000 per arm.

With 440 versus 528 purchasers from 22,000 people per version, the observed rates and relative lift are identical. The approximate interval is much narrower: +0.13 to +0.67 percentage points. This illustrates why the same headline lift can support different decisions.


The chart uses the normal-approximation formula documented by NIST for differences between proportions. It is illustrative; other interval methods can behave better, especially with sparse outcomes. It is not a Klaviyo or Alia probability-to-win calculation. A 95% frequentist confidence interval is also not a statement that this particular interval has a 95% probability of containing the true effect.


Statistical evidence still needs a commercial interpretation. A reliable but tiny gain may cost more to maintain than it earns. A promising result with a wide interval may justify another test. An inconclusive result does not establish equivalence or prove there is no effect.


Set up a campaign test in Klaviyo


1. Define the eligible audience and exclusions. Keep the offer, send timing, destination and non-tested creative consistent unless one of those is the treatment. Name the campaign and variants so the test can be found later.


2. Build the control and create the A/B test. Set the allocation, winning metric and test duration deliberately. Klaviyo supports open rate, click rate and placed-order rate for standard campaign tests, subject to the campaign and account setup. Placed-order rate is not available for personalised variant selection.


3. Decide whether the purpose is learning or choosing a version for the remaining audience. A 100% audience split gives everyone a test version. A smaller test can send the selected version to the remainder later. Keep that later rollout separate from the initial randomised comparison. Klaviyo send-time tests use the full audience and do not send a later winning remainder.


These controls are described in Klaviyo’s campaign A/B testing instructions. Check the live settings before launch because available options can change.


4. Send previews to mobile and desktop. Check links, tracked destinations, coupons, personalisation fallbacks, images and the final purchase event. Confirm both variants can be delivered under the same eligibility rules.


5. Save the initial cohort and read the result after the chosen outcome window. If the native dashboard cannot answer the brief’s business metric, export the cohort and analyse purchases against assignment. Retain delivery-based results as diagnostics alongside the assigned-person result.


Klaviyo’s campaign results distinguish statistical significance from other outcomes, and a version can be selected without a significant difference. Read the campaign significance explanation before interpreting a winner badge. Its probability-to-win is not interchangeable with the fixed-sample confidence interval above.


Test flow messages and whole journeys separately


For one message, keep the rest of the flow identical. For a journey test, assign people before the first point at which their experience differs. Otherwise, the experiment may accidentally exclude people affected by an earlier message.


Klaviyo’s native email variation test selects winners using opens or clicks. Automatic distribution can shift traffic, and automatic ending can use significance, a date, or whichever comes first. Publishing a test on a draft email also makes that email live. Review the whole flow before publishing, as explained in Klaviyo’s flow email testing instructions.


That interface is useful for message optimisation. It does not establish that a winning click variant improves contribution. Save the selection metric and review downstream purchases separately.


For the fixed-sample analysis illustrated here, keep allocation fixed and follow the pre-agreed ending rule. If you enable adaptive distribution, use an analysis suited to that design. Do not treat a pooled, changing allocation as though it were the same fixed 50/50 experiment.


To compare two complete paths, Klaviyo documents a conditional split using a random sample. Add the split before the paths diverge, allocate the sample, then build each sequence. Keep shared eligibility and purchase-exit rules aligned. A path that keeps emailing buyers after purchase is testing a different experience as well as a different cadence.


For example, compare the current checkout sequence with a sequence that leads with delivery clarity and help before presenting an offer. Follow every assigned entrant for the same pre-agreed window. Compare unique purchasers, contribution and opt-outs across the whole path. Do not add each message’s unique recipients together: the same person may receive several messages.


For a lasting programme holdout, verify that a customer keeps the same assignment across re-entry. A fresh random split on every entry can expose the same customer to both experiences. Use a stable cohort or assignment property where needed and test that implementation before launch. Do not assume a branch label alone guarantees persistent customer-level assignment.


Test popups on the people assigned to the experience


Signup rate among popup viewers is useful, but it can conceal what happens to the whole eligible audience. A trigger change alters who sees the form. A form can look stronger among viewers while collecting fewer confirmed subscribers overall.


Consider a hypothetical test with 10,000 assigned visitors per arm. A shows to 4,000 visitors and generates 400 confirmed subscribers. B shows to 2,000 and generates 300. B wins on subscribers per viewer: 15% versus 10%. A wins on subscribers per assigned visitor: 4% versus 3%.


Choose the denominator before looking at the result. For an offer change, also follow store purchases and contribution. For a popup-versus-no-popup test, the control has no popup signups by design; judging it on popup-attributed revenue would miss the point.


In Alia, sitewide metrics cover assigned visitors, while attributed metrics focus on people who sign up and use different timing anchors. The distinction is explained in Alia’s A/B result guide. Compare equivalent windows and use per-visitor measures when allocations differ.


For a no-popup control, follow Alia’s popup-versus-no-popup setup. Check that the control’s triggers are removed and the test genuinely assigns traffic to both versions. Test mobile close controls, returning-visitor suppression and coupon delivery before sending traffic.


If the question is confirmed list growth, verify that subscribers reach the correct destination with valid consent and usable source data. A higher front-end submission rate is not enough when sync failures or duplicates prevent the welcome flow from working. Our Alia integration guide covers that handoff.


Work through an offer test beyond the conversion rate


Here is a hypothetical offer test. Each version receives 10,000 eligible people. The stronger offer produces more buyers and slightly more retained revenue. It also carries higher product, fulfilment and messaging costs.


Outcome

A: current offer

B: stronger offer

Assigned people

10,000

10,000

Purchasers

200

240

Purchase rate

2.0%

2.4%

Revenue after discounts and refunds

$15,000

$15,400

Agreed variable and channel costs

$9,000

$10,800

Contribution

$6,000

$4,600

Contribution per assigned person

$0.60

$0.46


The observed purchase-rate lift is 20% relative. Contribution per person is about 23% lower. If the decision concerns immediate contribution, the stronger offer has moved the wrong way in this sample.


That is an accounting comparison, not a statistical significance claim. To quantify uncertainty in contribution, retain customer-level outcomes, including zeros, and use an analysis appropriate for their distribution and assignment unit. A few unusually large orders can move the mean substantially.


If you expect the offer to improve later retention, write that hypothesis and its follow-up window into the brief before launch. Track the original groups through repeat orders, returns and offer costs. Do not invent a future lifetime-value benefit after seeing that first-order profit declined.


Measure whether SMS adds value to email


Comparing SMS subscribers with email-only subscribers is not a randomised test. Those groups may differ in intent, purchase history and willingness to engage before any message is sent.


Start with people eligible and appropriately consented for the channels being tested. Randomly assign them to the normal email programme or that same programme plus the defined SMS layer. Keep the email treatment and other marketing rules consistent. The control withholds the optional SMS intervention being studied, while transactional messages and promised service continue.


Use stable customer assignments and enforce them across every campaign and flow within the test’s scope. If the question concerns promotional SMS campaigns only, say so. If flows still send SMS to both groups, the experiment cannot estimate the effect of removing all SMS.


Compare total store contribution per assigned person over a defined period, with SMS costs included. Check purchases, order value, opt-outs and substitution between channels. An increase in SMS-attributed revenue alongside a decline in email-attributed revenue may be credit shifting rather than additional business.


For a published industry example, Postscript’s Jones Road experiment describes subscriber groups receiving normal, zero and increased SMS campaign cadence. The useful lesson is the controlled comparison and its scope, rather than adopting another brand’s outcome as your forecast.


Protect the experiment from misleading results


Keep allocation and identity clean. A 50/50 plan will not produce exactly equal counts, but an unexplained large mismatch is a warning. Check assignments, event loss, bots, exclusions and implementation before interpreting the lift. The Microsoft researchers’ Experimentation Platform research collection includes work on sample ratio mismatch and other experiment failures.


For website tests, document whether assignment identifies a browser, visitor or known customer. Cookie deletion and cross-device visits can create repeat assignments. For lifecycle tests, check profile merging and re-entry. Count and analyse at the unit actually randomised, with appropriate handling when observations are clustered.


Keep customers in their assigned groups for the primary analysis. Comparing only clickers, subscribers, redeemers or delivered recipients can select different types of people after the treatment. Those cuts can explain behaviour, but they do not automatically retain the original causal comparison.


Do not stop a conventional fixed-sample test the first time it looks significant. Repeatedly checking and ending on a favourable result changes its false-positive behaviour. A pre-planned sequential design can support interim decisions, but the stopping rule and analysis must match. The UBC statistics guide to peeking in A/B tests explains the distinction.


You should still monitor broken links, missing discounts, delivery failures and customer harm. Stop a broken treatment when needed. Record why it stopped and avoid presenting an operationally interrupted test as a clean efficacy result.


Wait for equal outcome maturity. If purchases count for seven days after assignment, yesterday’s entrants cannot be compared with people who have had the full week. Set the final extraction date accordingly. Refund or retention questions need longer, explicitly chosen observation windows.


Limit the number of claims. Testing ten variants, twenty audience cuts and several metrics creates many chances to find an attractive number. Choose a primary comparison in advance. Label unplanned segment findings as exploratory and confirm them with fresh data. For several planned comparisons, use an analysis that accounts for multiplicity.


Document overlap. A simultaneous site redesign, new discount or SMS cadence test can interact with the email treatment. Avoid conflicting tests or design the randomisation and analysis to handle them. A before-versus-after revenue chart alone cannot separate the test from seasonality or other changes.


Choose stronger tests for your vertical


I prioritise tests that remove a known buying obstacle before polishing low-impact creative details. Customer support conversations, reviews, returns and search behaviour often give better hypotheses than an empty brainstorming document.


Vertical

Useful first question

Advanced extension

Fashion and footwear

Does fit guidance improve first purchases?

Follow size-related returns and contribution, then test the second-order journey separately

Beauty and skincare

Does a routine explanation outperform a product grid?

Test a replenishment window based on observed reorder timing; measure repeats and opt-outs

Food and consumables

Does usage guidance help customers finish and reorder?

Compare replenishment timing while tracking subscription substitution and margin

Subscription

Does onboarding improve activation or first renewal?

Use original assigned cohorts to measure retention and contribution over later renewals

High-ticket products

Does delivery, financing or comparison guidance reduce hesitation?

Choose a longer purchase window; track assisted sales consistently across both groups


For a larger team, a 2-by-2 factorial design can test two factors, such as fit guidance and an offer, in four randomised groups. It can investigate whether the combination behaves differently from the individual changes. That interaction needs its own power and analysis plan. Four cells do not automatically reveal an interaction from a small audience.


For most smaller programmes, a well-run two-arm test with a clear decision is more useful than several overlapping experiments nobody can finish.


What to do when traffic is too low


Low volume changes what you can learn. It does not make weak evidence stronger. At a 2% baseline purchase rate, hundreds of recipients per arm will usually struggle to distinguish modest commercial improvements.


Test a larger, evidence-backed change. A complete sizing explanation versus no sizing help can be more informative than changing a button from one near-identical phrase to another. You give up the ability to attribute the result to a tiny element, but you may answer a more useful question.


Use interviews, support tickets, usability checks and message QA to remove obvious friction. You do not need an A/B test to repair a broken coupon or a hidden mobile close button. Choose leading metrics when they fit the immediate question, then describe them honestly: a click improvement is evidence about clicks.


If you pool repeated tests, plan that approach before the results arrive. Keep treatment definitions and outcome windows consistent and account for repeat customers and campaign differences. Simply adding recipients across unrelated sends can produce a precise-looking answer to an unclear question.


Set a maximum duration and record an inconclusive result when necessary. Keep the current version or make a documented business judgement based on cost and customer experience. Do not turn that judgement into a claim of proven uplift.


Turn the readout into a useful operating decision


The readout should fit on one page, with the underlying export available. Include the question, assignment unit, exclusions, dates, allocation, outcome window, sample counts, absolute effect, relative effect, uncertainty, costs and guardrail results. Record any interrupted sends, tracking changes or deviations from the brief.


Write the decision in plain language. “Roll out fit guidance to this welcome audience because the purchase improvement clears our worthwhile threshold and the contribution guardrail is acceptable” is actionable. “B won” leaves the next person guessing.


Use four outcomes: roll out, retain the control, continue under the original plan, or close as inconclusive. A rollout can be staged when the change affects a large audience. Monitor delivery, complaints, margin and customer experience, and keep a reversal path. Revisit the result when audience, offer or season changes materially.


I keep a test log that captures losses and inconclusive results alongside wins. The strategist owns the hypothesis, the builder owns implementation and QA, the analyst owns the readout, and the CMO or founder owns the commercial decision. On a small team, one person can hold several roles, but each responsibility still needs a name.


Review active tests weekly for data quality and readiness. Decide on commercial results when the agreed sample and outcome window are complete. That keeps testing connected to the monthly lifecycle operating plan, rather than becoming a stream of screenshots with no accumulated learning.


Frequently asked questions


What should I A/B test first in ecommerce email marketing?


Start with a high-volume customer problem and a decision you can act on. Examples include fit uncertainty in welcome emails, delivery questions in checkout recovery or unclear product usage after purchase. Prioritise likely business impact, eligible audience and the cost of implementation before small cosmetic changes.


How long should an email A/B test run?


Set the sample target, calendar coverage and outcome window before launch. A campaign also needs enough time for the chosen purchase window to mature. A continuously enrolling flow needs follow-up after its final entrant. There is no universal number of days that makes every test reliable.


Is a 20% conversion lift enough to declare a winner?


The percentage alone is insufficient. A rise from 2.0% to 2.4% is a 20% relative lift, but its reliability depends on the sample and design. Check uncertainty, absolute value, contribution and guardrails before deciding. The worked examples in this guide show the same observed lift with very different precision.


Can I test email and SMS together?


Yes. Define the combined treatment and randomly assign people eligible for both channels. For example, compare the same email programme with and without a specified SMS layer. Keep assignment stable and compare total store outcomes after messaging costs. Subscriber groups chosen by channel preference alone are not equivalent randomised groups.


Should I choose the variant with the highest revenue?


Check revenue per assigned person, uncertainty and the costs behind it. Higher revenue can accompany deeper discounts, more returns or higher messaging costs. Choose the metric that matches the decision. If the aim is contribution, a revenue winner may still be the wrong commercial choice.


Can AI choose and interpret my A/B tests?


AI can organise customer feedback, draft hypotheses, generate variants and help write a readout. Give it the brief and anonymised aggregate data. Verify its calculations and platform instructions. Do not let it invent significance from summary revenue, treat attribution as incrementality or change the primary metric after seeing the result.


Build testing into the account


An email account should have a testing backlog, reliable event tracking, a shared results log and someone accountable for acting on the findings. Our Klaviyo setup guide covers the foundation. Our Hiro Analytics setup guide explains how to keep reporting definitions consistent.


If you want help deciding which tests deserve your traffic and implementing them across email, SMS and signup forms, apply to work with us.


 
 
 

Recent Posts

See All

Comments


Commenting on this post isn't available anymore. Contact the site owner for more info.
bottom of page