Updated: 6 days ago
Klaviyo A/B Testing: Choose a Useful Question and Read the Result Properly
By Trayan Hristov · Thrivelia · Updated 15 November 2026

Klaviyo A/B testing helps you compare alternative messages or send times. This guide focuses on email campaigns and flow emails. The useful starting point is a decision: what will your team change if one version performs better? Choose the audience, the changed element and the result that matters before building variations.
Campaigns and flows have different testing controls. A recorded winner, statistical significance and a worthwhile commercial result also tell you different things. Read them together before turning a test into a permanent rule.
Start with a customer problem you can act on
“Test more subject lines” is an activity. “Help first-time shoppers understand which size to choose” gives the experiment a purpose.
Find the uncertainty in the journey. Are customers struggling to compare products, understand delivery or feel confident about using what they bought? Use customer questions, service conversations and journey data to choose a credible alternative.
Change one factor for a clear comparison. If the subject line, offer, products and layout all change, the result can compare two complete executions, but it cannot tell you which individual change mattered. Keep the claim at the level the experiment supports.
Check that both versions are suitable to send. Missing products, broken links or an incorrect offer are implementation faults to fix before testing.
Write the test brief before opening the editor
Use a short shared record so the result leads to a decision.
Brief item | Write down | Example |
|---|---|---|
Decision | What could change afterwards? | Keep or remove a fit-help module |
Audience | Who is eligible? | Permitted non-buyers in one welcome message |
Hypothesis | Why might the change help? | Fit reassurance could reduce purchase hesitation |
Changed factor | What differs between versions? | Presence of the fit-help module |
Measurement | Primary result and guardrails | Native test metric, first orders and complaints |
Review | When and how will you decide? | Agreed review with clear and inconclusive outcomes |
Define the observation period around the decision and buying cycle. Avoid choosing a short window because it produces the most flattering result. Record stock, promotions and other journey changes that could affect interpretation.
Set up campaign tests around the available audience
Klaviyo's campaign testing guide covers content and send-time tests. For a content test, create the campaign, choose the eligible audience, build the first version and select Create A/B test. Name the variations clearly and change the planned factor.
Choose the winning metric deliberately. Campaign options include open rate, click rate and placed order rate. Placed order rate requires a Placed Order metric and is unavailable for personalized variations.
For a standard content test using part of the audience, Klaviyo tests the variations and sends the selected winner to the remainder. A test using 100% leaves nobody for that later send. Send-time tests always use 100%, so their learning informs later campaigns.
Account for the buying delay when selecting a test duration. A six-hour readout may suit some decisions and miss others. Check the actual settings rather than copying a universal percentage or duration.
Create a draft test with Composer, then inspect its settings
Klaviyo announced campaign A/B testing in Composer on 4 November 2026. Its campaign guide describes content tests for new email and SMS messages, with defaults of a 20% test audience and a click-based winner. This workflow does not apply to push. Use the campaign editor to change settings, test an existing message or arrange a send-time test.
Ask for the test explicitly, then inspect every saved variation. Confirm the planned factor changed and the controls stayed consistent. The default winning metric is an operating setting, not a judgement that clicks are the right commercial outcome for your question.
The Printfresh case provides a useful contrast: its reported first-email comparison produced more revenue and higher conversion with a slightly lower click rate. That is a qualified historical message comparison, not a guarantee about a new Composer test or proof of statistical significance.
Treat flow testing as a separate setup
Create variations inside the selected flow email. Flow winner metrics are click rate, the default, or open rate. Placed order rate is unavailable.
Set recipient weights, or enable automatic distribution: it starts equally and can shift towards better-performing versions while testing.
Configure ending separately. Automatic options are significance and a date limit; selecting both ends at whichever comes first. For manual review, clear both options before publishing and choose a winner yourself later.
Record the distribution and ending choices. Publishing a test on a draft email also makes it Live. Complete the surrounding journey checks using our Klaviyo flow build guide before releasing it.
Read the result beyond the winner label
Klaviyo's campaign significance guidance uses delivery counts and win probability. A variation can be selected as the winner without a statistically significant result. Check the status and underlying counts before describing the finding as reliable evidence.
Open-rate tests include Apple Mail Privacy Protection opens. A significant open-rate result therefore does not establish that people deliberately opened the message. Choose a metric that fits the question and acknowledge its limitations.
The flow significance criteria are different. They currently require more than 500 deliveries per variation, at least 10 winning-metric actions across variations, greater than 90% win probability and less than 10% overlap between the leading variation's likely performance range and each other variation's range. A date-ended flow test can select the highest observed rate before those criteria are met.
Those are platform classification rules, not a universal sample-size recommendation. Review how many meaningful outcomes occurred and whether the plausible difference would change your decision. An inconclusive test leaves uncertainty; it does not prove the versions equivalent.
A fictional footwear example: useful clicks or more buyers?
Imagine a footwear brand testing its second welcome email. Customers regularly ask about fit. Version A contains the existing product grid. Version B adds a short fit-help module; the products, subject line, offer and other content stay the same.
The hypothesis is that answering a real hesitation helps more subscribers reach a first purchase. The team chooses click rate for the native flow test and manual winner selection, then reviews purchasing and contribution over its agreed window.
This matters because the new guide link could attract more clicks without improving the purchase decision. Klaviyo's flow results documentation explains the conversion information available alongside the test metrics. That reporting does not make conversions an available automatic flow winner metric.
If Version B leads on clicks, investigate where people went and what happened next. Check orders, customer questions, unsubscribes and contribution using consistent definitions. If buying evidence is too sparse, keep the uncertainty visible and decide whether further testing is worth the audience and time. This example describes a decision process, not a claimed result.
Include the cost of the version that wins
A specialised version can improve a purchase metric while adding more production, approval and maintenance cost than the observed benefit. Keep the standard treatment competent, randomise within the relevant audience and compare contribution per assigned person before subtracting the extra execution cost.
The email personalisation ROI guide includes a fictional worked example and editable calculator. Its economic break-even threshold is separate from statistical confidence. Do not adopt a permanent multi-version workflow solely because a platform labels one variant a winner.
Turn the finding into an operating decision
Save the hypothesis, dates, allocation, winning metric, sample, result and chosen action. Record whether the winner was selected manually, by a deadline or at significance. Keep the original versions so the team can understand what changed.
Apply the finding to comparable situations. A result from first-time shoppers during a launch needs another review before becoming a rule for established customers or a different season.
A test between two messages answers which treatment performed better under the tested conditions. Measuring the effect of sending versus sending nothing requires a suitable control; our attributed versus incremental revenue guide covers that separate question. Our perfectwhitetee flow-efficiency review also explains the limits of account-level before-and-after evidence.
Go deeper on sample size, holdouts and commercial decisions
The platform settings answer how to run a test. The next question is whether the experiment has enough eligible people, a useful outcome and a result you can act on.
Our ecommerce email and SMS A/B testing playbook includes a complete test brief, sample-size calculations and an uncertainty chart. It also covers whole-flow comparisons, Alia popup controls, offer contribution, SMS holdouts and examples across five verticals.
Use it when planning a test that needs to answer a business question beyond the native winner metric. The worked examples distinguish a higher purchase rate from higher contribution, and a promising observed lift from a sufficiently precise result.
Frequently asked questions
How long should a Klaviyo A/B test run?
Plan around audience volume, the chosen metric and the time customers need to act. Inspect the statistical status and business context at the agreed review. A completed test or reached deadline does not guarantee a decisive result.
Can a flow automatically choose the version with the most orders?
Native flow email winner selection supports open or click rate. Review available conversion reporting separately and use a deliberate manual decision when commercial outcomes need to govern the choice.
What should a small audience test first?
Choose one consequential uncertainty and two credible versions. Avoid spreading a limited audience across many minor changes. When purchase outcomes are sparse, combine the experiment with customer feedback and journey diagnosis while keeping the result qualified.
Thrivelia's Klaviyo agency support and lifecycle marketing services connect test planning, creative, implementation and measurement to the customer transition your programme needs to improve.

.png)
Comments