Before you start: turn on both embeds
A/B testing only works once both Whatmore embeds are active in your theme. Open Configure A/B test and you'll see the one-time embed setup at the top.
There are two embeds, and you need both switched on:
Whatmore AB Test — splits visitors into your groups.
Whatmore — tracks sessions so we can measure what each group does.
Click Set up embeds in theme to open your theme editor and enable them. This is a one-time step — once both are live, every test you run is measured automatically.
💡 If either embed is off, visitors won't be split or measured, and your groups will show no data.
Split your traffic into groups
In Configure A/B test → Split your traffic into groups, you decide how visitors are divided. Every visitor is randomly placed into one group and stays in it for the whole test, so the comparison stays clean.
A few rules make this simple:
Set a share (%) for each group. The coloured bar shows the split at a glance.
Default automatically takes whatever's left, so your split always totals 100% — you never have to do the maths.
Use Distribute evenly to spread traffic equally across all groups in one click.
You can run as many groups as you need — a simple two-way test, or several variants plus a holdout side by side.
The three kinds of groups
Default (baseline) — sees all your live widgets. This is the benchmark every other group is compared against. Widgets you take un-live simply drop out on their own — we never touch your group lists.
Variant — sees a specific set of widgets you want to test (e.g. only a Carousel, or Stories + Spotlight). This is the change you're measuring.
Holdout — sees no widgets at all.
What is a holdout?
A holdout is a small group that gets none of your Whatmore widgets — a true control. Because they see nothing, comparing every other group against the holdout reveals the true lift your widgets add over showing nothing at all. It answers the question: "Are my videos actually driving extra sales, or would those sales have happened anyway?"
A holdout is optional. Add one from the Add holdout option in the groups row when you want that clean before/after read.
💡 Keep holdouts small (e.g. 10%). You only need enough traffic to read the control — you don't want to withhold widgets from a large share of shoppers.
Read the performance funnel
Each group card shows a live funnel of three tiers. Each tier is a subset of the one above it — fewer people, but deeper engagement.
All visitors — Everyone in the group, measured before they interact with a widget.
Saw the widget — Visitors who actually saw a widget.
Watched a video — Visitors who clicked and watched a video.
For each tier you'll see:
Sessions (or Users) — how many reached this tier.
Sales — revenue from purchases in this tier. On Watched a video, this is the revenue driven by video-watching sessions.
CVR (conversion rate) — the share of this tier that made a purchase. This is the headline number on the right.
Lift vs baseline — how this group's CVR compares to Default. Green means it's converting better, red means worse. Default itself reads "baseline."
Avg watch time — average seconds of video watched, shown beneath Watched a video.
Sessions vs Users
Use the Sessions / Users toggle at the top to switch how people are counted:
Sessions — every visit counts (one shopper who returns twice = two sessions).
Users — repeat visits from the same person roll up into one, so you're counting unique people.
Sort and compare
Use Sort cards by [metric] measured at [stage] to rank your groups — for example, sort by CVR at the Watched a video stage to see which group converts best among engaged viewers.
Versioning: every edit is a new version
Your test will evolve — you'll add groups, rebalance the split, or change which widgets a group shows. Whatmore handles this with versioning.
Here's how it works:
Every time you save a change, we create a new version (v18 → v19 → v20…) and start tracking from zero for the new setup.
Older versions stay archived, so you can always look back and compare what worked.
The Configure A/B test modal tells you which version your next save will create (e.g. "Will become v21") and reminds you that saving resets analytics from now.
This means each version's numbers are clean — they only reflect the setup that was live during that version, never a mix.
💡 Because saving starts a fresh count, give each version enough time to gather meaningful traffic before you judge it.
View a version and make a decision
Open any version from the Versioning list to see exactly how that setup performed. Archived versions show the same cards and funnel you see live.
This is where you decide what to keep:
A group is only marked the winner when it beats the Default baseline by a statistically significant margin on the metric and stage you've chosen — so you're not acting on random noise.
Compare groups shows a bar chart of the chosen metric (e.g. CVR) for each group, with the Default baseline drawn as a line. Bars above the line are beating the control; bars below are trailing it.
Switch the Sort cards by … measured at … controls to compare on Sessions, CVR, or Sales, at any funnel stage.
When a variant clearly and significantly beats Default (and your holdout, if you have one), you've found a setup worth rolling out to everyone — just update your groups, and the new version starts measuring the result.
💡 Want to verify the numbers yourself? Use Cross-check in Shopify to download a session-level sheet (with each Shopify session ID) and reconcile the figures against your own Shopify orders.
Quick recap
Turn on both embeds (one-time) so visitors are split and measured.
Split traffic into groups — Default (baseline), Variants, and an optional Holdout (control); shares always total 100%.
Read each group's funnel (All visitors → Saw the widget → Watched a video) and its CVR, sales, and lift vs Default.
Every edit saves a new version and starts tracking fresh; older versions stay archived.
Open a version, find the significant winner, compare against the baseline, and roll out what works.
Need a refresher inside the app? Click How groups & metrics work at the top of the Traffic & Groups section anytime.