A/B Test

5 min read

Also known as: Split Test, Bucket Test, Champion/Challenger Test

An A/B test compares two versions of a page, form, or asset against live traffic to determine which one drives better conversion.

Definition

An A/B test is a controlled experiment where you split incoming traffic between two variants — typically a control (A) and a challenger (B) — and measure which version produces better results on a defined metric like form submissions, clicks, or revenue per visitor. Each visitor sees only one version, and the difference in performance tells you which variant to keep.

In funnel work, operators run A/B tests on headlines, form length, CTA copy, hero images, pricing display, and chat widget triggers. The test runs until you hit statistical significance — usually a confidence level of 95% — and then the winning variant becomes the new baseline for the next round of testing.

A/B testing differs from multivariate testing, which changes several elements at once to measure interaction effects. A/B is faster, cleaner, and better suited to mid-market traffic volumes, where you rarely have enough sessions to power a multivariate test to significance in a reasonable timeframe.

Why It Matters

Funnel performance compounds. A 12% lift on a landing page that feeds a 20% lift on a form, followed by an 8% lift on the booking step, multiplies into a materially larger pipeline without spending another dollar on traffic. A/B testing is how you find those lifts instead of guessing at them in a planning meeting.

Teams that skip A/B testing tend to redesign by opinion — the loudest stakeholder wins, the page changes, and nobody knows whether conversions went up or down because there's no baseline to compare against. Worse, real regressions get attributed to seasonality or ad spend, and the team keeps shipping changes that quietly bleed pipeline.

Examples in Practice

A B2B SaaS sales team runs an A/B test on its demo-request form. Version A asks for company size; version B drops the field entirely. After two weeks and roughly 4,000 sessions, the shorter form converts 18% higher with no measurable drop in lead quality, so the team ships version B and reroutes the freed-up qualification step into the post-submit confirmation flow.

A 30-person agency tests two hero headlines on its services landing page — one outcome-focused ('Close 30% more deals in 90 days') and one capability-focused ('Full-funnel marketing automation'). The outcome variant wins by a wide margin on booked calls, and the team rewrites the rest of the site in that voice.

An e-commerce brand A/B tests a chat widget trigger: appearing at 30 seconds versus on exit intent. The exit-intent version captures 22% more emails without hurting checkout completion, so it becomes the default across all product pages.

Frequently Asked Questions

What is an A/B test and why does it matter?

An A/B test splits live traffic between two versions of an asset — a page, form, email, or widget — to measure which one performs better on a specific conversion metric. It matters because it replaces opinion-driven design decisions with evidence, and the compounding lifts across a funnel translate directly into more pipeline from the same ad spend.

How is A/B testing different from multivariate testing?

A/B testing compares two whole variants against each other, changing one meaningful thing at a time. Multivariate testing changes several elements simultaneously and measures how they interact. Multivariate needs significantly more traffic to reach statistical significance, which is why most mid-market teams stick with A/B unless they're running a high-volume e-commerce site.

When should I use an A/B test?

Run an A/B test whenever you have a hypothesis about a change that could measurably move a key conversion metric and enough traffic to reach significance within two to four weeks. Skip the test for cosmetic fixes, urgent bug patches, or pages with too little traffic to ever produce a confident result.

What metrics measure A/B test outcomes?

The primary metric is the conversion rate on the action you're testing — form submissions, clicks, signups, or revenue per visitor. Secondary metrics include statistical confidence (typically 95% or higher), minimum detectable effect, and downstream impact on pipeline or revenue. Always define the primary metric before launching the test, not after the data comes in.

What's the typical cost of running A/B tests?

Most A/B testing tools fall in the $0 to $500 per month range for mid-market traffic volumes, with enterprise platforms running $1,000 to $5,000 per month. The bigger cost is usually internal — strategist time to design tests, developer time to implement variants, and analyst time to interpret results. Budget 5-15 hours of team time per test.

What tools handle A/B testing?

A/B testing tools fall into several categories: dedicated experimentation platforms, conversion optimization suites, web analytics tools with built-in testing, and funnel builders with native split-test features. Many marketing automation platforms also include basic A/B testing for emails and landing pages. The right choice depends on traffic volume, technical resources, and whether testing is a core function or occasional need.

How do I implement A/B testing for a small team?

Start with one high-traffic page and one clear hypothesis. Pick a tool that handles traffic splitting and significance calculation for you, so you're not building it in a spreadsheet. Run one test at a time to keep results clean, document the hypothesis and result, and build a simple library of wins and losses so you stop retesting the same ideas every six months.

What's the biggest mistake teams make with A/B testing?

Calling tests too early. Teams see version B leading after three days, declare a winner, and ship it — then watch the lift evaporate because the result wasn't statistically significant. The fix is to define your sample size and confidence threshold before the test starts, and refuse to look at directional data until you hit it. The second-biggest mistake is testing trivial changes like button colors when bigger wins live in headlines, offers, and form length.

How long should an A/B test run?

Long enough to reach statistical significance on your primary metric, and at minimum one full business cycle — usually one to two weeks — to account for day-of-week traffic patterns. A test that hits significance in 48 hours is often a false positive driven by an unusual traffic mix. Plan for two to four weeks per test in most mid-market funnels.

Can A/B testing hurt SEO or user experience?

Properly implemented A/B tests don't hurt SEO — search engines accept testing as long as you're not cloaking content or showing different versions to bots. User experience risk is real if you run too many overlapping tests or ship changes without measuring downstream impact. Keep tests focused, monitor revenue and retention alongside conversion, and roll back losers quickly.

The AMW Suite