A/B test.
aka split test, controlled experiment, bucket test
Two versions, real users, and the data picks the winner. Not the boss.
What it actually means
An A/B test shows two versions of something (a headline, a button, a price, an email) to two random groups of users at the same time, then measures which version performs better on one chosen metric. It replaces opinions with evidence.
The setup matters more than the idea. Split traffic randomly so the groups are comparable. Change one thing, or you will not know which change worked. Decide the metric before you start (sign-ups, clicks, revenue per visitor). And run it long enough to collect enough data, because a “winner” after 200 visitors is usually noise.
That last part is where most tests go wrong. Statistical significance is the check that the difference is real and not luck. Teams that peek at results daily and stop when their favourite version is ahead are not testing; they are confirming. A good testing culture also accepts that most tests lose, and that a losing test is still useful because it kills a bad idea cheaply.
Where you'll hear it
The A/B on the pricing page finished: version B with annual pricing first lifted paid conversions 11%, 97% significance over three weeks. Rolling it out Monday.
Translation: the change was tested properly, it won by a real margin, and nobody had to argue about it in a meeting.
You will hear it in growth teams, product teams, email marketing and anywhere a website makes money. It also appears as “let’s test it” at the end of a disagreement, which is the healthiest way a meeting can end. Bigger cousins exist: multivariate tests change several things at once, but need far more traffic.
The pop-culture version
Big creators do this constantly. Same video, two thumbnails: one with a shocked face, one with a bold text title. Half the viewers see one, half see the other. After a day, one has a clearly higher click rate and becomes the thumbnail for everyone.
Nobody argued about which thumbnail was better. Nobody asked the creator’s cousin. The audience voted with clicks, and the vote was counted fairly because both thumbnails ran at the same time to similar people.
That is an A/B test. Businesses do exactly the same thing with checkout buttons, subject lines and prices. The skill is in running it fairly, and in accepting the result when your favourite loses.
The trap is impatience. A thumbnail that is ahead after 30 views is not a winner; it is a coin that landed heads twice. Real tests wait for enough data before calling it.
Say it like you mean it
“Version B lifted sign-ups 11% at 95% significance after 21 days; we’re shipping it.”
The change, the size, the confidence and the duration. That is a test result someone can trust.“We A/B tested it and B looked better.”
Looked better to whom, after how many users? Without a metric and a sample size, it is an opinion with extra steps.If someone asks “was it significant?”, they mean: could this result plausibly be luck? Know your sample size and confidence level before you present a winner, and you will never be caught out by that question.
The tools teams use to run tests properly, compared on statistics engine, traffic limits and price.
Decoded next
Decoded by the Kurums editorial team. Kurums Decoded translates the language of business for people who are new to it. Definitions are kept short on purpose; the serious version above goes deep. Nothing here is financial, legal or career advice.
Discover more from Kurums | Business Intelligence
Subscribe to get the latest posts sent to your email.
