“We ran an A/B test.” Fine. What was your hypothesis? What was the minimum detectable effect? How long did you run it before you called it?
Most marketing experiments don’t fail because the variant was wrong. They fail because they were never experiments — they were opinions with a dashboard attached.
An Experiment Has One Job: Kill a Belief
Before you write a single line of copy or spin up a variant, write this sentence: “We believe that [change] will cause [outcome] because [reason]. We’ll know this is true if [metric] changes by [amount] within [time].”
If you can’t complete that sentence, you don’t have an experiment. You have a decoration.
The goal of an experiment is to produce a result that forces you to update a belief — including the belief that you were right. A test that can only confirm your hypothesis is not a test.
Your Sample Size Is Probably Too Small
The most common failure mode in marketing experimentation is calling results early. A conversion rate shift that looks meaningful at 200 sessions is often noise by 2,000.
Run a power calculation before you start. Decide your minimum detectable effect — the smallest lift that would actually change a business decision — and calculate the sample size required to detect it at 80% power. Most teams discover they need 3–10x more traffic than they thought.
Shorter test duration with smaller samples produces decisions that would have been better made by flipping a coin.
One Variable. One Test.
“We changed the headline, the button color, the hero image, and the page layout.” What did you learn? Nothing you can act on.
Isolation is the price of learning. If you change multiple variables and the test wins, you don’t know why it won. If it loses, you don’t know what to fix. You’ve spent time and traffic to confirm that something changed — which is not information.
One test, one variable, one answer. Move fast by keeping the variable surface small.
Statistical Significance Is Not Business Significance
A result can be statistically significant and commercially irrelevant. A 0.2% conversion lift on a page with 400 monthly visitors moves no needle.
Always translate test results into business terms before celebrating: What does this lift mean in revenue per month? What was the cost to run the experiment (design, engineering, traffic)? Is the payback period under 90 days?
Statistical significance tells you the result is probably real. Business significance tells you whether it matters.
The Studio Take
We run experiments on every marketing surface we build for clients — landing pages, email sequences, lead nurture flows. The ones that produce the most learning are rarely the ones with the best results. They’re the ones that were designed to fail clearly.
A clean negative result is worth more than a messy positive. It eliminates a direction and points you toward the next test faster.
That’s what an experiment is for.
Excerpt: Most marketing experiments don’t fail because the variant was wrong. They fail because they were never experiments — they were opinions with a dashboard attached.