Risky-Assumption Growth Experiments
- Difficulty
- Moderate
- Time to result
- ~weeks to results
- Steps
- 7
- Confidence
- —
Step three of the growth levers process: a method for turning slow, expensive ideas into tests that finish in a week or two. Ideas are first triaged on four cuts, whether they hit the rate-limiting step, how big they could be if they worked, whether they carry real risk, and whether they would teach you something important about your customers or your business. Anything big, on-model and genuinely risk-free is simply done rather than tested. For the rest, you break the idea into its moving parts, name the risky assumptions, isolate one and find the fastest possible way to check it. Each experiment gets a written hypothesis, a prediction made in advance and a documented learning afterwards.
Origin
Extracted from Deep Dive with Ali Abdaal
How to run it
- 1
Write down every idea
Collect the full list without judging it. Most of the ideas will be bad, and that is expected; the filtering happens next, not during generation.
Pro tip Capture ideas that keep resurfacing in internal debates separately, because those are prime candidates for a learning experiment.
Watch out Do not start executing on the list. Volume of ideas is not the constraint.
- 2
First cut: is it aimed at the rate-limiting step?
Check each idea against the growth model. If it does not open the current bottleneck or serve a specific discrete goal such as building a second acquisition channel, it goes.
Pro tip Write the discrete goal on the board so the filter is objective rather than a matter of taste.
Watch out Ideas that improve a part of the system that is not constraining throughput will produce no measurable business effect.
- 3
Second cut: how big could it be if it works?
Estimate the ceiling. If it cannot be big, it does not get started, however easy or appealing it looks.
Pro tip Do the arithmetic at scale before falling in love with the tactic.
Watch out Small wins still consume headcount, attention and complexity, which is the hidden cost.
- 4
Third cut: is it actually risky?
Ask whether it could backfire, or is hard, expensive or distracting. If nothing risky comes to mind and it is a good idea, skip the experiment overhead and just do it.
Pro tip A great customer testimonial and an empty website is Lerner's example: no reason not to, thirty seconds of work, no experiment required.
Watch out Running a formal experiment on a zero-risk change wastes organisational overhead.
- 5
Fourth cut: will it teach you something important?
Some experiments are worth prioritising purely for the learning, even over size, particularly when they settle a debate the team keeps having about customers or product.
Pro tip Score learning value explicitly when the same argument has come up more than twice.
Watch out You can only run a finite number of experiments, so learning value has to compete honestly against impact.
- 6
Isolate one risky assumption and test it in a week
Break the idea into moving parts, list the risky assumptions and pick one you can check quickly, rather than building the whole thing to test them all at once.
Pro tip Often the fastest test is asking people who have already done the thing you are considering.
Watch out Testing a bundle of assumptions at once tells you the idea failed but never why.
- 7
Write the hypothesis, predict, run, document
State that you believe X, therefore you predict that doing this will move that number in this direction by this amount, and that if you are right you will do this differently in the business. Have the team make predictions, run it, then record results, learnings, open questions and screenshots.
Pro tip Make a game of the predictions; it engages the team and eliminates hindsight bias when the numbers arrive.
Watch out If there is no stated change you would make when the prediction proves right, there is no point running the experiment.
In the wild
Ali's example was writing a productivity book to feed a productivity course, a project taking three and a half years. Lerner listed the risky assumptions: that Ali can write a book quickly, that it will not distract him from the business, that people will buy it, that they will like it, and that people who like it will buy the course. Rather than testing all of them by writing the book, Ali realised he could email the handful of people he knows who have both a book and a course and ask what the book actually contributed. Two said nothing, because they already had a YouTube channel; one said it was massive, because they did not.
→ A three-and-a-half-year bet had its central assumption checked in about a week of emails.
Ali wanted vlogs on his main channel because loyal viewers love them, but they reliably get far fewer views than sit-down educational videos. Rather than debating it, Lerner asked whether there was a way to run an experiment and find out. The team committed to publishing vlogs on the main channel for the next few months and watching what happens to monthly returning viewers, the metric they had just agreed as the North Star.
→ A recurring internal argument was converted into a time-boxed test against a single agreed metric.
Common mistakes
Building the whole thing to test the idea
The instinct with a hard idea is to execute it and see. Lerner's point is that hard ideas are precisely where you must isolate one assumption and find a one-week proxy test.
No stated decision attached to the result
If you have not written down what you will do differently when the prediction holds, the experiment produces a number and no action. That last clause is what makes it worth running.
Over-worrying about what could go wrong
Lerner sees startups spend a long time de-risking every imaginable failure, then get hit by something unexpected anyway. Spend less time worrying about what could go wrong and more time figuring out what did.
From the transcript
“so the way you'll do that is you'll look at this idea and you'll say okay all the thing the moving parts of this what…”
“the third cut is is this risky in any way so could this backfire is this hard is this expensive is this distracting if you…”
“you write a well fored hypothesis We Believe X therefore we predicted if we do this thing this number will move in this direction by…”
From the episode
How to Scale your Startup with Growth Levers: Matt Lerner