Evidence-Based Growth

Growth Data Science You Can Trust is a promise with three parts.

  1. I show my uncertainty. You get ranges and probabilities, not a single confident number that hides how little it rests on.
  2. I name the evidence level. A correlation is called a correlation. A tested result is called a tested result.
  3. I tell you what I don’t know, and what it would take to find out.

Most bad marketing decisions are not caused by bad data. They are caused by weak evidence presented with strong confidence.

The Evidence Ladder

Every number behind a decision sits on one of six rungs. The higher the rung, the more you can safely bet on it.

Level 0 Opinion or anecdote

“I think paid social is working.” Useful for generating ideas. Not evidence.

Level 1 Correlation or platform-reported

Dashboards, trends and in-platform return on ad spend. Platforms mark their own homework, and two things moving together doesn’t mean one caused the other.

Level 2 Modelled evidence

Marketing mix models and attribution models. They can show patterns across channels and time, with honest uncertainty, but they rest on assumptions and can be wrong in ways that are hard to see.

Level 3 Quasi-experimental evidence

Before-and-after designs with a comparison group or a counterfactual: difference-in-differences, synthetic control, interrupted time series. Strong when the design is sound, and the design has to be argued for.

Level 4 Controlled experiment

Geo experiments, holdouts and randomised tests where I control who sees what. The cleanest answer to “what did this actually cause?”

Level 5 Triangulated and replicated

Several independent methods agree, and the result has held up more than once. This is what you build a large, hard-to-reverse budget move on.

The rule of thumb: the bigger and harder to reverse the decision, the higher the rung it should stand on. Small, reversible bets can run on Levels 1 and 2. Large budget moves should wait for Level 3 or above.

Illustrative

How much evidence for how big a decision?

Select a level (column) to see what it means. Each cell shows whether that level is enough for that size of decision.

A grid with three decision sizes down the side, small and reversible, medium, and large or hard to reverse, and the six evidence levels across the top. Each cell says whether that level of evidence is enough, thin or too thin for that size of decision. Small decisions are covered from Level 1, medium from Level 2, and large decisions from Level 3. Selecting a level shows what it means and examples.

Evidence level by decision size
Small and reversible Thin Enough Enough Enough Enough Enough
Medium Too thin Thin Enough Enough Enough Enough
Large or hard to reverse Too thin Too thin Thin Enough Enough Enough
  • Enough: this level can carry a decision of that size
  • Thin: one level short; fine if the stakes can be reversed
  • Too thin: wait for stronger evidence

Select a level above to see what it means and what you can decide with it.

Illustrative: a simple guide to matching evidence to decision size, not a rule that fits every business. Illustrative data. Built to show how the method works, not a client result.

Triangulation: three signals that keep each other honest

No single method is enough. I combine three.

  • Marketing mix modelling shows the big picture across channels and time, including diminishing returns and the baseline that would exist without paid media.
  • Experiments (geo tests, holdouts, lift studies) give cleaner answers to specific questions and are used to calibrate the model.
  • Attribution gives fast, granular day-to-day signals, treated as a signal and not as the truth.

When the three agree, you act with confidence. When they disagree, that disagreement is the most useful finding in the room.

Illustrative

Three signals that keep each other honest

Switch between the two situations to see what the overlap tells you.

Three overlapping circles for the marketing mix model, experiments and attribution. They overlap in the middle, and the overlap is the calibrated range, about 1.5 to 2.2 on the return per $1 of spend. When the three agree, you act with confidence. The same three circles pulled apart. The model and the experiments still overlap a little, but attribution sits well away from both and there is no overlap shared by all three. When they disagree, the disagreement is the most useful finding.

Three overlapping circles for the marketing mix model, experiments and attribution. They overlap in the middle, and the overlap is the calibrated range, about 1.5 to 2.2 on the return per $1 of spend. When the three agree, you act with confidence.

View data
Return per $1 of spend, plausible range, by method
MethodRange
Marketing mix model0.6 to 2.4
Experiments (geo experiment)1.2 to 2.6
Attribution (last-click)2.4 to 3.0
Calibrated range (when they agree)1.5 to 2.2
Illustrative: when the three methods agree, act on the overlap. When they disagree, the gap is the finding. Illustrative data. Built to show how the method works, not a client result.

How an engagement runs

  1. Diagnose. What decision has to be made, and what do the numbers really say today?
  2. Design. Choose the methods that fit the question and the data. Sometimes the answer is a test, not a model.
  3. Measure. Build the model or run the experiment, with uncertainty shown at every step.
  4. Decide. Turn findings into a specific action plan: what to change, by how much, in what order.
  5. Deliver. See the plan through. I stay accountable until the agreed objective is achieved, bringing in specialists where needed.

What I check before a model is allowed to influence a decision

  • It predicts data it has not seen (out-of-sample accuracy).
  • Its conclusions don’t flip when I change reasonable assumptions (sensitivity to priors and specification).
  • It can recover a known answer from synthetic data (parameter recovery).
  • It agrees with experiments, or I can explain exactly why it doesn’t.
  • The results are plausible to someone who has run the accounts.
  • Estimates are stable as new data arrives.

What I won’t claim

  • I won’t promise a percentage saving or uplift before I have seen your data. Every business is different.
  • I won’t present a correlation as a cause.
  • I won’t give you a single number without a range.
  • I won’t hand you a black box. You’ll know what went into every estimate.

What I build with

Custom Python packages for econometric modelling, plus open-source tools including PyMC, Robyn and Meridian. Data work in GA4, BigQuery and Databricks. The tools change; the method doesn’t.

Want the technical depth?

The Insights section and my YouTube channel go into the detail: Bayesian marketing mix modelling, causal inference, geo experimentation and forecasting.

Bring me the decision you can’t yet defend