Hypothesis-driven work: start with an answer, then try to break it

Faced with a problem, a careful analyst’s instinct is to gather all the data first and let the answer emerge. It feels rigorous. In practice it often means weeks of spreadsheets and still no clear view of what matters.

Hypothesis-driven work flips the order. You state your best guess at the answer early, then go looking for evidence that would prove it wrong. If the guess survives, you reach an answer sooner. If it dies, you’ve learned something and move to the next one. This guide covers how to form a working hypothesis, how to test it, and how to avoid fooling yourself along the way.

What a working hypothesis is

A hypothesis is a proposed explanation that makes a testable prediction. A working hypothesis is one you adopt provisionally, as a basis for further work, knowing it may be wrong. Wikipedia’s entry on hypotheses notes that working hypotheses are often discarded, and treats that as a normal part of research.

In business, a useful working hypothesis names a cause, implies something you can check, and could turn out to be false. “Our pricing is wrong” is too vague to test. “Paid sign-ups fell because the March price rise made the annual plan look expensive” can be tested, because it predicts when the drop started and which plan it hit.

Why starting with an answer speeds things up

A hypothesis tells you which data to collect. Without one, every number looks possibly relevant. With one, you can ask two questions: what would I expect to see if this were true, and what would I see if it were false? The analysis shrinks to the few cuts of data that can tell those two apart.

In a McKinsey podcast on problem solving, Charles Conn and Hugo Sarrazin talk about teams producing a “one-day answer” or “one-hour answer” and refining it as the work goes on. The early answer is rough. Its value is that it gives the team something specific to attack.

It also changes how you report. Sarrazin suggests presenting findings with their conditions attached, for example: “This is a likely answer under the following sets of conditions.” That is an accurate statement of where a hypothesis stands, and it tells the reader what would change the conclusion.

How to test a hypothesis

1. Write it down in one sentence, naming the cause and the effect.
2. List its predictions. If the hypothesis is true, what else must be true? Where, when and for whom should the effect show up?
3. Write the kill condition before you look at the data. Decide in advance which result would make you drop the hypothesis. It is easy to skip, and it does more than any other step to keep you honest.
4. Find the cheapest test. Often it is a filter on an existing report, a look at the week something changed, or five calls with customers.
5. Run it and record the result, including tests that went against you.
6. Decide whether to keep, refine or kill the hypothesis. A refined hypothesis is narrower and goes back through the same steps.

The logic behind the kill condition comes from Karl Popper’s idea of falsifiability. No number of white swans proves that all swans are white, but a single black swan disproves it. Supporting evidence can pile up for ever without settling the question. One clean contradiction can settle it.

Hold more than one hypothesis at a time

In 1964 the biophysicist John R. Platt published a paper in Science called “Strong inference”. The method starts by writing down several alternative hypotheses, then designing tests whose possible results would each rule out one or more of them. You then repeat the process with narrower hypotheses.

Holding several hypotheses weakens your attachment to any single one. Thomas Chamberlin, writing in 1897, warned that “the temptation to misinterpret results that contradict the desired hypothesis is probably irresistible.” A list of rivals makes that temptation easier to spot.

It also changes which tests you value. A test that separates two hypotheses is worth more than one that is merely consistent with your favourite.

The main risk: confirmation bias

Confirmation bias is the tendency to search for, interpret, favour and recall information in ways that support what you already believe. Hypothesis-driven work invites it. Once you’ve said an answer out loud, you want it to be right.

The classic demonstration is Peter Wason’s 2-4-6 task, published in 1960. Participants were told that the numbers 2, 4, 6 followed a hidden rule and were asked to find it by proposing other sets of three numbers. The rule was simply any ascending sequence. Many people settled on narrower rules, because they mostly tested sets that fitted their guess rather than sets that could break it. Later researchers argued this reflects a “positive test strategy” more than bias as such, but the practical lesson for analysts is the same: test where you could be wrong.

Defences that help in practice:

• Set kill conditions before you see the data.
• Ask a colleague to argue the opposite case.
• Find the test that would hurt most if you were wrong, and run it first.
• Keep a running note of evidence against the hypothesis, not only evidence for it.
• Where the team is big enough, have someone other than the author check the result.

An example (illustration only): a fall in paid sign-ups

Take an online course business, invented for this example, whose paid sign-ups fell over a month. The team lists three hypotheses.

• H1: a price rise on the annual plan put people off.
• H2: a change to the checkout broke payment for some users.
• H3: traffic shifted towards visitors who rarely buy, such as a burst of social media referrals.

Each makes different predictions. If H1 is true, the drop should start on the date of the price change and hit the annual plan harder than the monthly plan. If H2 is true, payment errors should rise and the drop should cluster on particular browsers or payment methods. If H3 is true, conversion within each traffic source should hold steady while the mix of sources changes.

The team writes kill conditions first. H1 dies if monthly and annual plans fell by the same proportion. H2 dies if payment error rates are flat. H3 dies if conversion fell within every source.

The first test is one report: conversion by traffic source by week. Suppose conversion fell within every source. H3 is out within the hour. A second report, payment errors by browser, comes back flat, so H2 goes too. H1 survives, and the team now has a narrower question: is the problem the price itself, or how the price is shown on the page?

The team ran two reports, each chosen because it could kill a hypothesis, instead of building a forty-metric dashboard first.

When the approach goes wrong

It fails when the first hypothesis becomes the answer by default. Watch for these signs: nobody wrote down what would disprove it, every new fact is described as “consistent with” it, and no alternative explanations were ever listed. If you see them, go back to the list of rivals and run the test most likely to embarrass the current favourite.

It also fails when the problem itself hasn’t been defined. A hypothesis about the wrong question can be well tested and still useless. Pair this method with an issue tree: the tree lays out the possible causes, and your hypotheses tell you which branch to test first.