UX research methods: which to use when

Most small teams that do user research reach for the same tool every time. Founders send a survey, analysts open the dashboard, designers book a few interviews. Each method answers some questions well and others badly, so a team that only uses one ends up with confident answers to the wrong questions.

This guide explains how to pick between the four methods most teams can afford (interviews, usability tests, surveys and analytics), using the framework the Nielsen Norman Group (NN/g) uses to sort research methods. It ends with a worked example and the mistakes that waste research budgets.

Three questions that sort every method

Christian Rohrer’s article When to use which user-experience research methods places methods on three dimensions. You don’t need the full chart to use the idea. Ask three questions about what you need to learn.

• Do you need to know what people say or what they do? NN/g calls this attitudinal versus behavioural. Surveys and interviews capture what people report. Analytics and usability tests show what they do. The article notes that most usability studies should rely on behaviour.
• Do you need to know why, or how many? Qualitative methods are better for understanding why a problem happens and how to fix it. Quantitative methods tell you how many people are affected and how much.
• How close to real use is the study? Analytics watch natural use with no interference. A usability test uses a script. An interview may not involve the product at all.

Once you’ve answered those three, the shortlist usually picks itself.

What each of the four methods is good and bad at

NN/g’s User interviews 101 is direct about the limits of interviews: they are not a way to evaluate a design, and self-reported behaviour suffers from faulty recall, missing detail and people wanting to look good. If you want to know what people do, it recommends observation or behavioural data such as analytics.

For usability tests, Jakob Nielsen’s article Why you only need to test with 5 users argues that about five participants find most of the problems in a design, and that several small rounds beat one large study. Raluca Budiu’s Quantitative vs. qualitative usability testing adds that quantitative studies, which put a number on usability, need far more people, often more than 30.

Match the method to where you are

NN/g groups methods by stage of product development:

• Strategise. You are looking for new directions. Generative methods fit here: interviews, field studies, diary studies, surveys and concept tests.
• Design. You are improving a design. Formative methods fit: usability testing, card sorting and tree testing.
• Launch and assess. You are measuring performance against your past self or competitors. Summative methods fit: analytics, A/B tests, benchmark usability studies and surveys.

A simple rule follows. Before you build, talk to people. While you build, watch people use it. After you launch, count.

A worked example

An example (illustration only). A small online shop selling refillable cleaning products sees that many visitors add items to the basket but few complete checkout. The founder’s first instinct is a survey asking customers why they didn’t buy.

Run through the three questions. The team needs to know what people do at checkout (behaviour), why they stop (qualitative), and in a realistic setting. A survey answers none of those well: people who abandoned a basket rarely answer, and those who do are guessing at their own reasons.

A better plan uses three methods in sequence:

1. Analytics first. Find which checkout step loses the most people, split by device. Suppose most of the loss is on the delivery options page on mobile.
2. Usability test second. Ask five people who match the customer profile to buy a refill on their own phones while thinking aloud. Watch what happens on the delivery page.
3. Analytics again. After fixing what the test revealed, compare drop-off on that step before and after the change.

Interviews would come in later if the team wanted to understand why people choose refills at all, which is a strategy question rather than a checkout question.

How to choose in five steps

1. Write the decision first. Phrase it as a choice you will make, such as “Should we cut the delivery options page?” Research without a decision attached tends to gather dust.
2. Turn it into questions. List what you need to know to make the decision.
3. Sort each question. Says or does? Why or how many? Real use or scripted?
4. Pick the cheapest method that fits. Analytics you already collect cost nothing. Five usability sessions cost a few days.
5. Plan a second method to check the first. Rohrer’s article notes that nearly all projects benefit from combining methods. Quantitative data shows that a problem exists, and qualitative work shows how to fix it.

Common mistakes

• Asking people to predict their own behaviour. “Would you use this feature?” invites a polite yes. Ask about what they did last time, or watch them try.
• Using interviews to test a design. Showing a mock-up and asking for opinions gives you preferences, not evidence of use. Give people a task instead.
• Treating five sessions as statistics. Five people will show you problems. They cannot tell you that 40% of users have them.
• Reading analytics as reasons. A high exit rate on a page tells you where to look, not what is wrong.
• Leading questions. NN/g warns that interview data quality depends heavily on the interviewer’s skill. “Don’t you find this confusing?” produces the answer it suggests.
• Researching only once. One big study at the start, then nothing, leaves the team guessing for the rest of the project.

When to use lighter research, or none

Not every decision needs a study. If a change is cheap to reverse and you can measure its effect, ship it and watch the numbers. If you already know a page is broken because support tickets say so, fix it first and test afterwards. Save the effort for decisions that are expensive to undo, such as a new pricing structure, a redesign of navigation or a new product line.

When you do run research, the GOV.UK Service Manual’s guidance on moderated usability testing is a good model of discipline: agree the research questions, the user types and the parts of the service to focus on before booking anyone. That habit alone stops most of the wasted studies described above.