Walk me through a test you designed. How did you decide it was a real result and not noise?

What the interviewer is looking for

• This question tests analytics, attribution & experimentation. Defines success metrics up front, designs sound tests, understands attribution limits and makes decisions from data.
• Strong answers: A clear hypothesis, one main metric, enough sample size or run time and not stopping early on a lucky spike.
• Likely follow-up: What do you do with a test that shows no difference?

Another way it may be asked: Tell me about an experiment like testing two versions of a page. How did you know which one truly won?

A structure to follow: Answer, explain, example, limits

1. Step 1

   Answer

   Give your direct answer in the first sentence so the interviewer knows where you are going.

2. Step 2

   Explain

   Say how it works or why you would do it that way, in plain words before jargon.

3. Step 3

   Example

   Anchor it in something you have actually done or seen, with the detail that shows hands-on experience.

4. Step 4

   Limits

   Say when your approach would not be the right one. It shows judgement.

This is guidance on shape, not a script. Use your own real experience and facts; never invent details to fill a gap. See the STAR guide and answer frameworks.

Common slips

• Reciting a definition without showing you have applied it.
• Using terms you cannot explain if asked a level deeper.
• Claiming experience beyond what you can discuss in detail.

How to practise it

Say the answer in 90 seconds, then answer the follow-up as if it were a new question.

Related questions

• What numbers would you look at to tell whether a marketing campaign worked?
• Different attribution models tell different stories. How do you decide where credit goes and convince leadership to trust the numbers?
• If you were launching a new product with a small budget, which channels would you try first and why?
• Pick a product you like. How would you explain the difference between its features and its benefits to a customer?
• Tell me about a paid campaign you ran or managed. How did you improve its cost per acquisition over time?

Browse all questions