When you can't run an experiment, how do you estimate whether a new feature actually caused an increase in retention?

What the interviewer is looking for

• This question tests causal reasoning. Separates correlation from causation and names suitable methods like difference in differences or matching along with their assumptions.
• Strong answers: Quasi experimental methods like difference in differences, synthetic control or matching, with stated assumptions and sensitivity checks.
• Likely follow-up: Which assumption would worry you most, and how would you test it?

Another way it may be asked: A feature launched to everyone at once and retention went up. How would you work out how much of that was the feature?

A structure to follow: Frame, structure, analyse, recommend

1. Step 1

   Frame

   Restate the goal and ask what success is and what the constraints are.

2. Step 2

   Structure

   Split the problem into a few separate parts you can check one by one, and say what you will look at first and why.

3. Step 3

   Analyse

   Reason with the information you have, state your assumptions out loud, and say what data would change your mind.

4. Step 4

   Recommend

   Give a clear recommendation, the main risk, and the first step you would take.

This is guidance on shape, not a script. Use your own real experience and facts; never invent details to fill a gap. See the STAR guide and answer frameworks.

Common slips

• Listing ideas without choosing between them.
• Hiding assumptions instead of stating them.
• Finishing without a recommendation.

How to practise it

Answer in four spoken parts, one per step, then check you ended on a recommendation.

Related questions

• How would you explain a p-value to a product manager who has never taken a statistics course?
• You get a dataset where about a fifth of the values in an important column are missing. How do you decide what to do?
• What is overfitting, and how would you spot it in a model you just trained?
• Tell me about a split test or experiment you designed or analysed. How did you choose the success metric and the sample size?
• You're building a fraud model where only one in a thousand transactions is fraud. How would you evaluate whether it's any good?

Browse all questions