Tell me about a time a model's performance dropped in production. How did you notice and what did you do?

What the interviewer is looking for

• This question tests model monitoring & reliability. Monitors drift, latency, errors and business outcomes, with alerts, dashboards and a plan for retraining or fallback.
• What to bring out: Detection through monitoring, root cause such as data drift or a pipeline bug, a fix and better safeguards afterwards.
• Expect a follow-up like: How long did it take to notice, and could that have been shorter?

Another way it may be asked: Think of a live model that started making worse predictions. How was it caught and how was it fixed?

A structure to follow: STAR, with the weight on Action and Result

1. Step 1

   Situation

   One or two sentences: where, when and what was at stake. Pick a real example you were close to.

2. Step 2

   Task

   What you personally were responsible for. Say "I", not only "we", so the model monitoring & reliability you showed is visible.

3. Step 3

   Action

   The two or three things you did and why you chose them over the alternatives. This should be about half your answer.

4. Step 4

   Result

   What changed, in your own measure of it, and what you would do differently. Only use numbers you can stand behind.

This is guidance on shape, not a script. Use your own real experience and facts; never invent details to fill a gap. See the STAR guide and answer frameworks.

Common slips

• Describing what the team did without saying what you did.
• Choosing an example with no clear outcome, or one you cannot answer detailed questions about.
• Ending on the action and skipping the result and what you learned.

How to practise it

Write two bullet points for each STAR letter, then say the answer aloud in about two minutes.

Related questions

• Beyond accuracy, what do you monitor for a model running in production?
• Why do we split data into training, validation and test sets, and what goes wrong if information leaks between them?
• How would you turn a notebook a data scientist gave you into code that's safe to run in production?
• What does it mean for features to be computed the same way in training and in production, and why does that matter?
• Walk me through how you've deployed a model. How was it packaged, served and rolled out safely?

Browse all questions