Beyond accuracy, what do you monitor for a model running in production?

What the interviewer is looking for

• This question tests model monitoring & reliability. Monitors drift, latency, errors and business outcomes, with alerts, dashboards and a plan for retraining or fallback.
• Strong answers: Latency, error rates, input and prediction drift, data freshness, fairness across groups, cost and business outcomes.
• Likely follow-up: Which of those would page someone at night, and which wouldn't?

Another way it may be asked: Your model's accuracy looks fine on paper. What else could go wrong that you'd want alerts for?

A structure to follow: Answer, explain, example, limits

1. Step 1

   Answer

   Give your direct answer in the first sentence so the interviewer knows where you are going.

2. Step 2

   Explain

   Say how it works or why you would do it that way, in plain words before jargon.

3. Step 3

   Example

   Anchor it in something you have actually done or seen, with the detail that shows hands-on experience.

4. Step 4

   Limits

   Say when your approach would not be the right one. It shows judgement.

This is guidance on shape, not a script. Use your own real experience and facts; never invent details to fill a gap. See the STAR guide and answer frameworks.

Common slips

• Reciting a definition without showing you have applied it.
• Using terms you cannot explain if asked a level deeper.
• Claiming experience beyond what you can discuss in detail.

How to practise it

Say the answer in 90 seconds, then answer the follow-up as if it were a new question.

Related questions

• Tell me about a time a model's performance dropped in production. How did you notice and what did you do?
• Why do we split data into training, validation and test sets, and what goes wrong if information leaks between them?
• How would you turn a notebook a data scientist gave you into code that's safe to run in production?
• What does it mean for features to be computed the same way in training and in production, and why does that matter?
• Walk me through how you've deployed a model. How was it packaged, served and rolled out safely?

Browse all questions