AI Engineer
How do you handle latency in a chat or voice product where users expect a reply almost immediately?
What the interviewer is looking for
- This question tests production reliability & cost. Manages latency, cost and failures with streaming, caching, model routing, retries, fallbacks and observability.
- Strong answers: Streaming responses, smaller or faster models, shorter prompts, caching, parallel calls and measuring time to first token.
- Likely follow-up: What quality trade-offs would you accept to make it faster?
Another way it may be asked: Your voice assistant takes four seconds to start answering and users hang up. What would you change?
A structure to follow: Answer, explain, example, limits
- Step 1
Answer
Give your direct answer in the first sentence so the interviewer knows where you are going.
- Step 2
Explain
Say how it works or why you would do it that way, in plain words before jargon.
- Step 3
Example
Anchor it in something you have actually done or seen, with the detail that shows hands-on experience.
- Step 4
Limits
Say when your approach would not be the right one. It shows judgement.
This is guidance on shape, not a script. Use your own real experience and facts; never invent details to fill a gap. See the STAR guide and answer frameworks.
Common slips
- Reciting a definition without showing you have applied it.
- Using terms you cannot explain if asked a level deeper.
- Claiming experience beyond what you can discuss in detail.
How to practise it
Say the answer in 90 seconds, then answer the follow-up as if it were a new question.
Related questions
- Your app's language model costs are growing faster than usage. What levers would you pull?
- In simple terms, what is retrieval augmented generation, and why would you use it instead of just asking a language model directly?
- A chatbot you built keeps giving long rambling answers. What would you change first?
- What is prompt injection, and how might it affect an assistant that reads customer emails?
- Your document search assistant often answers using the wrong document. How would you debug and improve retrieval?