Tell me about a time a Spark or similar distributed job was slow or failing. How did you find the cause and what did you change?

What the interviewer is looking for

• This question tests distributed processing. Diagnoses slow jobs using execution plans or the job UI, names causes like skew or heavy shuffles, and reports a measured improvement.
• What to bring out: Uses the job UI or plan to spot skew, shuffles, small files or memory pressure, applies a targeted fix and measures the result.
• Expect a follow-up like: How did you make sure the fix would still hold as the data kept growing?

Another way it may be asked: Think of a big data job that took far too long or kept crashing. Walk me through how you figured out why and fixed it.

A structure to follow: STAR, with the weight on Action and Result

1. Step 1

   Situation

   One or two sentences: where, when and what was at stake. Pick a real example you were close to.

2. Step 2

   Task

   What you personally were responsible for. Say "I", not only "we", so the distributed processing you showed is visible.

3. Step 3

   Action

   The two or three things you did and why you chose them over the alternatives. This should be about half your answer.

4. Step 4

   Result

   What changed, in your own measure of it, and what you would do differently. Only use numbers you can stand behind.

This is guidance on shape, not a script. Use your own real experience and facts; never invent details to fill a gap. See the STAR guide and answer frameworks.

Common slips

• Describing what the team did without saying what you did.
• Choosing an example with no clear outcome, or one you cannot answer detailed questions about.
• Ending on the action and skipping the result and what you learned.

How to practise it

Write two bullet points for each STAR letter, then say the answer aloud in about two minutes.

Related questions

• Can you explain the difference between a fact table and a dimension table, and when you would use each?
• If you had to load a daily file from a partner into a warehouse, how would you make sure running the job twice doesn't create duplicate rows?
• What checks would you put on a new data source before analysts start relying on it?
• How do you structure dependencies and retries in an orchestration tool like Airflow or Dagster so that one failure doesn't cascade into a mess?
• Walk me through how you would choose between a cloud warehouse like Snowflake or BigQuery and a lakehouse setup for a new analytics platform.

Browse all questions