Data engineering interviews test production judgment

A nightly job that normally finishes before breakfast is still running at 8am. Most tasks completed, one executor is crawling and the sales dashboard is blank. The interviewer asks what you inspect first. Naming Spark, Airflow and Kafka won’t answer the question.

Data engineering interviews keep returning to this kind of production mess because it exposes how you think. The data engineering interview track lets you work through Spark, pipelines, modeling, quality and missed SLAs with someone asking follow-ups. That pressure is useful. A diagram in your notes never asks whether the retry will duplicate yesterday’s orders.

Start with the constraint

Most candidates can sketch a modern data path. Events land somewhere, transformations run and modeled tables reach a warehouse on a schedule. The sketch earns very little by itself because the interesting decisions depend on the workload.

Ask about volume, freshness, consumers and the cost of being wrong. A finance table used for month-end reporting has a different tolerance than a recommendation feed. Batch is often a sensible default when the business can wait. Streaming brings out-of-order events, state and a harder on-call, so the latency requirement should pay for that complexity.

The same habit applies when choosing storage, orchestration or a modeling pattern. State the constraint that rules options in or out. “The source sends corrections 3 days late” gives you a reason to discuss watermarks and backfills. “I like Kafka” tells the interviewer what was open in your browser last week.

Diagnose the stuck Spark task

You should know the driver, cluster manager and executors, along with the difference between transformations and actions. That vocabulary lets you describe what is running. Diagnosis begins when you connect the execution model to the symptom.

A shuffle moves data so matching keys end up together. It can hit disk and network, and skew can leave one partition carrying most of the work. If nearly every task finishes while one crawls, inspect partition sizes, stage duration and shuffle read. Compare the slow task with its siblings before asking for more memory. Skew, partition sizing and shuffle behavior are the center of the Spark Performance Tuning session.

The fix should match the evidence. A small dimension may be safe to broadcast. A hot key may need salting or separate treatment. File counts and partition size matter too, especially when thousands of tiny object-store files turn planning into a tax. After any change, explain how you would compare record counts and business totals so a faster job doesn’t quietly change the answer. The Apache Spark execution session is a useful place to find out whether that diagnosis still holds when the symptoms change.

Lazy evaluation often appears as a follow-up. Spark builds a plan through transformations and runs it when an action is called. If the same source is recomputed twice, caching may help, though memory pressure can make that choice worse. Say what you would measure. Performance tuning without measurement is expensive folklore.

Make reruns safe

Pipeline design gets interesting after a partial write. Suppose a task writes half a partition and dies. The orchestrator retries it. Will the table contain duplicates, replace the partition cleanly or expose a mix of old and new records?

Idempotency answers that question. Use stable event IDs, transactional writes, merge semantics or partition replacement where they fit. The interviewer may push on late data and repeated events, so your incremental load needs a correction policy. A watermark is useful only if you can explain what happens outside its window.

Full reloads deserve a fair hearing for small tables. Copying the complete source can be cheaper than maintaining a fragile incremental path. For larger data, change data capture may preserve inserts, updates and deletes, but it also brings ordering and replay questions. Keep a dead-letter path for records that repeatedly fail, and define how they return after correction.

Delivery guarantees belong here. At-least-once processing expects duplicates and relies on consumers or writes that can absorb them. Claims of exactly-once behavior need a boundary, because the guarantee may end at the broker while an external database still sees a retry. Describe the guarantee the consumer requires and how you test it.

Grain decides whether the dashboard lies

A warehouse answer should begin with grain. What does one row mean? An order, an order line and a daily account snapshot lead to different joins and measures. If the grain is vague, a dashboard can repeat revenue with total confidence.

Facts record measurable events. Dimensions hold the people, products, accounts or places used to slice those events. A type 2 slowly changing dimension keeps history when a customer attribute changes, which lets an analyst ask what the customer looked like at the time of a sale. State which changes need history and which can overwrite the current value.

Star and snowflake designs involve join cost, duplication and ease of change. A data lake keeps raw or lightly processed data cheaply. A warehouse gives analysts modeled tables and SQL access. Analytics engineering with staging models, marts and tests helps keep the contract legible, but dbt cannot rescue a table whose grain was never decided.

Silent failures need business language

A red pipeline is almost generous because it creates an alert. Bad data can arrive on time and pass every technical check. Completeness, accuracy, consistency, freshness and uniqueness each fail in a different way, and a single “quality score” can hide the dimension that matters.

Tie the check to a decision. If close dates are missing on a portion of deals, the revenue forecast may be low. If customer IDs repeat, a campaign may contact the same person twice. A stakeholder can respond to those consequences. “The data has quality issues” leaves them guessing.

Blocking every anomaly can freeze a morning load, while warnings can become inbox wallpaper. Choose which failures stop publication and which publish with a visible warning. Define ownership and an escalation path. The judgment sits in that threshold, which is where the Data Quality session spends its time.

A missed SLA tests communication as much as debugging. Tell affected partners what is late, which reports or decisions are affected and when the next update will arrive. You can share the root cause later. Waiting for perfect certainty often means the first business update arrives after someone has already presented the wrong number.

What the track covers

The track has 2 Spark sessions, one on execution and another on performance tuning. It also covers ingestion, orchestration, star schemas and fact tables, warehouses with dbt, Kafka, data quality and a senior session on owning a missed SLA. SQL exercises and take-home coding belong beside this practice, since the live sessions concentrate on explaining decisions and failures aloud.

Prepare one pipeline you shipped and one that broke. Be ready to name the grain, freshness promise, retry behavior, late-data policy and the first screen you would open when the job hangs. The data engineering interview track gives those details somewhere to be questioned, which is where production judgment becomes visible.