Core concept

Wide dep = stage boundary (shuffle). Narrow dep = same stage.

Advertisement

How it works

Long DAGs slow to schedule. Checkpoint to break lineage.

Advertisement

Trade-offs + gotchas

Every Spark job.