Why it matters
Every Spark performance question can be answered by understanding stages and tasks. Skew, shuffle cost, and executor sizing all show up as stage-level or task-level phenomena. The Spark UI is designed around this model.
The architecture
Action → job. When you call an action (count, save, collect), the driver builds a job. Each job is independent and runs to completion.
Job → stages. The DAG scheduler walks backwards from the action, identifying shuffle boundaries. Each contiguous chunk between shuffles is a stage. Stages within a job can run in parallel where possible.
Stage → tasks. Each stage produces one task per partition. Tasks are identical Java functions applied to different data slices. They can run on any executor.
How it works end to end
The task scheduler receives tasks from the DAG scheduler and dispatches them to executors based on locality preferences and available resources. Locality levels: PROCESS_LOCAL (executor has cached data), NODE_LOCAL (same host), RACK_LOCAL, ANY.
Task execution: executor deserializes task, runs it against local partition data, and returns result (small results serialized back to driver; large results written to disk or shuffled to next stage).
Failure handling: task failure triggers automatic retry (default 4 attempts). Persistent failures fail the stage and abort the job.