Why architecture matters here
Spark architecture matters because Spark performance is 90% about avoiding wrong choices and 10% about clever tuning. A misplaced repartition, a missing broadcast hint, or an unnecessary shuffle turns a well-behaved job into a runaway one. The architecture is what tells you why each choice matters.
Cost matters. Spark jobs commonly run on clusters that cost thousands of dollars per hour. A poorly written job can spend 80% of its time on shuffle instead of business logic. Learning to read a Spark UI — where the driver, executors, stages, and shuffle each contribute — is the fastest way to cut cost by half without changing any business logic.
Reliability matters too. Spark jobs fail in specific ways: driver OOM, executor OOM, disk-full during shuffle, cluster loss due to preemption. Each of these has a signature in the driver logs and the UI. Being able to spot the signature and apply the right fix (broadcast join, adaptive skew handling, larger executors, retry logic) is what separates a data engineer who ships weekly from one who firefights weekly.
The architecture: every piece explained
Walk the diagram top to bottom.
SparkSession. The user-facing entry point. It configures the Spark application (spark.conf), holds the SparkContext, and provides the SQL and DataFrame APIs. When you call spark.read.parquet() or spark.sql(), you are talking to the SparkSession.
Driver. The driver runs your application code. It plans the query, submits jobs to the cluster manager, tracks task completion, and returns results to the caller. The driver holds the query plan, the DAG, the RDD lineage, and any collected results. Driver OOM is a common source of failures for jobs that call .collect() on large results.
Cluster Manager. YARN, Kubernetes, standalone, or Mesos. The cluster manager allocates executors on cluster nodes. It handles resource requests, container lifecycle, and failure recovery for the driver-to-executor boundary. Modern deployments increasingly favor Kubernetes.
Logical Plan. Your DataFrame or SQL query is parsed into a logical plan — a tree of relational operations (Project, Filter, Join, Aggregate) that captures intent. Catalyst analyzes the plan, resolves references, and applies rule-based optimizations: predicate pushdown, constant folding, join reordering, column pruning. This step is CPU-only on the driver.
Catalyst Optimizer. Spark's query optimizer. It applies both rule-based and cost-based optimizations to produce an optimized logical plan. Cost-based optimization uses table statistics to pick better join orders and physical operators. Adaptive Query Execution adds runtime optimizations that use actual data statistics from earlier stages.
Physical Plan and DAG. The optimized logical plan is translated into a physical plan (specific operators like HashJoin vs SortMergeJoin) and then into a DAG of stages. Stages are split at shuffle boundaries — where data must be moved between executors. Within a stage, tasks can run in parallel without communication.
Executors. Each executor is a JVM process on a worker node with a fixed number of cores and memory. Each core runs one task at a time. Executors cache RDD partitions in memory, write shuffle output to local disk, and report task completion back to the driver.
Task slots. Executor cores × executor count = total task slots. If your job has 1000 tasks and you have 100 slots, tasks run in 10 waves. This is where parallelism lives.
Shuffle service. When a stage boundary is crossed, tasks in the previous stage write their output — grouped by target partition — to local disk. Tasks in the next stage fetch their inputs from all previous executors. An external shuffle service (deployed as a separate process on each node) survives executor loss and improves reliability under preemption.
Adaptive Query Execution. AQE (Spark 3.0+) re-plans the query at runtime using actual data statistics from completed stages. It coalesces small shuffle partitions, converts large-vs-small joins to broadcast, and splits skewed partitions. Enabling AQE typically improves job performance by 20-50% with zero code changes.
Storage. HDFS, S3, GCS, or a table format on top (Delta Lake, Iceberg, Hudi). The storage layer determines read parallelism (partition count), predicate pushdown effectiveness (Parquet filters), and update semantics (ACID via table formats).
End-to-end job flow
Trace a job. You call spark.read.parquet("s3://data/events").filter(...).join(smallTable, "id").groupBy("category").count(). The DataFrame chain is lazy — nothing runs yet.
When you call .show() or write to storage, the SparkSession sends the logical plan to Catalyst. Catalyst applies rules: predicate pushdown pushes the filter into the Parquet reader; column pruning drops unused columns; the small side of the join is identified for broadcast. The optimized plan is converted to a physical plan.
The physical plan is broken into stages: (1) scan events + filter + broadcast join, (2) shuffle for groupBy, (3) aggregate + write. Stage 2 requires a shuffle boundary. The driver submits stage 1 first.
Stage 1 tasks run on executors in parallel. Each task reads a subset of the Parquet files (one task per file split), applies the filter, joins with the broadcast side, and writes the result. When all tasks of stage 1 complete, the shuffle write on each executor produces one file per target partition.
Stage 2 tasks run next. Each task fetches its input from every stage 1 executor's local disk (or the shuffle service). AQE may detect that some target partitions are much larger than others (skew) and split them into smaller sub-partitions. Tasks aggregate their portion and complete.
Stage 3 writes the final output to storage. Task completions flow back to the driver; the driver signals success to the caller.
The Spark UI shows every stage, every task, and every shuffle read/write. Reading it is the essential Spark operator skill.