Why it matters
Spark's unification is its biggest value. Before Spark, teams needed MapReduce for batch, Storm for streaming, and Mahout for ML. Now one API and one cluster do all three. The operational simplification alone justifies the switch for most organizations.
Spark's memory-oriented execution also cut cluster costs. Workloads that took 100-node MR clusters now run on 10 Spark nodes, dropping infrastructure spend proportionally.
The architecture
Spark has three main API layers. RDDs (Resilient Distributed Datasets) are the original low-level API — collections of Java/Scala objects partitioned across nodes with lineage-based recovery. DataFrames add schema and Catalyst optimization, resembling SQL tables. Structured Streaming extends DataFrames to unbounded data.
Below the APIs, Spark uses a driver-executor model. The driver plans and coordinates; executors run tasks on worker nodes. Cluster manager (YARN, Kubernetes, standalone) provides the resources.
How it works end to end
A Spark job starts when the driver receives an action (count, save, collect). The driver builds a DAG of stages by analyzing the transformations. Each stage becomes a set of tasks — one per partition. Tasks are shipped to executors and run in parallel.
Between stages, data shuffles across the cluster if the transformation requires it (groupBy, join, etc.). Wide transformations (shuffle-inducing) are expensive; narrow transformations (map, filter) are cheap and pipelined.
Executors cache intermediate RDDs in memory when persist() is called. This is the mechanism behind Spark's speed on iterative algorithms.