Why it matters

Partitioning dominates Spark performance. Understanding is central.

Advertisement

The architecture

Input partitioning: files → partitions. spark.sql.files.maxPartitionBytes controls.

Shuffle partitioning: spark.sql.shuffle.partitions. Default 200; often needs tuning.

Partitioning knobsInput partitionsfrom filesShuffle partitionsspark.sql.shuffleExplicit repartition / coalescemanual controlAQE now handles many partition-count decisions automatically
Partitioning levels.
Advertisement

How it works end to end

Repartition: forces shuffle to N partitions.

Coalesce: reduces without shuffle.

PartitionBy on write: creates directory-per-value structure. Enables partition pruning on read.

Skew: uneven partition sizes. AQE detects and splits.