Why it matters

AQE is one of Spark 3's biggest wins. Understanding enables getting most out of Spark.

Advertisement

The architecture

After shuffle, AQE reads statistics. Decides: coalesce small partitions, promote SMJ to BHJ if suitable, split skewed partitions.

AQE optimizationsCoalesce partitionscombine small onesSkew handlingsplit large onesJoin strategy switchSMJ → BHJEnable with spark.sql.adaptive.enabled=true; on by default in Spark 3.2+
AQE features.
Advertisement

How it works end to end

Coalesce: after shuffle, small partitions cost overhead. AQE combines to target size.

Skew: detect partitions much larger than others; split them.

Broadcast conversion: if actual side of join is small, use broadcast even if planner didn't.