Core concept
df.coalesce(10) // avoids shuffle, only reduces
df.repartition(200) // full shuffleAdvertisement
How it works
coalesce quick but may skew. repartition costs shuffle.
Advertisement
Trade-offs + gotchas
Reduce output files: coalesce. Rebalance: repartition.