Why it matters

Dynamic allocation saves resources. Understanding enables efficient multi-tenant Spark.

Advertisement

The architecture

Enable: spark.dynamicAllocation.enabled=true.

Range: spark.dynamicAllocation.minExecutors, maxExecutors.

Dynamic allocation flowBacklog taskno executorRequest moreup to maxIdle timeoutrelease executorRequires external shuffle service so executors can be removed without losing shuffle files
Dynamic allocation cycle.
Advertisement

How it works end to end

External shuffle service required: executors can be removed without losing shuffle output.

Requesting new: when pending tasks exceed current capacity.

Removing: idle for timeout period.