Why it matters

Shuffle cost usually dominates job runtime. A join over ten terabytes moves ten terabytes across the network at least once — over-provisioned bandwidth is expensive but under-provisioned bandwidth makes every job slow. Understanding shuffle mechanics helps you know when to add network, when to add reducer memory, and when to restructure the job to reduce shuffle volume.

Combiner design, partitioner design, and reducer count all interact with shuffle. Getting them right is worth hours of runtime on any large job.

Advertisement

The architecture

Once map tasks begin completing, reducers start fetching partitions. Each reducer runs mapreduce.reduce.shuffle.parallelcopies fetcher threads (default 5) that pull data over HTTP from map task outputs via the auxiliary Shuffle service running in NodeManagers. This is why the Shuffle service must be running on every NM.

Fetched partitions land initially in reducer memory. When memory fills, older data spills to reducer disk. As more partitions arrive, an in-memory merger and an on-disk merger run continuously to keep the working set manageable and produce sorted intermediate files.

Shuffle + sort — network transfer between map and reduceMap outputssorted per-partition files on local diskReducer fetchersparallel HTTP pulls from each mapperpartition NFetched partitions merge-sorted in reducer before reduce function runs
Reducers fetch partitions from every mapper in parallel, then merge-sort for the reduce phase.
Advertisement

How it works end to end

The fetch phase runs in parallel with the tail of the map phase. As soon as a map task finishes, its output is available for fetch. Reducers report status back to the AM so the AM knows when all partitions have arrived. If a map task fails and needs re-execution, its output becomes unavailable and any reducer that already fetched from it must re-fetch from the retried map.

The merge phase can run entirely in memory if the total partition data fits, or spill to disk if not. The mapreduce.reduce.shuffle.input.buffer.percent setting controls what fraction of reducer heap is available for shuffle buffering; the remainder is reserved for the reduce function.

Once all partitions have arrived and been merged into a single sorted stream, the reduce function begins iterating over key groups. The switch from shuffle-heavy to reduce-heavy CPU usage is visible as a sharp phase transition in reducer metrics.