Why it matters
Combiners are the single highest-leverage optimization in most MapReduce jobs. A ten-line combiner can cut shuffle time by 90 percent, which cuts total job runtime by often 50 percent or more. Understanding combiner semantics is more important than any tuning knob.
Combiners also reduce network cost, which matters on cloud clusters where cross-AZ bandwidth is metered. In some deployments, the cost savings from combiners are significant enough to justify the analytical effort.
The architecture
A combiner is a class extending Reducer that runs on mapper output before shuffle. It has the same interface as the reducer: takes a key and an iterator of values, emits key-value pairs. The framework runs it opportunistically during spill file writes and during the final merge at end of map.
Combiners are not guaranteed to run. The framework may skip them if buffer size or spill count does not warrant. This means the combiner must produce the same key-value type as the mapper output, and its logic must be safe to run zero, one, or multiple times per key.
How it works end to end
During spill, the framework reads records from the circular buffer partition by partition, sorts them by key within each partition, and writes to disk. Between sort and write, the combiner is invoked on each key group. It receives the values for one key and emits a smaller set of values (typically one). The emitted records are what actually land in the spill file.
The same combiner also runs during the final merge at end of map, when all spill files are combined into the single output file per partition. If a combiner is safe to run repeatedly, this second application squeezes out even more redundancy.
On the reduce side, the reducer receives combined intermediate pairs and applies the reduce function. For associative operations, combining partial aggregates gives the same final result as reducing all raw values.