Why it matters

The reduce phase determines the final output layout of the job. File count, file naming, partitioning, and record format are all controlled here. Getting this right affects every downstream consumer of the output, whether that is another MapReduce job or a Hive query or a Presto scan.

Reducer memory management is also where many jobs quietly waste resources. A poorly written reduce function that materializes all values for a key in memory OOMs on skewed keys. Understanding the iterator semantics is worth learning to avoid this.

Advertisement

The architecture

After shuffle, the reducer holds a sorted stream of key-value pairs. Consecutive pairs with the same key form a group. The reduce framework calls the user reduce function once per unique key, passing the key and an Iterable over the values for that key. This is a lazy iterator; values are streamed from disk or memory as the user code iterates.

OutputFormat controls how emitted records become files on HDFS. TextOutputFormat writes newline-delimited text; SequenceFileOutputFormat writes binary sequence files; ParquetOutputFormat writes columnar Parquet. The OutputCommitter handles the two-phase commit that ensures partial output does not become visible on task failure.

Reduce phase — aggregate values per keySorted key groupsone iterator per unique keyReduce function(key, values) → outputsOutputFormat writes to HDFSOne output file per reducer
Reduce iterates sorted key groups, applies user function, writes through OutputFormat.
Advertisement

How it works end to end

The reduce loop reads one key at a time. For that key, the framework provides a values iterator. The user code typically consumes the iterator once, computing an aggregate (sum, max, top-k), and emits one or more output records via context.write(). The framework moves to the next key when the reduce function returns.

Emitted records flow to a RecordWriter that buffers writes and periodically flushes to HDFS. Writes go to a temporary staging directory. When all reducers complete successfully, the OutputCommitter atomically renames the staging directory to the final output directory. If any reducer fails, its staging output is deleted and the retry runs cleanly.

Reducer heap is split between shuffle buffer and reduce work. Once the reduce function starts, the remainder of the heap is available for user code (aggregation state, output buffers). Sizing reducer memory correctly means understanding both phases.