Why it matters
Map phase performance dominates most MapReduce jobs. If maps are slow, everything downstream waits. Understanding the mechanics — where CPU is spent, where I/O happens, what triggers a spill — is what lets an operator tune a job from three hours to thirty minutes without changing user code.
Many map performance problems are configuration problems. Buffer size, io.sort.mb, io.sort.factor, io.sort.record.percent — these knobs are opaque without understanding what they control. This article demystifies them.
The architecture
The InputFormat is a pluggable component that computes input splits from a data source. TextInputFormat computes one split per HDFS block; SequenceFileInputFormat does the same for SequenceFiles; DBInputFormat pulls from JDBC. Each split represents work for one map task.
A RecordReader inside the map task turns bytes from the split into records. For text input, each newline-terminated line becomes one record. For SequenceFile, each key-value pair is one record. The RecordReader also handles cross-split boundaries, reading a bit past the split end when the record straddles it.
How it works end to end
The mapper receives records one at a time and calls context.write() to emit key-value pairs. Emitted pairs land in a circular buffer sized by mapreduce.task.io.sort.mb (default 100 MB). The buffer has a soft limit (default 80 percent full) that triggers a background spill to disk while the mapper continues writing.
The spill process sorts buffer contents by partition then by key, and writes a spill file to local disk. The partitioner (default hash-based) assigns each key to a reducer partition; sort ensures per-partition records are contiguous. If a combiner is configured, it runs during spill to reduce data volume.
When the map function completes, all remaining buffer contents are flushed, and all spill files are merged into a single sorted output file per partition. The output stays on local disk waiting for reducers to fetch it.