Why it matters
Cluster migrations are surprisingly common. Companies move from on-premise to cloud, from one cloud region to another, or from one Hadoop distribution to another. Every one of these involves copying a lot of data, and every one has DistCp somewhere in the runbook. The efficiency of the copy is a direct driver of migration timeline and therefore cost.
Cross-cluster disaster recovery is another common use. Nightly DistCp jobs replicate critical directories from the primary to a DR cluster, giving you a warm standby that can take over if the primary fails. Incremental DistCp using snapshots keeps the replication fast even for petabyte-scale directories.
The architecture
DistCp is a MapReduce job written specifically for the copy problem. When you run distcp source destination, it kicks off a driver that first enumerates the source files, groups them into chunks by size, and then launches a MapReduce job whose mappers each get a chunk to copy. The number of mappers is configurable and directly controls parallelism.
Each mapper opens a stream from the source, reads the file bytes, and writes them to the destination. There is no reduce phase because there is nothing to aggregate; the mappers do all the work. When the last mapper completes, the DistCp job succeeds and you have a copy of the source at the destination.
How it works end to end
The workflow starts with the file listing phase. The driver walks the source directory tree, either through native HDFS or the source's filesystem client. For very large trees this can take minutes and consume significant NameNode RPC bandwidth. Options like -useiterator make this streaming instead of building an in-memory list.
Once the file list is ready, the driver partitions files into approximately equal-sized chunks. Each chunk becomes one mapper. Bytes are the unit of balancing, not file count, so chunks may contain many small files or a single large file. This keeps mapper duration roughly equal even when file sizes vary wildly.
Each mapper runs the actual copy. For HDFS-to-HDFS, it uses the native HDFS client on both ends. For HDFS-to-S3 or similar, it uses the S3A filesystem which speaks the S3 API. Cloud copies bring extra concerns: multipart upload, credential management, and the fact that S3 has no atomic rename. DistCp handles these transparently but tuning matters.