Why it matters
An unbalanced cluster wastes capacity. If half the DataNodes are eighty percent full and the other half are twenty percent full, the cluster is effectively at fifty percent utilization but behaves as if it were nearly full, because writes are increasingly directed at only the underused half. Adding more capacity to the cluster does not fix this; the new nodes just become the new empty half while the old nodes stay imbalanced.
Balancing also affects performance. If a hot file happens to have all its replicas on a small set of DataNodes, those nodes become read hotspots while the rest of the cluster sits idle. Balancing spreads the load and keeps aggregate throughput at cluster-wide potential.
The architecture
The Balancer is not part of the NameNode or DataNode process; it is an external command-line tool that runs on any host with cluster access. When you invoke it, it queries the NameNode for the current disk usage of every DataNode, computes the cluster average, and identifies nodes above and below the average by more than a configurable threshold (default ten percent).
For each overused node, the Balancer finds an underused node in a compatible rack topology and instructs the overused node to stream a block directly to the underused node. This is peer-to-peer streaming; the NameNode is informed after the fact so it can update its block map. The Balancer manages many source-target pairs concurrently, subject to bandwidth caps.
How it works end to end
The Balancer runs in iterations. Each iteration it selects the most overused nodes, pairs them with underused targets, and initiates block moves. When blocks arrive at the target, the NameNode updates its block map to reflect the new location. Once an iteration completes, the Balancer recomputes usage statistics and starts the next iteration. It continues until the cluster is within the threshold or a maximum time limit is reached.
The rate of movement is controlled by dfs.datanode.balance.bandwidthPerSec, which caps how much bandwidth each DataNode can use for balancing traffic. Setting this too low means balancing takes forever; too high and it competes with client workloads. A rule of thumb is ten to twenty megabytes per second per DataNode during off-peak hours.
The Balancer does not move blocks in a way that violates rack placement policy. If moving a block would put two replicas on the same rack, the move is skipped. This preserves rack failure tolerance but sometimes means the Balancer cannot fully rebalance a cluster whose rack topology is very uneven.