Why it matters
Cluster architects hit the NameNode heap ceiling for two reasons. First, applications that create millions of small files quickly consume the metadata budget. Second, well-behaved applications running on very large clusters simply accumulate lots of files over years of operation. Either way, once the NameNode heap goes above a hundred gigabytes, garbage collection pauses grow, RPC queue times spike, and operators end up rebooting NameNodes as a coping mechanism.
Federation buys headroom by dividing the problem. Instead of one NameNode holding a billion files, you run four NameNodes each holding two hundred fifty million. Each NameNode gets a manageable heap size, its own edit log, its own HA pair. The DataNode fleet stays shared, so raw storage capacity is not divided; only metadata management is.
The architecture
Each federation NameNode owns a namespace. Namespaces are independent trees, so paths in one namespace do not appear in another. Each namespace has its own block pool, which is a distinct set of block IDs. DataNodes participate in every block pool concurrently: a single DataNode stores blocks from every NameNode's block pool and heartbeats to each NameNode separately.
Historically, clients had to know which NameNode owned which namespace and configure a ViewFs mount table locally to route requests. This was operationally awkward. Router-Based Federation, or RBF, replaces this with a small stateless routing service that presents a unified view. Clients talk to the routers, routers consult a mount table in the state store, and requests are proxied to the correct NameNode transparently.
How it works end to end
A client wanting to read /finance/reports/2026 first contacts a router. The router looks up /finance in its mount table, finds that NameNode A owns that namespace, and forwards the RPC to NameNode A. NameNode A returns the block map. From this point the client talks directly to DataNodes over the block transfer protocol, exactly like non-federated HDFS. The router is not on the data path.
Because each block ID includes a block pool identifier, DataNodes can distinguish which NameNode's blocks belong to which NameNode. The DataNode maintains a separate block report to each NameNode, sending each report only about its own block pool. This lets each NameNode independently maintain its own view of block placement without any inter-NameNode coordination.
Adding a new namespace is straightforward: bring up a new NameNode with a fresh namespace ID, configure it in the RBF mount table, and DataNodes automatically join the new block pool during their next block report cycle. Removing a namespace is more involved because block pools need to be drained explicitly to avoid orphaned blocks.