Why it matters
When the NameNode is healthy, HDFS feels magical: clients open files, list directories, and stream data at wire speed. When the NameNode is unhealthy, everything else is dead weight. A cluster with hundreds of DataNodes and petabytes of blocks is worth nothing if the NameNode is stuck in a garbage collection pause or has lost quorum with its JournalNodes. This is why the on-call playbook for any HDFS cluster is dominated by NameNode topics: heap, edit log, checkpoint cadence, failover, and safe mode.
Cost also lives here. NameNode hardware is disproportionately expensive per node because it needs large RAM and fast local disks, but you can only have one active at a time. Scaling past a few hundred million files requires either federation, which splits the namespace across multiple NameNodes, or a full migration to a different filesystem like Ozone.
The architecture
The NameNode holds the namespace as a tree of inodes in JVM memory. Every directory, every file, and every block is an inode-like object taking roughly one hundred fifty bytes. On startup, it loads a checkpoint file called the FsImage from local disk and then replays every mutation in the edit log to bring itself up to the last transaction. Once loaded, the NameNode is authoritative for the filesystem.
The edit log is the durability mechanism. Every create, delete, rename, append, and permission change is written to the edit log as an append-only record before the mutation is acknowledged to the client. Without the edit log, a crash would lose in-flight metadata. With it, a crash simply requires replaying the log during recovery. In a high-availability deployment the edit log lives on a quorum of three or five JournalNodes, which are themselves lightweight Java processes serving the log to both the active and the standby NameNodes.
How it works end to end
Clients contact the NameNode via an RPC protocol. When a client creates a file, the NameNode allocates an inode, chooses three DataNodes for the first block, appends the create record to the edit log, waits for the JournalNode quorum to acknowledge, and then returns the block target list to the client. The client streams the block data directly to those DataNodes and returns to the NameNode for the next block. The NameNode is never on the data path, only the metadata path.
The standby NameNode continuously tails the edit log by reading from the JournalNode quorum. It replays every mutation against its own in-memory namespace, so at any moment it is within milliseconds of the active NameNode's state. Periodically the standby takes a checkpoint: it dumps its FsImage to local disk and truncates the portion of the edit log that has been captured. This keeps startup time bounded even for clusters with billions of mutations in history.
Failover is coordinated by ZooKeeper. Both NameNodes maintain a ZooKeeper session and race to acquire an active lock. If the active dies, its session expires, ZooKeeper releases the lock, the standby grabs it, and it announces itself as the new active. Clients discover the switch by using logical service names that resolve to the active endpoint. Total failover typically completes in twenty to sixty seconds.