Why architecture matters here
Hadoop architecture matters because the operational choices — replication factor, block size, YARN queue design, HA configuration — determine how the cluster behaves under load, under failure, and under budget pressure. Get one wrong and you either lose data (rare but catastrophic) or waste half the cluster's capacity (common).
Cost is the primary driver at most Hadoop shops. A 500-node cluster is a multi-million-dollar investment; even 10% utilization improvement is worth low six figures per year. Understanding YARN queues, container sizing, and speculative execution is the difference between running one production workload per cluster and running four.
Reliability is the second driver. NameNode failures, DataNode disk loss, and network partitions all have specific remedies built into the architecture. HDFS High Availability with dual NameNodes plus a quorum-based edit log is the standard for any production cluster; teams that skip it discover the gap the hard way.
The architecture: every process explained
Walk the diagram top to bottom.
Client. Applications talk to Hadoop through client libraries. HDFS clients open files, write blocks, and read data; YARN clients submit jobs, monitor progress, and retrieve results. The client is thick — it caches the namespace state, computes block locations, and directly streams data to and from DataNodes without proxying through the NameNode. This is the design decision that lets HDFS scale to thousands of nodes.
Active NameNode. The single authoritative namespace holder. It stores the file system tree, block-to-DataNode mapping, and permissions. All namespace operations (create, delete, rename) go through here. NameNode metadata lives in memory for speed; the on-disk edit log records every change for durability. A 1 PB cluster typically needs about 20-30 GB of NameNode RAM for metadata.
Standby NameNode. An identical process that stays in sync with the active via a shared edit log stored on a JournalNode quorum. Failover from active to standby takes seconds when configured properly with ZooKeeper Failover Controllers. Without HA, NameNode failure means cluster downtime; with HA, it means a few seconds of paused metadata operations.
DataNodes. The workers of HDFS. Each DataNode stores blocks on local disks and heartbeats to the NameNode every three seconds. Blocks are replicated to three DataNodes by default; the NameNode chooses replicas to spread across racks for fault tolerance. When a DataNode fails, its blocks are re-replicated automatically from surviving copies.
YARN ResourceManager. The cluster-wide scheduler. It tracks total capacity, receives resource requests from application masters, and grants containers on specific NodeManagers. ResourceManager HA works similarly to NameNode HA — active plus standby with ZooKeeper coordination.
NodeManagers. One per data node. Each NodeManager reports available resources (CPU, memory, disk), launches containers on request, monitors container health, and reports back to the ResourceManager. Containers are Linux cgroups or Docker containers depending on configuration.
Application Master. Every YARN application (a MapReduce job, a Spark app, a Tez query) has an Application Master container that coordinates the app's own execution. The AM asks the ResourceManager for containers, launches tasks on them, tracks progress, and handles failures. This decoupling — cluster scheduler versus per-app coordinator — is what makes YARN able to run heterogeneous workloads.
Capacity/Fair Scheduler. The plug-in scheduler policy. Capacity Scheduler gives fixed shares of the cluster to named queues; Fair Scheduler dynamically balances shares based on demand. Both support preemption, queue hierarchies, and node labels. Choosing and tuning the scheduler is a major operational lever.
Kerberos + Ranger + audit. The security stack. Kerberos authenticates users and services; Apache Ranger enforces authorization policies across HDFS, YARN, Hive, HBase, and Kafka; audit logs record every access. Without this stack a Hadoop cluster is a multi-user filesystem with no protection.
End-to-end job flow
Trace a job. A data scientist submits a Spark job. The client contacts the ResourceManager, requesting an Application Master container with 4 GB memory and 2 cores.
The ResourceManager consults the Capacity Scheduler, finds a queue with available resources, and grants the container. The AM starts on some NodeManager. The AM then requests executor containers from the ResourceManager based on the job's spark.executor.instances setting.
Executors start on other NodeManagers. Each executor announces its location to the AM. The AM begins scheduling Spark tasks onto the executors. Tasks read data from HDFS.
The read follows a Hadoop-standard path. The Spark task calls the HDFS client, which asks the NameNode for the block locations of the input file. The NameNode returns a list of DataNode addresses for each block. The task connects directly to a nearby DataNode (rack-aware routing chooses the nearest replica) and streams the block data. Metadata never bottlenecks; data flows peer to peer.
Tasks complete; results are written back to HDFS. The AM tracks progress, retries failed tasks, and reports overall status to the ResourceManager. When the job finishes, the AM unregisters. All containers are released. Freed capacity flows back to the scheduler.
Under the covers, DataNodes have been heartbeating constantly. If one fails, the NameNode marks its blocks under-replicated and coordinates re-replication from surviving copies. If a NodeManager fails, its containers are killed and the AM requests new ones. If the ResourceManager fails, standby takes over via ZooKeeper.