Why it matters
Most catastrophic Hadoop outages have a ZooKeeper root cause. When ZooKeeper stops working, HA fails to fail over, HBase regions cannot be assigned, and dependent services grind to a halt. Understanding ZooKeeper deployment and monitoring is the difference between a cluster that survives a network blip and one that requires a two-hour recovery.
ZooKeeper is also the source of most Hadoop configuration mistakes. Wrong session timeout, wrong quorum size, wrong storage disk — each of these produces subtle bugs that only manifest under load or during failure.
The architecture
ZooKeeper deploys as an ensemble of three or five nodes running the ZooKeeper daemon. The ensemble uses a majority quorum protocol (Zab) to keep an in-memory data tree consistent across nodes. Clients connect to any node and can read or write the tree; writes must be acknowledged by a majority before returning.
The data tree is organized as znodes, which look like a filesystem tree. Each znode holds a small value (typically under a megabyte) and can have children. Ephemeral znodes are automatically deleted when the client session that created them ends. Sequential znodes get an auto-appended sequence number. These primitives are enough to build leader election, distributed locks, and configuration distribution.
How it works end to end
HA leader election works by creating an ephemeral znode with a well-known name. The first to create it wins; others watch the znode and wait for it to disappear. When the leader dies or its session times out, the znode auto-deletes, watchers get notified, and the next candidate creates the znode and becomes leader.
Watches are one-shot notifications on znode changes. A client can register a watch on a znode; when the znode changes, the client gets a callback. Watches let services react to configuration changes and membership changes without polling.
Sessions have a heartbeat and a timeout. A client that fails to heartbeat within the session timeout has its ephemeral znodes deleted. This is how ZooKeeper detects dead clients. Tuning session timeout is a trade between fast failure detection (short timeout) and tolerance of transient issues (long timeout).