HA and fencing: one writer, a shared journal, and a lock that decides who is leader

Split-brain, defined. Split-brain is what happens when two sides of a partition both believe they are the leader and both write: the metadata ledger forks into two incompatible versions, and no later merge can reconcile them. Everything in this node exists to make that impossible.

The order is fixed: decide the new leader, then guarantee the old one cannot write. Fencing is voiding the old leader's write authority before the new one starts. It is not an afterthought bolted onto failover; it is the step that makes failover safe.

Three fencing methods. First, ssh fencing: remotely kill the old active's process. It works until SSH itself is unreachable, which is exactly when you need fencing most. Second, epoch fencing through QJM: when a NameNode is promoted, the epoch (term) number is bumped, and JournalNodes accept writes only from the newest epoch, so a still-living old active simply cannot write — the analogy is changing the lock so the old key is rejected. Third, the ZooKeeper lock through ZKFC: only the holder of the lock may be active, and losing the lock means losing the right to write.

JournalNodes are a role, not one machine. A QJM group has 3 or 5 members, an odd number. The active writes its edits to a majority of them; the standby reads from a majority. Losing 1 of 3 changes nothing. Losing a majority — 2 of 3 — stops new writes, because the active cannot commit edits, but it loses no committed data: writes resume when the JournalNodes return or a replacement syncs in. JournalNodes store only edit-log segments on disk, not the whole ledger.

JournalNodes are the WAL. The EditLog is the NameNode's write-ahead log, and in HA mode its storage is the JournalNode group. It records only the active NameNode's metadata changes: no file content, no DataNode data, no client data. And it is not ZooKeeper's own transaction log. Without HA, the EditLog lives on the NameNode's own disk.

ZooKeeper: no per-system adapters inside it. ZooKeeper itself contains no adapters for HDFS or any other system; the adapter lives on the application side. In HDFS it is the ZKFC running beside each NameNode: it health-checks its local NameNode, races for a lock in ZooKeeper, and on winning promotes its NameNode and fences the old one. ZooKeeper's own principle: an ensemble of 3, 5 or 7 servers, one leader and the rest followers, all writes ordered by the leader and committed only on majority acknowledgement (the ZAB protocol), a hierarchical znode tree holding small data (at most 1 MB), persistent nodes and ephemeral nodes — ephemeral nodes are bound to a client session and vanish on session timeout, which is exactly how the HA lock works — and one-shot watches instead of polling.

The synthesis, with one common misreading corrected. What you get from ZooKeeper is a small notebook synchronised for everyone, delivered through a leader-follower, strongly consistent architecture. And a small trusted control sub-cluster — 2 NameNodes, a ZooKeeper ensemble, a JournalNode group — sits above the DataNode data plane. But the ZooKeeper notebook does not record the journal: it records only the leader lock, while the journal and EditLog live on the separate JournalNode group. The separation exists because ZooKeeper keeps its whole dataset in memory and is only fit for small state, while JournalNodes are disk-based, append-only segment storage. Historically QJM replaced a single NFS server storing edits; it did not replace ZooKeeper.

Log consistency is for the standby to catch up, not for learning who the leader is. Election plus journal consistency are both real, but the last link is often stated wrong: log consistency exists so the standby can stay in sync, not so anyone learns who the leader is. Identity comes from the ZooKeeper lock; currency of data comes from the journal.

The standby is simply the other NameNode. Active and standby are states, not fixed machines; without HA there is no standby at all. And the standby is a hot standby, doing three things continuously: tailing the JournalNodes and replaying every edit into its own in-memory ledger (lagging by seconds); receiving heartbeats and block reports from DataNodes directly, so it already knows block locations at takeover; and periodic checkpoint merging. It accepts no client writes.

Failover. The standby's ZKFC wins the ZooKeeper lock, fences the old active, applies the last journal edits, and flips to active. Clients use a logical nameservice name and retry. Immediately here means seconds to tens of seconds, not a cold start.

CP, not AP. NameNode HA chooses CP: election needs a ZooKeeper majority, edit writes need a JournalNode majority, and the old active must be fenced first. On a partition the minority side stalls rather than writing a divergent ledger.

WAL replay, refined. Replay runs continuously before any failover, but replay alone is insufficient: the full guarantee is quorum writes plus the ZooKeeper lock plus epoch fencing. The standby's ledger is the active's ledger at an earlier moment — a prefix of the same log, differing only in time, never in content or order — so failover needs only the remaining suffix, with no merging. Continuous replay buys recovery speed; correctness comes from it being the same ordered log.