Ledger persistence: FsImage, EditLog, and the checkpoint that is not a standby

The motivation: memory is fast and power-volatile. The NameNode keeps its ledger in memory for speed, but a power cut or a crash would erase it. Persistence must therefore record every change somewhere durable, without slowing the in-memory ledger to disk speed on every operation.

Two structures, one rule. FsImage is a snapshot of the whole ledger at one moment: the complete directory tree, file-to-block lists, and permissions as of that moment. EditLog is an append-only log of every change after the snapshot, in order. The rule is log first, then apply in memory: a change is appended to the EditLog before it is applied to the in-memory ledger, so a crash between the two leaves the log ahead of memory, never behind it.

Restart is replay. On restart the NameNode loads the FsImage and replays the EditLog in order, rebuilding exactly the ledger as of the last logged change. This is the same structure as the WAL recovery taught for databases: a snapshot plus a log of everything since, replayed in order. The ledger is, in effect, a large file-addressing system — but it is an addressing and catalogue centre, not a bus: data never passes through it, only addresses do.

The log grows, so it must be checkpointed. Left alone, the EditLog grows without bound and restart replay gets slower and slower. Periodically, the FsImage and the EditLog are merged into a new FsImage, and a fresh, empty EditLog starts. That merging job is called a checkpoint.

The Secondary NameNode is not a standby. The checkpointing job is what the Secondary NameNode does: it fetches the FsImage and EditLog, merges them, and hands the new FsImage back. It holds no live ledger, serves no clients, and cannot take over if the NameNode dies. Calling it a backup NameNode is the classic misunderstanding of this component. Under high availability, the standby NameNode does the merging instead, and the Secondary NameNode is not used — see ha-and-fencing.