Federation: many ledgers under one view, and what it cannot split

The motivation: one ledger's ceiling. A single NameNode's memory and RPC throughput are a ceiling, and high availability does not raise it: HA keeps two copies of the same ledger, so capacity is still one ledger's capacity. As namespaces grow and tenants multiply, one machine's memory and processing eventually run out.

The mechanism: split the ledger, not the directory. Multiple NameNodes each manage their own namespace volume, each with its own FsImage and EditLog, coordinating with nobody and sharing no ledger. DataNodes register with all NameNodes at once, maintaining one Block Pool per NameNode — the set of blocks belonging to that namespace — stored logically separate, never mixed. Clients see a single federated view through ViewFS: a mount table maps paths to namespaces, for example /user to namespace A and /data to namespace B, and each operation goes to exactly one NameNode, the one owning that path.

The benefits. Adding a NameNode adds ledger capacity and RPC capacity, so metadata scales horizontally. Namespaces isolate tenants: one tenant's namespace growing explosively does not affect the others. And each namespace can run its own HA pair independently, so failover in one namespace does not disturb another.

The limits, stated plainly. Federation is not automatic sharding of a single directory: one directory tree cannot be split across NameNodes by itself, and the mount layout must be designed up front. Operations across namespaces — renaming or moving between them — are not supported. And it complements, rather than replaces, the small-file cure: small files all piled into one namespace will still explode that namespace's NameNode, so merging comes first (see small-files). Federation scales namespaces; merging fixes file count.