HDFS: one ledger, many disks, data straight from the DataNodes

What it is. HDFS is the Hadoop Distributed File System: a file system that spreads very large files across many cheap machines and lets many clients read them in parallel. It is built for huge data, single files that are themselves huge, sequential reads over whole files, and hardware that fails every day — not for many tiny files, low-latency random access, or files rewritten in place.

The whole picture first. There are only three roles. The NameNode keeps the ledger: which files exist, which blocks each file is made of, and which DataNodes hold each block. The DataNodes hold the actual bytes. The client asks the NameNode for addresses, then talks directly to DataNodes for data. Data never passes through the NameNode.

Its consistency character: CP. There is one logical authority over the metadata. High availability adds a physical active-standby pair, but it does not turn the system into an AP system: there is still only one active writer at a time, and under a partition the minority side goes unavailable rather than forking the metadata ledger. HA shortens recovery; it does not make the system available on both sides of a split.

Contents. origin-and-motivation, block-size-and-metadata-math, namenode, ledger-persistence, datanode-heartbeat-and-death, writer-lease, safe-mode, ha-and-fencing, read-path, write-path, small-files, federation