HDFS origins: a clone of a paper, and a chain of necessities

Google did not build Hadoop or HDFS. Google built GFS, described in its 2003 paper, and never open-sourced it; its successor inside Google was later called Colossus. What the open-source world got was a clone of the paper, not the system itself.

The clone. Doug Cutting was building Nutch, a web search engine, and needed the same two things the GFS and MapReduce papers described. He implemented them from the papers: the file-system part became HDFS, the compute part became MapReduce. In 2006 the pair split out of Nutch as its own project, Hadoop — named after his son's toy elephant. Yahoo then hired Cutting and funded Hadoop, which is how a paper clone became the industry's default big-data file system.

Why it had to exist: the motivation chain. Each step forces the next, and none of them is optional once the first is accepted.

Step 1: one machine is too small, twice over. A single disk cannot hold the data, and a single machine cannot read it fast enough. At roughly 100 MB/s of sequential throughput, reading 1 TB takes hours. Capacity and bandwidth both fail on one machine.

Step 2: split across many cheap machines. Cutting the data into pieces and spreading them gives parallel bandwidth: many disks reading at once. But splitting creates a new problem of its own.

Step 3: somebody has to remember where every piece is. Once a file is scattered, no single DataNode knows the whole file. So there must be a separate bookkeeper whose only job is the map from files to pieces to machines. That is the NameNode, and it is the reason HDFS has a single logical authority at all.

Step 4: cheap machines fail daily. The whole point of cheap machines is that they are expendable, so a piece that exists only once will disappear. The answer is replication across machines and across racks: every block exists several times, so one machine dying is a repair job, not data loss.

Related. The bookkeeper of step 3 is namenode; the replication of step 4 is run through datanode-heartbeat-and-death; the size of the pieces is block-size-and-metadata-math.