Small files: the metadata bill, and the four ways to pay less of it

The motivation is the metadata arithmetic. The bill is in block-size-and-metadata-math: a file, however small, costs at least one file inode plus one block, and file count — not data volume — decides memory. A million 1 KB files cost far more metadata than one 1 GB file, for a thousandth of the data.

It is expensive three ways. Memory first: the NameNode ledger grows with file count, and at the extreme it cannot fit in one machine at all. Startup second: loading the FsImage and absorbing block reports takes longer as the count grows. Processing third: reading a small file still pays the full procedure — ask the NameNode, find a DataNode, open a connection — and that overhead is larger than a few kilobytes of data itself. In MapReduce, one small file starts one mapper, and the scheduling overhead is larger than the computation.

Four ways to pay less. First, merge at write time: batch records and write one large file instead of one file per record. This is the cure; the rest are treatments for files that already exist. Second, container formats: SequenceFile, Avro, and Parquet pack many records into one large file, readable at record level and splittable, so the file count collapses while record access survives. Third, HAR archiving: Hadoop Archives pack many small files into a few large files plus an index, transparent to readers — but packed files cannot be modified afterwards, so HAR is for cold, settled data. Fourth, combine splits at job time: CombineFileInputFormat merges many small splits into one task, cutting the mapper count without touching the files themselves.

What does not help. Adding DataNodes does not help the NameNode at all: the bottleneck is the ledger, and DataNodes hold data, not ledger entries. The scaling answer for the ledger is federation, and even that only helps if the small files are spread across namespaces: a single namespace stuffed with small files will still explode its own NameNode. Merge first, federate second.