Block size and the metadata arithmetic: file count, not data volume, decides memory

The default is empirical, not derived. The HDFS block size defaults to 128 MB, and it is configurable with dfs.blocksize. It is an empirical default that balances competing costs, not a value solved from an equation. History: GFS used 64 MB, early Hadoop used 64 MB, Hadoop 2 moved the default to 128 MB, and some clusters holding very large files use 256 MB.

One rule for the arithmetic. Each metadata object — one file inode, one block — costs roughly 150 bytes in the NameNode. That figure is approximate and version-dependent; other guides put it at 150–200 bytes per inode. Use it only for order-of-magnitude estimates, never as an exact heap size.

objects = files + blocks

metadata \approx objects \times 150\ \text{bytes}

The Cloudera official examples. The same amount of data costs wildly different metadata depending on how it is cut. A 192 MB file is 2 blocks plus its inode, so 3 objects:

3 \times 150 = 450\ \text{bytes}

The same 1 GB as one large file is 8 blocks plus its inode, so 9 objects:

9 \times 150 = 1350\ \text{bytes}

The same 1 GB cut into 1,024 files of 1 MB is 1,024 inodes plus 1,024 blocks, so 2,048 objects:

2048 \times 150 = 307200\ \text{bytes}

That is about 228 times the large-file case, for the same 1 GB of data:

307200 / 1350 \approx 228

Scaled to 1 TB. As 8,192 files of 128 MB, the data is 8,192 inodes plus 8,192 blocks, 16,384 objects, about 2.3 MiB of metadata. As 1,048,576 files of 1 MB, it is 2,097,152 objects, 300 MiB of metadata. Same data, 128 times the metadata:

2097152 / 16384 = 128

Why 128 MB, from seek amortisation. A hard disk seeks in roughly 10 ms and streams at roughly 100 MB/s. Transferring 128 MB takes 1.28 s, so the seek is under 1% of the transfer. Transferring 1 MB takes about 10 ms, so the seek is about 50% of the transfer. Blocks must be large enough that seeking to them is negligible next to reading them. But a block too large hurts the other way: less parallelism (fewer pieces to read at once) and coarser recovery granularity (a lost block means re-copying more data).

Never report 150 times objects as heap size. Replicas are locations, not new block objects, but the block map and other per-block overhead mean Cloudera's practical sizing rule is 1 GB of heap per 1 million unique blocks: 36 million blocks need about 36 GB of heap, far above 150 bytes times objects. The 150-byte figure sizes the ledger; the heap rule sizes the machine.

File count, not data volume. 10 PB stored as large files is about 80 million blocks, tens of gigabytes of metadata — one large-RAM server holds it. 10 PB stored as 10 KB files is about 1 billion files, hundreds of gigabytes of metadata — one machine cannot hold it. And large heaps bring their own cost: garbage-collection pauses grow with heap size. A standby in an HA pair only duplicates the same ledger in memory; it does not raise capacity. Beyond one machine, the answer is splitting the ledger, which is federation; the disease itself is small-files.