DataNode heartbeat, death, decommission, and the client face
The core mechanism: DataNodes push, the NameNode never polls. Every 3 seconds each DataNode sends a heartbeat: a small packet carrying only status — that it is alive, how much space is free, and what transfers are in flight. The NameNode does not ask; it only updates the last-seen time for that node. Heartbeats are timed independently per node, so the load is spread out instead of arriving in one burst. And the NameNode piggybacks commands on heartbeat replies: replicate this block, delete that block. The heartbeat is therefore also the command channel.
Death is declared slowly, on purpose. A node is declared dead only after 10 minutes 30 seconds without a heartbeat. That is intentionally slow: a garbage-collection pause or a network jitter must not trigger a false death, because a false death causes useless re-replication of data that was never lost. Patience here is cheaper than the repair work a false alarm would cause.
Block locations come from a separate channel. Heartbeats say nothing about which blocks a node holds. That knowledge comes from block reports: a full report at registration, incremental reports when blocks change, and a full report every 6 hours by default. Keeping the two channels separate is what makes the design scale.
Why it scales, and where the ceiling is. The heartbeat packet is small, and the heavy block list is infrequent and separate, so the system survives to thousands of nodes. The ceiling is the ledger, not the heartbeat. Two terminological points matter here. First, what is small is the heartbeat packet, not the data block: a block is the storage unit (128 MB), a packet is the transmission unit, and a block is itself split into packets when transferred. Second, a large block helps scaling rather than hurting it: block count per node, block-report size, and ledger entries all stay bounded when blocks are large, and all three explode if blocks are tiny.
Planned decommission: copy first, then leave. When a node must come out for maintenance, the admin puts it on the decommission (exclude) list and runs hdfs dfsadmin -refreshNodes. The NameNode stops placing new blocks there, copies its existing blocks elsewhere one by one, and only marks it Decommissioned when enough replicas exist elsewhere. Meanwhile the node keeps heartbeating and keeps serving reads, and removing it from the list recommissions it. Contrast sudden death, below: sudden death is declare dead, then emergency-copy; planned decommission is copy first, then allow shutdown.
Replication is block by block, never a whole-disk clone. A retiring node's 1 TB is 8,192 blocks of 128 MB, and each block gets its own target, chosen from the free space reported in heartbeats, rack diversity, and load. A node with 1 GB free holds at most about 8 blocks, and a nearly full node is skipped entirely; 30 GB holds roughly 200 or more; a 3 TB node holds many but never all 8,192. The NameNode moves no data itself: it tells the target, via a heartbeat command, to pull the block from a surviving source. One precondition is absolute: total cluster free space must cover the data, or decommission stays In Progress.
A stalled decommission is a deliberate fail-safe, not a bug. Note that refreshNodes succeeding only means the list was read — it is not a completion acknowledgement, so there is no immediate command-line error when space is short. The node visibly stays Decommission In Progress (via -report or the web UI) indefinitely; its blocks stay in place and readable; a new replica only counts once fully written and reported, so nothing is silently lost. Data is endangered only if an admin force-shuts the node before decommission completes, which manufactures a sudden death. The stall is stuck-but-safe, which beats falsely-complete-and-lossy; it resumes automatically when space is freed or nodes are added, and cancelling the decommission restores the node.
Why there is no hard pre-check, though the admin could do the arithmetic. The user-proposed third option — pre-check, since the NameNode knows every node's remaining space — is correct in principle, but the core flow has no hard gate, for two reasons. The NameNode knows a momentary snapshot, not the future: other writers, deleters, re-replication and the balancer all change free space during a long decommission. And it schedules per block without reserving space for the whole job against competing normal writes. So the up-front arithmetic is an admin and monitoring step before starting, not a built-in reject-decommission check.
Dynamic space and mixed workloads. A DataNode reports total, HDFS-used, non-HDFS-used and remaining space on every heartbeat, so a co-resident workload suddenly eating disk shows up within seconds and the NameNode stops placing new blocks there. Defences: reserved space (dfs.datanode.du.reserved) is deducted before HDFS computes remaining; a write that lands after space vanished fails and is retried on another node; existing blocks are not evicted by other programs' usage. The conclusion is practical: production prefers dedicated HDFS machines, because mixed use makes capacity unpredictable.
Slow nodes and heterogeneous IOPS. The write pipeline runs as fast as its slowest hop; a node that times out is kicked out and the pipeline rebuilt without it. Reads hitting a slow nearest replica are mitigated by hedged reads: after a short wait, issue the same read to another replica, and the first reply wins. The honest limit: placement uses space, rack and load, it does not measure IOPS, so a slow node is only discovered at runtime. Production answers with homogeneous nodes, or with SSD and HDD storage tiering.
The client face. The user's face is the client, not any DataNode. Shell commands (hdfs dfs -ls, -put, -get, -cat) run on any machine with a Hadoop client and a configuration pointing at the NameNode address, for example hdfs://namenode:9000. Behind them, DFSClient splits each command into metadata RPC to the NameNode and data transfer directly to DataNodes. Admin commands go to the NameNode too (hdfs dfsadmin -refreshNodes, -report); programs use the Java FileSystem API or WebHDFS, which is the same DFSClient logic underneath; the NameNode web UI is view and manage only, and no data passes through it.
Sudden death: already replicated, so it is a repair, not a loss. Each block already has 3 replicas on different nodes and racks. The heartbeat stops, after 10.5 minutes the node is declared dead, its blocks drop from 3 to 2 in the ledger, reads continue from the other two, the NameNode queues re-replication from a surviving replica back to 3, and a write in flight rebuilds its pipeline excluding the dead node without rewriting already-acknowledged packets. The only no-way-out cases are all 3 replica locations dying together (for example replicas that never crossed racks) or a replication factor of 1 — then only an off-cluster backup helps.
Re-replication queueing, resurrection, and corrupt replicas. After a death, re-replication is queued by priority and rate-limited, not fired all at once: the most under-replicated blocks go first, and the rate is capped so repair traffic does not starve client traffic. A node that returns re-registers, and excess replicas created in its absence are deleted once the ledger is reconciled. And a corrupt replica — a checksum mismatch reported by a client — does not count toward the replication factor: it is marked bad, reads fall through to a good replica, and a good copy is re-replicated in its place.
Typical node sizes and decommission duration — estimates, not disclosed figures. Early Hadoop nodes were 4–12 disks of 1–2 TB, 8–24 TB per node. Modern large clusters are 12–24 disks of 10–18 TB, 120–400 TB per node, in practice often capped at 100–200 TB per node because rebuild and decommission time grow with node size; small cloud clusters can be a few TB per virtual machine. Duration follows one formula: time is roughly data to move divided by the aggregate replication bandwidth the cluster can spare in parallel. Rough examples: a 1 TB node in a 10-node cluster at about 1 GB/s aggregate takes about 17 minutes in theory, 30–60 minutes in reality; 50 TB at about 2 GB/s takes about 7 hours; 200 TB at about 5 GB/s takes about 11 hours. Rule of thumb: at 100 TB scale with 10 GbE, decommission is measured in hours — not minutes, not days.
Elastic membership. Cluster size is not fixed. A new DataNode started with the NameNode address registers, heartbeats, reports blocks, and immediately receives new placements, with no NameNode restart. But the scaling driver is external — operations, cloud autoscaling, or Kubernetes. HDFS itself never auto-provisions machines. And shrinking is slow elasticity, not instant: a node can only leave after its blocks are copied, so a 1 TB node leaves far slower than a 10 GB one.