Writer lease: one writer per file, and two different heartbeats

The motivation: single-writer, multiple-readers, with data bypassing the NameNode. HDFS files have one writer and many readers. Because data goes directly from client to DataNodes, the NameNode cannot see the write happening. Without an exclusive token, two writers could interleave the same file, and the final block's length and content would be unresolvable: whose bytes count, and how long is the block really?

The five steps. First, the client asks the NameNode to create or append, and the NameNode checks that nobody holds the file, then grants the lease, recording the holder and the last renewal time. Second, the client asks for blocks, and the NameNode allocates a block and a DataNode pipeline. Third, data goes directly to the DataNodes, and a full block triggers a request for the next block. Fourth, the writer keeps renewing. Fifth, close releases the lease.

The key distinction: two separate lines, not one heartbeat. DataNode heartbeats, every 3 seconds, prove only that a storage node is alive. They say nothing about the writing client, which sits outside the cluster. The client therefore renews its own lease separately: a renewal thread renews when the lease period is about half over. Confusing the two lines is the most common misunderstanding of this mechanism.

Expiry, in two tiers. The soft limit is 60 seconds (LEASE_SOFTLIMIT_PERIOD = 60 × 1000): past it without renewal, another client can preempt the lease and run lease recovery, finalising the unfinished last block to a consistent length. The hard limit is 1 hour (60 times the soft limit): past it, HDFS assumes the client has quit, closes the file on its behalf, and recovers the lease. A lease-manager monitor thread checks about every 2 seconds. The manual trigger is hdfs debug recoverLease. And a lease never blocks reading: readers are not gated by who holds the write lease.

The cluster view: a row lock, not a database lock. The lease is per file, not per cluster. One hundred applications writing one hundred different files hold one hundred leases in parallel, each with its own pipeline. Only writers of the same file queue up. That is why HDFS workloads are usually split into one part file per task instead of many processes squeezing into one file. And one writer stalled on acknowledgements stalls only its own pipeline: it does not stall other writers, and it does not stall the NameNode, which moves no data.

Is writing blocking? Half-async, with backpressure. A write() call only copies data into a packet buffer; once a 64 KB packet is full, a background DataStreamer thread sends it, and the application thread can keep filling the next packet without waiting for that packet's acknowledgement. But there are two places it does block: when the buffer or the pending-acknowledgement queue fills, the writing thread stops and waits — that is backpressure, and a slow replica drags the writer through it. And close() and flush() (hflush/hsync) always wait until every acknowledgement is back before returning, so a caller is never told a write finished when the data is not yet stable. If the pipeline breaks mid-write, writing pauses while the pipeline is rebuilt and unacknowledged packets are resent.