Streaming's three moves, and why agent interaction stays synchronous
Parent: design-principles
Streaming processing has three moves: a bounded buffer, back-pressure, and degrade-first — shed model cost before dropping words. They are the standard answer when output must stay live while input keeps coming.
- Bounded buffer. A small queue between capture and the model, capped in seconds of audio. It never grows without limit.
- Back-pressure. Measure real-time factor and queue age. When the transcript trails the speaker past a threshold, stop chewing the backlog and resume at the live edge.
- Degrade-first. Before dropping anything, make the model cheaper: smaller model, greedy decoding, shorter context — so words come out imperfect but current. Dropping audio is the last resort, not the first.
But today's agent interaction is generally not streaming. And it is not batch processing either: batching means accumulating a pile of jobs and computing them together for throughput. What agent interaction does is synchronous processing — the user finishes a turn, the whole turn goes up, the whole turn is processed, one answer comes back.
Why synchronous wins here: a user turn in conversation is short, generally no more than a paragraph. The latency streaming saves on a short turn is imperceptible, while the streaming machinery — socket, queue, real-time-factor measurement, degrade ladder — is built for long speech. StandMeet voice is the case in point: a spoken question, a 3-second pause sends it, whole-clip decode takes well under a second. Streaming would add machinery for partials nobody needs.
Boundary: the three moves stay correct for long speech — dictation, meetings, live captions. The test is not "newer is better" but turn length and latency sensitivity: short turns go synchronous; long, latency-sensitive streams earn the three moves.
Transfer: when deciding whether an agent interaction needs streaming, look at how long one user turn runs first. If turns stay paragraph-sized, synchronous is simpler, cheaper, and more predictable — and stays that way until turns get long (continuous conversation, long dictation), at which point streaming earns its complexity.