한국어
Scaling an agent system from one user to three hundred: concurrency is not the wall — model rate limits and per-session cost are, then session state.essay · standmeet2026.10.15 · essay
2026.10.15·2 min read#essay#agents

The First Wall Is the Quota

Scaling an agent system from one user to three hundred: concurrency is not the wall — model rate limits and per-session cost are, then session state.

Ask what breaks first when a system like mine goes from one user to three hundred, and most people reach for concurrency. Requests per second. Connections. The usual web numbers.

Three hundred users is not a big number for normal requests. But this is not a normal service. Every user is running an agent. So the first wall is not the servers. It is the model.

Rate limits on the model calls — and behind them, the cost of every session. A normal user costs you some bandwidth. An agent user costs you a stream of model calls, all day, and thirty of them starting work at nine in the morning will hit a provider quota long before they trouble your web tier.

The second wall is state. Every user has a long session, and a context that has to live somewhere. Hundreds of those, on machines you do not control, going to sleep, going offline, coming back.

My own systems serve exactly one user — me — on one host. I know precisely which of my assumptions are single-user assumptions, because I put them there. So here is what I would deliberately do differently from day one.

Keep the server light and the client heavy. The less the server holds, the less it can break.

Put quotas in from the start. A quota for each team, rate limits under it. Because the tool is internal, the governance can be human-scale: when a team gets close to its limit, send an email to the team lead. Every knob — more quota, less quota, different limits — can be discussed with the leads and set together. A quota negotiated is a quota nobody resents.

And do not guess the numbers on day one. Nobody knows what three hundred developers' agents actually draw until someone measures it. Pressure-test the system, get the real numbers, and set the quotas from the measurements.

Scale questions sound like questions about size. For agent systems, they are questions about quotas and state. Size is the easy part.

ask the AI about this essay·context: “the first wall is the quota”
›