Convergence needs an observable target (not just a capable model)

#thread/agent-loop

Observation: the cases that do converge are exactly the ones where the outside world hands over complete information.

/goal vs self-evolution — who carries the hidden state across the observability boundary

• /goal works because the more detailed the spec, the better the agent does — i.e. you are externalizing your hidden state (intent, your current standard, your mood) into the observable spec. Detail = exporting the latent variable across the boundary. The target becomes identifiable → it converges.
• Self-evolution drifts because it must infer the target from behavior, and that behavior is driven by variables the agent can't observe (mood, fatigue, context) — and the source itself is unskilled / inconsistent. The latent variable is never externalized.

Formal: convergence requires the target be identifiable from observations

Let correct behavior be f(X, M), where X = what the agent observes, M = the hidden mood/state (unobserved). The agent can at best learn the marginal:

$$g(X) = \mathbb{E}_M\big[f(X,M)\mid X\big].$$

• g(X) ≠ f(X,M) — it converges to a version averaged over mood (a blur), not the true target;
• the variation it can't explain (mood-driven) looks like irreducible noise from its viewpoint → the target is underdetermined.

So "how can it converge?" — it does converge, but to the confounded marginal, not the truth; the hidden variable becomes noise in its eyes. This is not a sampling problem (more data just samples the blur more precisely) — it's an identifiability / observability problem. The true target stays unidentifiable no matter how much data.

This replaces the previous ceiling

recursion-is-a-phase-transition put the ceiling at model capability (k < 1 needs a strong-enough model). This is a second, harder ceiling:

Even an infinitely capable model can't converge to a target it can't observe. k < 1 needs not just a strong model but a target identifiable from observations.

The decisive cut: is the hidden state carried across the observability boundary?

• /goal — the human carries it (a detailed spec exports the latent variable) → identifiable → converges.
• self-evolution — nobody carries it; mood etc. never enter the observation space → unidentifiable → converges to the confounded blur.

Ashby: a controller can't match variety it can't observe (mood's variety is unobserved) → can't regulate → can't converge (control-methods). POMDP (steering-without-control) but worse — the latent variable makes the target itself multi-valued / non-stationary from the agent's view.

Ways out (all cross that same boundary)

1. Make the latent variable observable (give the agent a mood signal / explicit user labels);
2. Have the human supply the target (the /goal route — the human is the sensor for the hidden state);
3. Accept convergence only to the observable marginal (the blur — a degraded target).

One line

Convergence is gated by the observability of the target, not just model capability. Every case that converges is one where a human externalized complete information across the boundary (/goal: more detail → better). Self-evolution infers the target from latent-variable-driven behavior, so the target is unidentifiable — it doesn't fail to converge, it converges to a mood-averaged wrong fixed point.