J-space is (conjecturally) the rate-reduction subspace

A bridge between Yi Ma's sparse rate reduction derivation of the transformer (CRATE) and Anthropic's empirical J-space. Claim: in the rate-reduction frame, J-space is not a surprise — it is the union of occupied low-dimensional subspaces the layers compress toward. The CRATE half is established; the bridge (Section 4) is a conjecture. Even for CRATE the argument in 4a is a heuristic linearization, not a proof — the open lemmas are in 4b.

Symbols: Z in R^{d x n} a representation (d-dim features of n tokens), column z_i per token; eps coding precision; U = {U_1..U_K} orthonormal bases spanning K low-dim subspaces, span(U) = sum_k range(U_k); R(.) the coding rate — see maximal-coding-rate-reduction.

1. Coding rate and its gradient

R(Z) = (1/2) log det( I + a Z Z^T ),      a = d / (n eps^2)
grad_Z R(Z) = a ( I + a Z Z^T )^{-1} Z

Compression term against subspaces: R(Z | U) = sum_k R(U_k^T Z) — the bits to code the tokens after projecting onto each subspace. It sees only the projections U_k^T Z and is blind to the span(U)^perp component; it is small when the within-subspace projections are collapsed to low volume (or when tokens sit in the complement) — not "when tokens lie inside span(U)". The diversity term R(Z) (maximized) prevents within-subspace collapse (gives the occupied subspaces volume) — but R(Z) is U-agnostic (it rewards volume in any direction, complement included), so it does not by itself localize tokens to span(U); that localization comes from the joint optimization over U and the projection operators, not from R(Z).

2. CRATE: one transformer layer = one step of sparse rate reduction

The full objective (CRATE Eq. 1) is min_Z R(Z | U) - R(Z) + lambda ||Z||_1: compress each subspace's projection, expand the whole (the -R(Z) diversity term — load-bearing: without it min R(Z|U) just collapses everything to 0), and sparsify. Alternating minimization gives the two blocks:

(a) Compress -> self-attention. The block is Z <- Z + MSSA, where MSSA approx -kappa * grad_Z R(Z|U) (a step that reduces the compression term — mind the sign: +MSSA = -kappa*grad). Per head k, MSSA carries a term U_k U_k^T Z * softmax( (U_k^T Z)^T (U_k^T Z) ) — multi-head subspace self-attention (MSSA): tokens attend by similarity inside each subspace and are pulled toward it. Ma reads attention as a compression step against U (contested — see the caveat below); note its output lies in span(U) (used in 4b).

(b) Sparsify -> MLP. A proximal step for min ||Z||_1 + (1/2)||Z - D Z||^2 (dictionary D aligned to the subspaces) is ISTA: Z <- ReLU( Z + step*(...) ) = the feed-forward block.

Layer: Z^{l+1} = ISTA( Z^l + MSSA( Z^l ) ). The residual stream is the iterate Z^l, and the residual connection is precisely the Z - kappa*grad (identity + update) shape of a descent step. So a trained transformer approximates an unrolled optimizer driving Z toward span(U) — orthogonal low-dim subspaces, sparse within. (The formulas are faithful to the CRATE paper. But the interpretation "attention = a compression step" is contested: arXiv:2411.17182 shows the Eq-4 approximation — replacing (I+aZZ^T)^{-1} by softmax — is a derivation artifact, and that the MSSA gradient approximation can increase the coding rate (de-compression) rather than reduce it. So even "attention compresses" is unsettled — which independently reinforces 4b-item-1.)

3. J-lens, formally

Let F : h -> future-output logits be the downstream map from an activation h (a column of Z at some layer/position) through the remaining layers and the unembedding. The Jacobian J = dF/dh is |V| x d; row w is grad_h F_w, the direction that most raises the future log-prob of token w. Define J-space = the significant right-singular directions of J — the subspace of activation the output is sensitive to; its readable "contents" are the words w whose row grad_h F_w aligns with the current state.

Two caveats — this operationalization is ours, not Anthropic's (they define J-lens informally): (i) "future-output logits" is not a differentiable function of a single h without an unstated construction (autoregressive, multi-step, resampled) — a teacher-forced continuation or an expected-influence surrogate is being assumed; (ii) J is |V| x d, so its significant directions can span up to d dims — "J-space is small" is Anthropic's empirical finding, not something this formalization guarantees.

4. The bridge (conjecture + derivation)

Claim. J-space = span(U): the output-sensitive directions coincide with the occupied subspaces the layers compress toward.

4a. Heuristic argument (why to expect J-space = span(U)):

1. At/near the optimum, Z in span(U); the complement span(U)^perp is compressed to ~0 (no variance, no coding-rate contribution).
2. Every downstream operator is built from U (MSSA projects by U_k; ISTA uses the U-aligned dictionary). A perturbation dh in span(U)^perp is (i) barely read by the projections — U_k^T dh is small — and (ii) contracted by the next compression step. So one expects dF/dh to have little component along span(U)^perp.
3. A perturbation dh in span(U) is the signal every layer reads and propagates. So one expects dF/dh to be large on span(U).
4. If 1-3 held exactly, the significant directions of J would equal span(U): J-space = the occupied subspaces. They do not hold exactly — see 4b.

4b. Where 4a is NOT rigorous (open lemmas, even for CRATE):

1. Nothing annihilates the complement; the skip connection preserves it (the sharpest hole). MSSA(Z) in span(U) and the update is Z + MSSA, so the identity/skip path carries the old span(U)^perp component forward unchanged; ISTA acts through the dictionary D, not by projecting out span(U)^perp. No CRATE operator drives the complement to 0. So 4a-step-1 ("complement compressed to ~0") is not obviously true even as L -> oo, and at finite depth the complement is plainly O(nonzero). A real argument must exhibit where the complement is suppressed — structurally, it isn't.
2. The nonlinear Jacobian is ignored (the main hole). Steps 2-3 argue only about the linear projections U_k^T. But dF/dh runs through the softmax (couples tokens) and the ReLU/ISTA gate (state-dependent), whose Jacobians can leak output-sensitivity into span(U)^perp. A proof must compute the composed Jacobian dF/dh = prod_l J_l through the nonlinearities and bound its complement component — not done here.
3. Forward contraction is not derivative annihilation. "The complement's value is compressed toward 0" (forward) differs from "dF/dh kills the complement" (derivative). The honest link is chain-rule damping — a per-layer contraction factor c < 1 gives sensitivity ~ c^{L-l} — which is only heuristic, finite (not zero), and unproven to be uniform or not re-amplified by the nonlinearities.
4. No lower bound on span(U). Step 3 asserts dF/dh is "large" on span(U) with no bound; some in-subspace directions may also wash out.
5. Learned U vs optimum U. The trained operators' subspaces need not equal the occupied subspaces of the ideal optimum.
6. The unembedding is not a U-operator. 4a-step-2 claims "every downstream operator is built from U" — but the readout W_U in R^{|V| x d} (activation -> logits) is a separately learned map with no reason to respect span(U). So dF/dh contains W_U's Jacobian, a leak path off span(U) unless the stack already nulled the complement first (which loops back to 1). Breaks step 2 even for CRATE.
7. dim span(U) is not controlled. span(U) = sum_k range(U_k) over K heads x p dims can cover a large fraction of R^d. For the empirically small J-space to equal span(U), the subspaces must be jointly low-rank / heavily overlapping — an unstated assumption; no bound on dim span(U) is given.
8. Which layer's U? Each CRATE layer has its own U^l. J from an activation at layer l integrates U^l, U^{l+1}, ..., U^L and W_U; "J-space = span(U)" never says which span(U), and silently assumes U^l is roughly layer-invariant so the multi-layer Jacobian concentrates on one subspace.

Honest status: 4a shows why one expects J-space ~ span(U) (a linearization heuristic), not a theorem — even for CRATE; and per the caveat in Sec 2, even "attention compresses" is itself contested (arXiv:2411.17182). Making it rigorous = discharging 1-8 as quantitative bounds on the composed nonlinear Jacobian (softmax/ReLU + unembedding paths, layer-varying U) at finite depth, and controlling dim span(U). (Items 1, 6, 7, 8 and the Sec-2 contestation were caught by two independent decorrelated audits, not the author — a live instance of certificate-is-the-subagent-boundary: the generator is the worst verifier of its own proof.)

The three empirical J-space signatures then collapse to one object:

5. Status, gaps, tests

• Conjecture for black-box LLMs: a normally-trained transformer implicitly learns U and lands near the rate-reduction solution, so J-space approx span(U). Then J-space (found in a black-box model) is evidence of implicit sparse rate reduction.
• Gap 1 — labels. MCR2's within-class R_c needs a class partition; an LLM is next-token self-supervised. But CRATE's compression against a learned U needs no labels, so the bridge leans on the unsupervised compress+sparsify, not the supervised rate reduction; the "classes" become implicit context / next-token structure.
• Gap 2 — different cut. U is defined by feature compression, J-space by output sensitivity. The step "operators built from U => output insensitive off U" is not proven even for CRATE (4b.2); for black-box it is doubly open (operators not manifestly U-parameterized).
• Gap 3 — future aggregation. J-lens weights influence on future tokens; the argument needs each layer to preserve the U-structure so the multi-step Jacobian still concentrates on span(U).
• Test. Apply J-lens to a trained CRATE model; predict J-space equals its explicit span(U). In a black-box model, predict J-space aligns with the top subspaces recovered by a rate-reduction / sparse-dictionary probe.

Punchline. Under rate reduction the "global workspace" de-mystifies: it is the low-dimensional signal subspace span(U) that compression keeps, the "automatic ocean" is the complement it throws away, and J-lens reads span(U) because that is where the output's sensitivity lives — which, for CRATE, is plausibly the same subspace the layers were built to compress toward (modulo the open lemmas in 4b). Whether even CRATE makes this a theorem — let alone a black-box transformer — is open and testable.

kin: maximal-coding-rate-reduction · reads-structure-not-output: recognizer-not-generator · certificate-is-the-subagent-boundary