Maximal Coding Rate Reduction (MCR2)
A principle for what a good learned representation is, stated as an objective on the features rather than a loss on labels. Yi Ma's line; the basis of the white-box deep-learning program.
The measure: coding rate
For a feature matrix Z (columns = the n samples' d-dim features), the coding rate = the bits per sample to encode Z up to precision eps (a rate-distortion / Gaussian-ball count):
R(Z) = (1/2) log det( I + (d / (n * eps^2)) * Z Z^T )
Bigger R = the features fill more volume (more spread / higher effective dimension).
The objective: expand the whole, compress each class
Given a class partition, let R_c(Z) = the sum of coding rates within each class (each class encoded on its own). A good representation should be:
- expansive as a whole —
R(Z)large (features diverse, fill the space); - compact per class —
R_c(Z)small (each class low-dimensional).
So maximize the gap, the rate reduction:
DeltaR(Z) = R(Z) - R_c(Z) -> maximize
(PCA is the one-class special case; MCR2 is the multi-class generalization.)
What the optimum looks like (the payoff)
Maximizing DeltaR forces the classes into orthogonal, low-dimensional subspaces, each maximally spread within itself. So one objective buys both properties at once:
- discriminative — classes are separated (orthogonal subspaces);
- diverse — within a class the features aren't collapsed (high within-class volume), unlike naive cross-entropy which can collapse a class to a point (neural collapse).
ReduNet: the white-box network
Unroll projected gradient ascent on DeltaR: each iteration becomes one layer. The result (ReduNet) is a deep network whose every operator (a linear map + a softmax-like competition between class-subspaces) is derived, not learned by backprop — interpretable by construction. Architecture as an unrolled optimizer of the rate-reduction objective.
Why it matters / connections
- A principled, information-theoretic alternative to contrastive/self-supervised objectives: no negatives, no augmentation tricks — just "spread the whole, compress each part".
- Ties representation learning to rate-distortion theory and subspace clustering.
- Part of the "understand deep nets as optimizing an explicit, measurable objective" program (vs black-box) — kin in spirit to reading a network's structure rather than trusting its output.