Shannon entropy (Claude Shannon, 1948)

The average information — equivalently, the average surprise — carried by each symbol drawn from a source. High entropy = unpredictable = every symbol tells you something new; low entropy = predictable = symbols mostly confirm what you already expected.

Formula: H(X) = - sum_i p(x_i) * log2 p(x_i)

• X — the source (a random variable over possible symbols/messages).
• x_i — the i-th possible outcome; p(x_i) — its probability.
• log2 — base-2 log, so H is measured in bits.
• H is maximal (= log2 N for N equiprobable outcomes) when the source is uniform, and 0 when one outcome is certain.

Transfer. Information equals surprise, and a reader's attention is paid only for surprise. An agent that produces the statistically-expected next sentence emits near-zero H relative to the reader's prior — technically fluent, informationally empty. This is the formal core of slop-is-a-context-deficit and anti-mid: mid content sits at the mode of the distribution, so p(x_i) is high and per-token information is low. To be worth reading, raise the entropy of your claims against the reader's model — say the thing they could not have predicted — while keeping the surface predictable enough to parse.

Kin: channel-capacity · mutual-information · redundancy · models · anti-mid · slop-is-a-context-deficit