Grounding the Greek letters in token probabilities

Parent: recursive-harness

The theory runs on α, ε, δ, k — measured indirectly (mutation tests, residual ratios). But the LLM emits REAL numbers at every step: per-token logprobs. Can those numbers feed the theory? Yes — through three bridges, behind one warning sign.

The trap first: token probability ≠ correctness probability

Three known failure modes:

• RLHF destroys calibration — base models' logprobs are surprisingly well-calibrated (e.g. on multiple choice); post-RLHF models are badly overconfident (the famous GPT-4 tech-report calibration plots);
• surface-form competition — one meaning has countless phrasings, so sequence probability is diluted across paraphrases; low $P(\text{sequence})$ does not mean wrong;
• confident hallucination off-manifold — precisely where the model leaves its competence landscape (coupling-to-the-learner), high logprob and wrongness coexist.

Conclusion up front: raw logprobs are pre-gates and budget routers, never acceptance — the metric proposes, the gate disposes (soft-identity-transfer's two-tier rule).

Bridge 1: semantic aggregation → an α̂ signal

Sample n answers, cluster by semantic equivalence (entailment both ways — note this is soft equality again), and compute entropy over the CLUSTERS: semantic entropy (Farquhar et al., Nature 2024). It defeats surface-form competition and is a far better hallucination signal than raw logprob. This is the rigorous form of the "self-consistency as cheap pre-gate" idea in coupling-to-the-learner: low semantic entropy → proceed; high → route to a stronger gate or a further split, before real verification budget is spent.

Bridge 2: calibration → p's that may enter the ledger

Calibration error (ECE) is measurable, and repairable with a small labeled set (temperature scaling, isotonic regression). Only a calibrated confidence deserves to touch the α ledger of root-protocol. And the harness has a free calibration set: the gates' own historical verdicts — every gate decision is a labeled example of "model claimed p, truth was y." A beautiful loop: the longer the harness runs, the better calibrated its learner's numbers become — gates manufacture the ground truth that makes logprobs meaningful.

Bridge 3: entropy localization → a token-level readout of ◇/□

The per-token next-token entropy profile has modal meaning: long low-entropy stretches = □ (the context forces the continuation — inertia); high-entropy spikes = ◇ (a genuine choice point). modal-status-labels's labels acquire a real-number, token-level readout. Consequence for budget: concentrate verification on the high-entropy spans — that is where the model actually chose (and could have chosen wrong); low-entropy stretches are coasting. This also instruments in-stage search (non-determinism-and-infinite-branching): branch points = entropy spikes.

A direct instrument: perplexity as the terrain probe

coupling-to-the-learner said ε is a landscape but gave no meter. Perplexity of the task/context under the model is a real-valued familiarity probe: high perplexity = off-manifold = expect high ε. So the P of P×C gets a partial instrument, and probe-then-split (split-granularity-equilibrium) gains a zeroth step: read the perplexity before spending real probe budget.

The dashboard extension (table): signal | feeds | caveat

• semantic entropy (n samples, cluster, entropy) | pre-gate α̂, routing | costs n samples; clusters need the entailment judge
• calibrated confidence (ECE-corrected) | the α ledger | needs labeled set — use gate history; recalibrate per task family
• token entropy profile | ◇/□ labels, verification allocation | entropy ≠ importance always; some low-entropy tokens are load-bearing
• perplexity of context | ε-landscape probe, P×C | familiarity ≠ correctness; watch Goodhart if optimized against

One line (bold)

The LLM never delivers just an artifact — it delivers (artifact, a stream of real numbers); aggregated, calibrated, and localized, the numbers measure the terrain, label the choice points, and route the budget — but acceptance belongs to the gate, always.