Benchmarks & how to validate the harness

Parent: recursive-harness

The point isn't a leaderboard score — it's showing each piece of the architecture actually helps. So: pick benchmarks that isolate the mechanism, then ablate.

Tier 1 — isolates collapse + backtrack + satisfice (constraint planning)

Failure mode = constraint violation; metric = constraint feasibility = the direct output of the CSP-collapse machinery. Known result: models drop sharply past ~7 constraints — exactly the curve collapse+backtrack should bend.

TravelPlanner — itinerary planning, 6 tools, hard/soft constraints.
Natural Plan (Google) — calendar / meeting / trip scheduling = constraint propagation.
COMPASS (2025) — constrained optimization in LLM agents.
DeepPlanning (2025–26) — long-horizon planning with verifiable constraints.
(Even cleaner first: toy CSPs — logic-grid puzzles, N-Queens, graph coloring — to isolate propagation/collapse.)

Tier 2 — gate = mechanically-verifiable done (decompose + verify)

Gate is the test/state.

SWE-bench Verified (gate = tests pass; saturating ~89%) · SWE-EVO / RoadmapBench (long-horizon, multi-file, weeks — tests "gates prevent cross-stage compounding") · AppWorld (verifiable app state).

Tier 3 — long-horizon convergence + process (does the scaffold prevent drift)

GAIA (multi-step tool chains, ~75% ceiling) · UltraHorizon (ultra-long) · LH-Bench — scores process/workflow (did it follow the right stages), not just the end state → directly tests the stages-and-gates claim.

How to validate (the part that matters: ablations)

A single benchmark score proves nothing about which piece works. So:

Baselines: naive single-pass / ReAct / CoT, plus Tree-of-Thoughts / LATS (existing tree-search agents).
Ablate each piece and show degradation:
- −gate (no soft→hard re-grounding) → expect drift on long tasks (validates stages-gates-bound-the-search);
- −backtrack / −no-good learning → expect thrashing / re-exploration (validates conflict-learning-and-backjumping);
- −decomposition (flat search) → expect collapse on complex tasks;
- −explanation (LLM gives yes/no, not why) → expect weak pruning (validates llm-as-fallible-theory-solver).
Measure cost, not just score (LLM calls / tokens) — the architecture trades compute for reliability; show it's Pareto-better than ToT/LATS (same score cheaper, or same cost more accurate).
Track mechanism signals: #backtracks, #no-goods learned, constraint-violation rate vs horizon, how many errors the gates catch — direct evidence the mechanism does what the theory predicts.

Honest note

Real-world benchmarks (SWE-bench/GAIA) are messy and saturating — hard to isolate which piece helped. Validate the mechanism first on constraint-isolated, metric-aligned ones (TravelPlanner / Natural Plan / COMPASS), then move to SWE-EVO / GAIA for the whole.