Skip to main content

71  Benchmark Ratchets and Anti-Goodhart Evidence

71.1 Chapter status

Field	Value
Chapter ID	benchmark-ratchets-and-anti-goodhart-evidence
Part	Part IV - Evidence, Implementation, and the Living Book
Status	conceptual
Manuscript maturity	v0.3 manuscript draft
Last updated	2026-07-31
Primary source records	benchmaxxing, rmi, rgs, cognitive_loop_closure, uat, coherence_exchange, tokenmana, moecot, road_to_agi, project_theseus_whitepaper, theseus_architecture_gate, theseus_self_evolution_system, theseus_circle_transfer, reflexive_router_whitepaper
Claim label	Design rationale
Evidence level	argument
Source queue	primary: benchmaxxing, rmi; supporting: rgs, cognitive_loop_closure, uat, coherence_exchange, tokenmana, project_theseus_whitepaper, theseus_architecture_gate, theseus_self_evolution_system, theseus_circle_transfer, reflexive_router_whitepaper, relational_dimension_compiler; connector/recovery: moecot, road_to_agi
Source loading state	source notes: benchmaxxing, deterministic_capability_compilation, platonic_world_model, rmi, rgs, cognitive_loop_closure, uat, coherence_exchange, tokenmana, moecot, road_to_agi, project_theseus_whitepaper, theseus_architecture_gate, theseus_self_evolution_system, theseus_circle_transfer, ext_mem0_2025, ext_mlcommons_croissant_1_1_2026, ext_inspect_ai_2024, ext_agentdojo_2024, ext_darwin_godel_machine_2025, ext_adas_2024, ext_dynamic_compute_recurrent_transformers_2026, cca_project, moecot_manifest_project, beastbrain_project, bugbrain_project, corbens_trainer_project, corbens_best_model_possible_project, ext_claw_swe_bench_2026, ext_muse_unlearning_2025, ext_unlearning_benchmarks_weak_2024, ext_openunlearning_2025, reflexive_router_whitepaper, kernel_english_residual_compiler, ext_swe_rebench_v2_2026, ext_mlperf_training_v6_2026, relational_dimension_compiler, ext_cultural_alignment_llms_2024, ext_multilingual_evaluation_state_2026, portia_synapse, spider_synapse, capability_ratchet_whitepaper, attd, temporal_coil_research, regret_engine, ext_elizaos_agent_runtime_2026; raw cache: benchmaxxing, rmi, rgs, cognitive_loop_closure, uat, tokenmana, road_to_agi, temporal_coil_research; connector/recovery: coherence_exchange, moecot
Test state	benchmark_ratchet_record.valid.json passes repository-level protocol fixture validation; AsiStackProofs.BenchmarkRatchets implements three derived readiness-promotion, regression-floor, and contaminated-review consequences; python3 scripts/validate_benchmark_antigoodhart.py validates synthetic saturation, holdout/contamination/mutation-or-transfer, regression-ref, negative-result, and promotion-blocking semantics; python3 scripts/validate_benchmark_fixture_bridge.py separately validates experiments/benchmark_antigoodhart/results/2026-07-02-fixture-bridge.json against 2 valid fixtures and 5 expected-invalid controls. The former copied Lean summary mirror is retired. Empirical benchmark runs, hidden holdout execution, contamination audits, policy-training runs, steward release execution, source-reported replay, and current Theseus report verification remain planned.

71.2 Drafting guardrail

No benchmark success is reported here. Source-reported benchmark commands or results remain source-reported until artifacts are imported or reproduced. Local validation in this book checks the book repo, not ASI capability.

Executable specs can say what a proof or schema certifies. Measured results carry a different burden before they can move a capability claim. A green test, leaderboard number, source-reported command, or local fixture is not enough by itself; the stack needs run records, baselines, regressions, residuals, anti-Goodhart checks, and promotion decisions that remain visible after the result is summarized.

Benchmark ratchets are the public evidence counterpart to procedural memory. Procedural memory asks what the system may reuse. Benchmark ratchets ask what the system may believe about the reused capability. Both are memory systems, and both can fail by forgetting negative cases.

71.3 Human Reading Path

Concrete lens. The leaderboard baseline ranks equal scores equally. The instrument lease distinguishes clean, contaminated, saturated, and selected histories.

Executable specs say what a narrow artifact can certify before use. Benchmarks ask how measured performance should change future behavior without becoming a game. Benchmark ratchets can move a floor forward only when regressions, scope, baselines, and negative results remain visible.

The anti-Goodhart theme matters throughout this layer. A score is not evidence by itself. It becomes useful when the stack records what was measured, what was not measured, what got worse, what incentives changed, and what claim the result is allowed to support. Improvement should become cumulative without allowing the scoreboard to become the mission. A failed benchmark should teach as much as a win when the ledger is honest.

The measured frontier should become a disciplined memory, not a target to exploit. A benchmark only strengthens the stack when it keeps failures useful after success.

A ratchet should make regression harder to hide, not merely victory easier to announce, because the evidence floor matters more than the score that clears it. Regression floors discipline winning scores by preserving what improvement forgot earlier.

71.4 Problem

Capability claims need pressure, but fixed benchmarks can become traps. A system can overfit a public style, erase regressions, optimize cheap metrics, or confuse saturation with general competence. Benchmaxxing reframes benchmarks as temporary pressure surfaces. RMI adds residual escrow and regression floors.

The benchmark ratchet exists to make progress cumulative without making claims inflationary. A frontier should push the system into new work. Once the frontier becomes easy, it should become a floor. The score matters less than the transition record that says what improved, what stayed fragile, and what must never regress silently.

Self-improving stacks make benchmark pressure especially sharp. The system may learn what the benchmark rewards, route around expensive checks, or optimize the presentation of progress rather than the underlying capability. Benchmarks have to create memory: baselines, negative cases, residuals, floor regressions, reviewer notes, and downgrade triggers that survive after the headline score stops being novel.

71.5 Why existing approaches are insufficient

Benchmarks can saturate, be gamed, erase regressions, or fail to represent real requirements. A result is especially dangerous when it lacks command, environment, baseline, residuals, negative results, holdouts, transfer checks, and contamination notes. The stack must make benchmark state explicit enough to decide whether a result can update readiness.

External benchmark science gives this layer its comparison set. MMLU (ext_mmlu_2020), BIG-bench (ext_bigbench_2022), HELM (ext_helm_2022), GPQA (ext_gpqa_2023), SWE-bench (ext_swe_bench_2023), SWE-rebench V2 (ext_swe_rebench_v2_2026), LiveBench (ext_livebench_2024), Dynabench (ext_dynabench_2021), CheckList (ext_checklist_2020), contamination analysis (ext_benchmark_contamination_2023), and Goodhart variants (ext_goodhart_variants_2018) show broad, living, adversarial, behavioral, contamination-aware, and proxy-failure pressures. SWE-rebench V2 is especially useful because its setup and clarity-filter ablations expose how one-shot environment generation, brittle tests, implicit naming, and external context can manufacture false negatives. Its released tasks and automated annotations remain a substrate, not a validity certificate or reproduced local benchmark.

The important distinction is between a benchmark as an event and a benchmark as a managed instrument. An event produces a score. An instrument produces a history: what was tried, what changed, what regressed, what saturated, what transferred, and what still failed. The score can be useful, but only the history can support governance. Without that history, the system has no way to tell whether it learned the capability, learned the test, learned a presentation pattern, or merely found a path around the measurement.

Goodhart pressure also appears before anyone is acting maliciously. If the easiest visible metric is speed, the system may drop verification work. If the easiest visible metric is pass rate, the system may avoid difficult cases. If the easiest visible metric is a public benchmark, the system may shape memory, examples, prompts, and training pressure around that surface until the benchmark stops measuring the thing it was meant to measure. A ratchet is the countermeasure: every success preserves a floor, exposes residuals, and asks for a harder frontier instead of turning one number into a broad capability claim.

This also protects the reader. A living book can accumulate validation badges, generated fixtures, source notes, and local renders, but those are not interchangeable forms of evidence. The ratchet record should tell the reader whether a result is a book-build check, schema fixture, synthetic example, source-reported benchmark, reproduced benchmark, or empirical capability result.

71.6 Core Claim

[benchmark-ratchets-and-anti-goodhart-evidence.core, label: Design rationale, support: argument] Benchmark Ratchets and Anti-Goodhart Evidence owns a construct-, task-, dataset-, metric-, harness-, model-, checkpoint-, output-, evaluator-, baseline-, retry-lineage-, budget-, environment-, claim-axis-, and time-specific Benchmark Instrument Lease: a score or evaluation event may update only the exact claim whose construct validity, target capacity, data and metric provenance, output binding, contamination and public-calibration boundary, strong baselines and negative controls, complete selection and failure lineage, regression floors, frontier state, uncertainty, costs, causal checks, transfer, residuals, and decision authority survive review; a green fixture, synthetic probe, source-reported result, leaderboard gain, held-out score, saturation label, or archived winner alone confers no capability, safety, readiness, deployment, unlearning, support, transfer, or SOTA authority.

Reader claim. Two benchmark runs can have the same pass count and opposite evidentiary value when one is clean and the other is contaminated, selected, or saturated.

Operational rule. Bind each score to construct, dataset, harness, checkpoint, output, evaluator, baseline, retry lineage, contamination state, regression floors, uncertainty, cost, and claim axis. Quarantine persists until the causal problem is repaired; aggregate score cannot wash it away.

71.6.1 Worked benchmark collision: same score, opposite decision

Candidate A and candidate B pass the same number of benchmark cases. Candidate A was evaluated once on a clean held-out set with its failures preserved. Candidate B selected its checkpoint and prompt after public calibration leakage, then reproduced the same aggregate count. A score-only ratchet treats them as equal. The instrument lease admits A’s narrow claim update and quarantines B because the lineage and contamination state differ.

The finite benchmark model includes an explicit same-pass-count witness with opposite promotion admissibility. It explores 19 reachable states and 114 transitions, preserves quarantine across 12 arbitrary suffixes, and rejects 15 lifecycle plus 11 semantic mutations. This proves that aggregate count is insufficient for the modeled decision. It does not establish construct validity, hidden-holdout integrity, contamination detection, benchmark quality, capability, or anti-Goodhart effectiveness.

The claim remains at argument support. The assigned sources support design discussion of benchmark lifecycle, anti-Goodhart controls, residual escrow, regression floors, readiness gates, and the mined distinction between metrics that were measured, derived, modeled, fixed, proxied, or never causally connected to a returned output. They do not prove benchmark gains in this repository.

71.6.2 Claim-source mapping status

Appendix C now records thirty-one exact passage-reviewed mappings across thirty-six assigned sources, including the six historical projects as one related local implementation lineage. Those mappings make the source map traceable enough for drafting and support review, but they do not move the claim above argument support. They also do not convert source-reported benchmark commands, local-project status notes, schemas, fixtures, or finite Lean predicates into benchmark evidence.

Source ID	Reviewed support	Boundary
benchmaxxing	Benchmark lifecycle, wall diagnosis, frontier-to-regression transition, anti-Goodhart safeguards, holdouts, mutation, transfer checks, contamination audits, and ledgers.	No benchmark harness, mutation test, holdout run, transfer check, contamination audit, live benchmark, empirical result, or model-capability result.
rmi	Frontier/floor discipline, residual escrow, mastery thresholds, critical-failure vetoes, public calibration, and benchmark/model/residual ledgers.	No RMI prototype inspection, benchmark run, specialist-arm evaluation, router benchmark, safety result, or capability result.
cognitive_loop_closure	Repeated trajectories, tool creation triggers, parameter discovery, verification/routing/monitoring, tool cards, and retirement criteria.	No loop detector, tool synthesis run, verification harness, regression suite, runtime monitor, or procedural-memory benchmark.
uat	Retrieval dossiers, proposition tiers, unsupported-claim deletion, bounded adversarial review, double-lock termination, and atomic compression.	No local tribunal pipeline, proposition extractor, dossier verifier, adversarial-review run, or benchmark adjudication run.
coherence_exchange	Source-note context for claim/evidence units, verification supply chains, audit, fork/exit, and contestability.	Source-note only; no implemented economics, governance mechanism, external corroboration, security evidence, or benchmark evidence.
tokenmana	Latency, load variance, human friction, regenerative capacity, hard quotas, temporal compression, and temporal-elasticity hypotheses.	Theoretical/source-reported only; no theorem formalization, simulation, pricing experiment, human study, infrastructure result, or quality result.
moecot	Source-note runtime context for compact orchestration, specialist lanes, fail-closed ledgers, readiness gates, replay, promotion blockers, and residual tracking.	Source-note only; no runtime artifact, benchmark artifact, log, release evidence, or reproduced benchmark result.
road_to_agi	Source-note roadmap/status context for remaining work, source-reported commands/results, readiness status, and promotion blockers.	Source-note only; no reported command, benchmark result, readiness result, or roadmap status was reproduced.
project_theseus_whitepaper	Report-first benchmark states, candidate gates, residual escrow, loop closure, governed data ingress, safety boundaries, and observability caveats.	No current Theseus reports, commands, numbers, dashboard state, model artifacts, or benchmark ledgers were rerun or independently inspected.
theseus_architecture_gate	Pre-training gate checks, failed-gate residual rules, rerun triggers, and necessary-not-sufficient caveat.	The reported snapshot was not regenerated and is not current readiness, deployment, safety, or heavy-training evidence.
theseus_self_evolution_system	Intervention ladder, ATTD health gate, guarded teacher flow, architecture experiment governance, outcome ledgers, and loop-closure design.	No self-evolution scripts, ATTD reports, teacher runner, benchmark adapter flow, dashboard endpoints, or outcome ledgers were executed or inspected.
theseus_circle_transfer	Report-only bridge, structural smoke/proxy boundaries, claim axes, allowed structural claim, and disallowed quality/runtime/context/speed claims.	No Circle-to-Theseus consumer, structural smoke workload, proxy benchmark, private benchmark, model-quality evaluation, runtime measurement, or memory measurement.
Six-project local lineage	CCA, MoECOT Manifest, BeastBrain, BugBrain, Corben’s Trainer, and Corben’s Best Model Possible expose proxy/closure divergence, internal/external split failures, simulated metrics, stale promotions, constant or vacuous evaluators, retry pressure, architecture-inexpressible targets, and checkpoint/output disconnects.	One evolving local lineage, not six independent experiments; project-reported metrics and bounded host checks are not reproduced benchmark evidence for the book.

71.7 Mechanism

A benchmark ratchet turns a result into an evidence-state transition, not a celebration. The record captures the run, baseline, frontier state, residuals, regressions, holdouts, contamination checks, and promotion decision before any capability claim is allowed to move.

The ratchet needs evidence states that are harder to confuse than a single pass/fail flag: candidate_metric, schema_fixture_pass, synthetic_probe, source_reported, locally_reproduced, regression_floor, frontier_probe, contaminated, blocked, and retired. These states are not all encoded as separate fields in the current minimal schema, but future agents need them as drafting and implementation vocabulary so they cannot spend a fixture pass as benchmark evidence or a source-reported command as local reproduction.

The ratchet has two memories. The first is a regression memory: once a behavior matters, later systems must keep passing it or explicitly explain why the requirement changed. The second is a frontier memory: once a benchmark stops applying pressure, it should be retired into a floor and replaced by harder, less predictable work. The stack therefore does not ask whether a benchmark is “solved” in the abstract. It asks what the benchmark is allowed to prove now, what it must continue to guard, and what new pressure is needed next.

This creates an asymmetry that is useful for governance. Failure can immediately create a residual, a diagnosis, a regression, or a blocked promotion. Success only opens a review path. To become evidence, success must survive baseline comparison, negative-result accounting, hidden or transfer checks where appropriate, contamination review, and a claim-specific promotion decision. That asymmetry is deliberate: it is cheaper to record uncertainty than to repair an overpromoted capability.

flowchart LR
  A["Candidate metric or frontier"] --> B["Evidence-state classification"]
  B --> C["Run record, baseline, environment"]
  C --> D["Mastery, saturation, wall diagnosis"]
  D --> E["Residual escrow + negative results"]
  E --> F["Regression floor and harder frontier split"]
  F --> G["Holdout, transfer, mutation, contamination"]
  G --> H{"Promote readiness or claims?"}
  H -- "yes" --> I["Scoped support-state transition"]
  H -- "no" --> J["Blocked promotion / residual queue"]
  I --> K["Claim/evidence matrix"]
  J --> K

What the benchmark ratchet shows: The ratchet converts benchmark work into an evidence-state decision, not a celebration of a score. Residuals, regressions, holdouts, transfer checks, mutation, contamination, and negative results all gate whether readiness or a claim can move.

A benchmark ratchet record tracks benchmark state, baseline, command, environment, mastery threshold, saturation, residuals, regressions, holdouts, mutation/transfer checks, contamination audit, claim support effect, and negative results.

In practice, this means benchmark work should leave behind enough material for a future reader or agent to reconstruct the decision. If a model clears a threshold but fails a neighboring task family, the residual belongs beside the result. If a benchmark is useful only as a smoke test, the record should say so. If a source reports a command that this repository has not rerun, the record should preserve that boundary instead of laundering it into local evidence. Benchmark evidence discipline depends on this mundane bookkeeping.

The ratchet should also name the capability field it pressures. Without an SCF boundary, benchmark success can attach to a vague product, model, or agent identity and then survive migrations it was never meant to cover. With an SCF boundary, the benchmark can preserve a regression floor for the specific capability that earned it.

The benchmark-ratchet layer owns the evaluation instrument, not the underlying construct, model, claim transition, or release decision. Task owners define what behavior matters; Data and Supply Chain own the dataset; Runtime owns the executed artifact; Evaluator Governance owns the scorer; Claims/Evidence owns support; and Readiness/Release owns deployment authority. The Benchmark Instrument Lease is the bounded record that lets those owners consume a result without turning the benchmark into a truth machine.

71.7.1 A reusable-knowledge benchmark needs causal controls

A benchmark for learned libraries must distinguish access from effect. The minimum comparison exposes future tasks to a frozen candidate library and to a matched absence condition under the same search, verifier, and downstream budget. Stronger controls include an irrelevant abstraction of similar size, a duplicate, a no-op, a random valid abstraction, an expert library, and an oracle library. A removal condition asks whether a used abstraction was operationally necessary rather than merely present in the trace.

When library construction and search improve together, a two-by-two factorial is required: baseline library with baseline search, candidate library with baseline search, baseline library with candidate search, and both candidates together. Otherwise a better enumerator can be misreported as useful knowledge, or a useful library can be blamed for a search regression. Bounded set-removal tests can expose interactions without pretending that all subsets are tractable.

The evaluation round freezes task lineages, grammar, knowledge version, search procedure, verifier, budgets, and analysis before held-out exposure. Every attempt remains in the denominator, and verified, refuted, unknown, timeout, and invalid remain distinct terminal observations. These controls come from the proposed protocol in From Compression to Forward Transfer; they have not yet been executed here and do not move benchmark or chapter support.

71.7.2 A regret instrument must survive adversarial measurement

The Regret Engine source (regret_engine) contributes an experiment family for whether an adaptive system can learn from avoidable deficits without learning to hide them. The minimal instrument separates lucky reckless choices from sound unlucky ones, charges inaction for opportunity closure, preserves delayed harm, and reports intervals when the sign of an event-level counterfactual is unresolved. Its protected dimensions include hard violations, process and epistemic error, recovery burden, recurrence, counterfactual surplus, and learning eligibility; aggregate reward or one regret scalar cannot substitute for that vector.

The decisive ablations remove the Decision Capsule, let the actor choose comparators or scalar weights, grant ledger write access, suppress adverse evidence, collapse independent evaluators, omit competence-preservation replay, or route every severe outcome into learning. Outcomes must include false blame, false absolution, interval coverage, paralysis, repeated-defect rate, protected-capability regression, tampering, recovery time, human burden, and full compute cost. The paper proposes these tests but reports none of them; its citation keys also require independent bibliography recovery before they can support external-comparator claims.

71.7.3 Residual-specific intervention before architecture change

Ratcheting Modular Intelligence adds an ordering rule that is more useful than the slogan “try a new architecture.” When a frontier stalls, the team should first determine which intervention family the residual actually implicates:

1. Instrument audit: inspect labels, tests, solvability, construct validity, contamination, evaluator capacity, metric incentives, and transfer.
2. Data: change coverage, label quality, examples, curriculum, or data allocation while preserving the frozen evaluation boundary.
3. Training: test loss, optimizer, schedule, post-training, preference or reinforcement learning, distillation, and compute-allocation hypotheses.
4. Inference: test retrieval, memory, search, planning, deliberation, verification, tool use, and decomposition.
5. Proceduralization: compile a recurring successful trajectory only after its parameters, preconditions, effects, verification, monitoring, and retirement conditions are known.
6. Bridge frontier: insert an intermediate task only when it has a declared causal relationship and transfer test to the blocked target.
7. Architecture: change the representational or computational substrate only with a residual-specific mechanism hypothesis and preservation plan.

The sequence is a default diagnostic order, not an absolute prohibition. A target that the current representation cannot express, a critical safety failure, or an active incident may justify an immediate stop or redesign. The record must state why lower-impact levels were inapplicable, unsafe, exhausted, or non-identifying. Otherwise architecture novelty can erase null results and manufacture necessity.

The Capability Ratchet paper usefully separates three transitions that this book otherwise discusses in different chapters. A benchmark ratchet moves an instrument from frontier pressure toward diagnosis, regression, live, or retired duty. A procedural ratchet turns a recurrent successful trajectory into a versioned tool with preconditions, verification, monitoring, and retirement. A structural ratchet changes the underlying architecture only after the residual and lower-impact intervention record justify it. One event may touch all three, but the evidence ledger must record them separately; a new tool is not a new architecture, and a higher score is not automatically a durable procedure. This synthesis is same-author lineage, not independent confirmation of its component mechanisms.

An intervention record freezes the residual cluster, current system, instrument, evaluator, data boundary, budget, baselines, predicted causal signature, affected floors, and stopping rule. Each tested level records accepted, failed, null, inconclusive, skipped, and superseded attempts in one selection lineage. Architecture change becomes a testable statement—mechanism X should improve residual class B while preserving floor G—rather than a post hoc story attached to a higher score.

Time-decayed mastery thresholds require the same discipline. Waiting cannot be an intervention. Any planned decay must bind a stable construct and dataset, uncertainty, subgroup floors, patience window, slope, absolute floor, critical-failure vetoes, residual severity and recurrence, and a reason that advancing has greater expected value than continued work. The paper’s 90% and 70% examples are illustrative parameters, not defaults. A decayed threshold can graduate an ordinary instrument; it cannot delete its tail, authorize a critical miss, broaden a claim, or make the successor ready.

Public calibration is also a separate instrument role. It asks how a frozen system compares under a shared external setting. Private holdouts, live tasks, diagnostics, and internal frontiers ask different questions about generalization and next work. Public exposure therefore receives its own budget and contamination record; a public calibration score cannot directly serve as the complete internal optimizer without changing its evidence value.

71.7.4 Saturation and a wall are different diagnoses

Benchmaxxing distinguishes two events that are easy to collapse. A benchmark is saturated when it no longer provides useful development information. A system hits a wall when it stops improving on an instrument that is still valid, unsaturated, and capable of distinguishing the target behavior. A high score can coexist with an unsaturated tail; a low score can coexist with a saturated or broken instrument. Neither event can be inferred from score alone.

Saturation is a joint diagnostic vector. Candidate signals include ceiling performance, clustering among competent systems, overlapping uncertainty, noise or ambiguity dominating the remaining errors, high exposure or contamination risk, weak transfer, and low residual value. The last signal is especially important: an instrument may remain difficult while its failures no longer identify a useful intervention. Conversely, a benchmark with a high average remains frontier pressure when a consequential subgroup or task family is still discriminative. Suite-wide saturation therefore cannot be computed as a naive product of per-task threshold flags; task weights, vetoes, uncertainty, critical tails, construct drift, and consumer scope remain explicit.

A wall diagnosis freezes the development object as M_t = (A_t, theta_t, D_t, I_t): architecture or model class, trained parameters, training/post-training data, and inference procedure including tools, memory, retrieval, and scaffolding. Interventions must say which component changed, what residual-specific movement was predicted, what budget and stopping rule counted as a fair attempt, and which components remained fixed. If the benchmark changes materially, the record is an instrument repair or frontier transition, not evidence that the old model wall moved.

Architecture necessity is counterfactual and prospective. The claim is not “the new architecture scored higher” but “mechanism X should improve residual class B after adequate instrument, data, training, inference, and procedural hypotheses failed, while preserving regression floor G.” A result that changes architecture, data, compute, inference, evaluator, and task distribution together cannot identify an architecture wall.

Assurance-Shift Learning (assurance_shift_learning) adds a second diagnostic: the selection gap between the best supported candidate and the candidate the current selector chooses. A small gap under adequate coverage can be consistent with local selection saturation; a large gap points first toward ranking, routing, or evaluator failure. Neither diagnosis is valid when coverage, observability, contamination, or scaffolding attribution is unknown.

Its proposed SaturationShiftBench freezes twelve learning conditions under equal total cost and compares ordinary positive acquisition, negative-only pressure, hard-negative variants, evaluator-first repair, boundary bundles, placement-aware repair, composition-aware repair, and governed full-system variants. The endpoint jointly tracks residual risk, useful coverage, false inhibition, observability, recurrence, probe yield, evaluator adequacy, repair collateral, train-deployment divergence, compatibility, recovery, and full assurance cost. The design includes decisive falsifiers; no condition has run, so it is a benchmark specification rather than a result.

71.7.5 A benchmark portfolio is a multi-rate control system

One benchmark cannot simultaneously be cheap enough for every training loop, deep enough for diagnosis, difficult enough for architectural decisions, fresh enough to resist contamination, and broad enough to guard safety. The portfolio therefore operates at multiple rates:

Instrument class	Primary use	Admission boundary
Fast inner loop	Frequent training feedback and cheap regression detection	Narrow construct; known proxy gap; cannot establish broad capability
Diagnostic	Isolate a failure mechanism or residual cluster	Evaluator must distinguish competing diagnoses
Frontier	Guide major development decisions on difficult unsolved work	Frozen target, adequate capacity, strong baselines, full costs
Private or temporal holdout	Test generalization outside the development surface	Protected custody, exposure accounting, one-way result flow
Live external	Test freshness and real-work transfer	Versioned population, sampling frame, rights, and operational context
Regression floor	Preserve a previously accepted obligation	Exact inherited scope and change-triggered requalification
Safety or misuse	Measure unacceptable behavior and control adequacy	Separate vetoes and authority; capability gains cannot compensate for failure
Retired or historical	Preserve lineage without steering current optimization	Explicit retirement reason and prohibited downstream uses

Promotion between roles is governed. A frontier can become a regression floor only for the exact behavior it validly measured. A public benchmark can remain a calibration surface without becoming a private development target. A contaminated instrument may retain historical value but lose promotion authority. Retirement is not deletion: label errors, ambiguity, cost, contamination, narrowness, metric failure, or zero residual value remain in the ledger so descendants cannot silently resurrect the same instrument.

Each score also receives a capability narrative: the task population, environment, allowed tools, horizon, reliability, exclusions, failure tail, and human or operational meaning of the result. “82%” is not a capability. “Resolves short, well-specified repository defects under a frozen tool and time contract, but fails ambiguous requests and large refactors” is a bounded claim that another layer can evaluate.

71.7.6 Eighteen-stage benchmark-instrument lifecycle

The lifecycle registers the construct, movable claim, task population, data, metric, harness, model, checkpoint, output path, evaluator, baselines, budget, authority, environment, and expiry. It validates that the architecture and observation channel can express the target; versions data, splits, provenance, rights, public exposure, contamination, and hidden custody; preregisters natural and adversarial cases, thresholds, uncertainty, tails, retries, falsifiers, narrowing, and retirement; and freezes strong matched baselines and negative controls.

Every metric then receives an origin class and a causal binding to raw observations, returned outputs, evaluator outputs, and the exact claim. Equivalent retries, prompt and hyperparameter search, checkpoint selection, archive ancestry, repairs, discarded candidates, and validation feedback share one selection lineage. Instruments move through candidate, calibration, frontier, regression-floor, contamination-review, quarantine, blocked, retired, and historical states. Natural runs preserve raw inputs and outputs, traces, seeds, resources, evaluator records, failures, divergence, timeouts, OOMs, exclusions, fallbacks, repairs, and costs.

Decontamination, canary strings, public-calibration budgets, hidden and temporal holdouts, mutation, paraphrase, distribution shift, tool injection, memorization, privacy, and answer-exposure checks precede interpretation. Wall and saturation diagnosis separates learning from leakage, proxy drift, shortcut behavior, evaluator ceiling, target inexpressibility, and presentation gaming. Utility, calibration, tails, safety, rights, retained capabilities, robustness, latency, resources, human burden, costs, and residuals are measured together. Causal ablations and metric meta-evaluation test whether the named mechanism and metric behave as claimed.

Success opens a least-authority review; it never promotes automatically. Every prior floor, exclusion, contamination finding, negative or inconclusive result, critical failure, and residual survives into descendants. Monitoring watches exposure, drift, evaluator capture, saturation, proxy gaming, floor regressions, retry pressure, displaced cost, and consumer misuse. Independent harnesses, evaluators, data custody, implementations, and organizations must reproduce a decisive result before heterogeneous transfer. Material change or failure expires, quarantines, narrows, downgrades, refutes, supersedes, or retires the instrument while preserving evidence and lineage.

71.7.7 Metric provenance and closure inheritance

Benchmark Ratchets owns metric admissibility because Evidence States decides whether support may move, while this layer decides whether a result is a valid measurement input at all. The Metric Provenance Record sits beneath the ratchet and names the metric’s origin as measured, derived, recomputed, modeled, fixture, constant, declared, proxy, vacuous, source_reported, or unknown. The class is not decorative. A source-reported number stays source-reported; a constant bonus stays constant; a proxy cannot be narrated as the target; and a fixture can preserve a regression without becoming empirical capability evidence.

The record also installs four inherited gates:

1. Benchmark closure inheritance. A descendant frontier inherits prior regression floors, exclusions, contamination findings, and residuals. Renaming a suite, checkpoint, model family, or report cannot shed them without an explicit override record.
2. Representation-capacity gate. A metric is inadmissible when the evaluated architecture cannot express the target class. A repeated-token adaptor cannot be judged as a general sequence generator; a status-polling workload cannot establish cognitive throughput; a tone codec cannot establish ordinary speech recognition.
3. Retry-lineage ceiling. Materially equivalent retries share one lineage and a declared ceiling. Once the ceiling is crossed, another attempt requires a new intervention, transfer-distance change, or authorized override; favorable-number search is not evidence accumulation.
4. Checkpoint-to-output binding. A capability metric binds checkpoint, tokenizer, runtime path, raw output, evaluator output, and exact claim. If the runtime bypasses the trained component, an adaptor score cannot support the returned answer.

The historical projects make all four gates concrete. CCA records falling loss without honest benchmark lift and canonical closure that can disagree with partial score surfaces. Corben’s Trainer records quarantined fixture runs that had already influenced claim and promotion records. Corben’s Best Model Possible contains constants, formulas, proxies, vacuous passes, explicit lane binding, and a runtime answer path not generally driven by the trained adaptor. BeastBrain and BugBrain contain simulated I/O, fixed random paths, and status workloads that resemble stronger evaluations. MoECOT Manifest records the useful opposite discipline—separate internal contracts and external holdouts—while also showing that strong internal scores can coexist with weak external performance.

flowchart LR
  M["Metric value"] --> O{"Origin class"}
  O -->|"measured / recomputed"| C["Observation + evaluator"]
  O -->|"source / fixture / constant / proxy / vacuous"| B["Block or regression-only"]
  C --> R{"Target representable?"}
  R -->|"no / unknown"| B
  R -->|"yes"| L["Inherit floors, exclusions, residuals"]
  L --> T{"Retry ceiling intact?"}
  T -->|"no"| B
  T -->|"yes"| K{"Checkpoint-to-output binding complete?"}
  K -->|"no"| B
  K -->|"yes"| A["Admissible ratchet input"]
  A --> P["Claim-specific promotion review"]

How to read the provenance gate: the number enters from the left, but its origin and causal route determine whether it can reach a promotion review. A blocked metric may still be valuable as a regression, negative case, or residual. The figure does not report a benchmark result.

The strongest objection is that metric provenance can become paperwork around bad constructs. That objection is correct when fields are filled without changing a decision. The gate therefore rejects records that cannot name a target construct, representation-capacity basis, observation, evaluator, inherited floor, retry lineage, and causal output binding. An exact blocker is preferable to a complete-looking but meaningless packet.

71.7.8 ReflexBench as an argument-exit program

The Reflexive Router contributes a benchmark design rather than a benchmark result. ReflexBench must cover eight task tracks: atomic exact operations, paraphrase, context-dependent resolution, composite DAGs, temporal reasoning, effectful work, adversarial routing, and trace-to-reflex compilation. Each track should exercise unmarked automatic routing, forced-route commands, direct typed commands, and compiled workflows where applicable.

The matched comparator matrix includes LLM-first execution, hard rules, a learned router, semantic cache, tool agent, modular specialist routing, the full reflexive stack, and an oracle analysis arm. Comparisons freeze models, candidate bytes, context, tools, authority, retry policy, verifier access, budgets, and total-cost accounting. Route labels are evaluated separately from answer or artifact usefulness and separately again from observed effects. Independent route, outcome, and effect evaluators must retain disagreement and may not share one implementation as hidden ground truth.

The negative-control surface includes command-looking untrusted text, literal escaping, ambiguity, stale context, OOD inputs, contract rejection, silent fallback, cache collision, Chronicle poisoning, authority widening, partial effects, duplicate retries, verifier monoculture, premature compilation, distribution drift, quarantine, rollback, and failed decompilation. Complete denominators include proposals, disqualifications, abstentions, retries, fallbacks, human interventions, partial effects, verifier failures, and discarded compiler candidates.

Primary measures remain plural: useful task outcome, qualified and fast-path coverage, wrong-fast-path rate, selective risk, route regret, override fidelity, silent fallback, argument binding, DAG validity, context continuity, temporal accuracy, initial-correct corruption, unauthorized effects, verification escape, postcondition satisfaction, rollback completeness, compilation transfer, latency, compute, energy, money, human and monitoring work, governance cost, residuals, and displaced work. A scalar summary cannot override any safety, usefulness, effect, or denominator failure.

No ReflexBench run accompanies the paper. This section freezes what a proper attempt must test; it promotes no routing, safety, efficiency, or compilation claim.

71.7.9 P4/M6 instrument and metric lesson

M6 demonstrates why instrument failures and joint denominators belong in the benchmark record. A repeated-role JSON array failed at 1/4 sacrificial admissibility; a single prospectively frozen fixed-key repair passed 4/4 before the 32-task denominator opened. The terminal report keeps route correctness, useful outcomes, wrong-fast-path count, unsafe outputs, selective coverage, effect observations, and deliberation corruption/repair together. Reporting 31/32 route correctness or 21/32 useful outcomes alone would erase the two unsafe outputs; reporting zero unsafe LLM-first outputs alone would erase its 18 wrong fast paths and nine useful outcomes. The validator rejects both forms of laundering and preserves all seventeen active control classes.

M7 adds a second benchmark lesson: integrity is not target validity. Its v1 validator correctly reproduced frozen bytes, arm counts, and state digests, yet every arm—including deletion-aware retraining—scored zero on the deletion target because the cohort was single-class and marker-shifted. That result is preserved as an instrument failure. A balanced v2 repair failed its deletion- like preflight before held-out execution. Only v3 opened the denominator after a three-way ablation showed Transformer-only at 0.7667/0.7000 general/deletion- like accuracy and fused structured state at 1.0000/1.0000. The benchmark record therefore binds target adequacy, representation choice, failure lineage, and claim disposition together; a green integrity check cannot promote an uninformative outcome.

Campaign 4 adds a third lesson: favorable sacrificial semantics cannot waive a failed consumer contract. Its terminal v3 preflight reached 6/6 release- eligibility accuracy, 3/3 clean releases, perfect required-residual recall, and zero false reassurance. It still admitted only 3/6 extraction objects because all defect cases used undeclared route retain and omitted requested-check IDs. The frozen gate required 5/6, so the heldout workload remained sealed. Reporting only the favorable semantic values would manufacture a residual- honesty result from an instrument that downstream verification and capacity components could not consume.

71.7.10 Campaign 5: a promotion gate that preserves heterogeneous effects

Campaign 5 is a concrete benchmark-ratchet example because the decision was frozen before 11,250 episodes were generated. Two authored environments, five seeds, ten arms, 6,000 held-out episodes, matched authority ceilings, six named ablations, a result schema, code and environment hashes, an independent scorer, and a no-repair-after-outcome rule existed before the result. A validator then recomputed the central metrics and rejected ten laundering mutations.

Every integrity and baseline gate passed, and six of six preregistered aggregate ablation signatures moved in the declared direction. The resulting review accepted one synthetic-test-backed non-core transition. The ratchet still preserves inconvenient details: governed task success fell slightly relative to the ungoverned predictive baseline; active information was mildly harmful in one environment; shift detection and intervention-effect prediction remained weak; the environments were finite authored tables; and no neural model or external evaluator was used.

This is the intended asymmetry. Success opened a scoped transition only after the denominator, baselines, artifacts, causal comparisons, uncertainty, costs, lineage, rollback, and non-claims survived review. It did not promote a chapter core, validate the simulators as reality, turn a shared three-state abstraction into general transfer, or authorize release. The complete receipt is docs/p4_situated_world_model_campaign.md.

71.7.11 Capability-preservation and semantic-continuity campaigns

The two new Corben papers add complementary argument-exit programs. The capability-compilation campaign must test transactional software, controlled dynamics, contact-rich control, and verifiable reasoning/tool use against scratch learning, imitation, distillation, merge, MoE, sparse-link, dense, and full-policy baselines. It must ablate traces, learner-induced-state collection, counterexample memory, shields, reification, and evaluator independence while charging router, verifier, fallback, residual, and maintenance cost.

The semantic-continuity campaign must be longitudinal. It introduces controlled changes to labels, definitions, prototypes, roles, relations, units, grounding models, policies, contexts, and world branches, then measures historical replay, silent equivocation, migration accuracy, defeater recall, context leakage, branch contamination, patch rejection, blast-radius recall, packet sufficiency, planning value, information-flow leakage, and total governance burden. Strong retrieval, graph, ontology/provenance, context-aware, latent-world-model, and concept-model baselines are mandatory.

Both campaigns have explicit disconfirmation: retain only mechanisms that beat well-tuned simpler systems on a measured frontier after total cost and failure denominators. A neat schema, self-consistent receipt, or internally generated test suite cannot promote a claim by itself.

71.7.12 RODIE: relational structure as a vector instrument

The Relational Dimension Compiler contributes a future instrument family, RODIE—Relational Order and Dimensional Intelligence Evaluation. It should not become one leaderboard number. Its job is to isolate which structural operation occurred and whether a simpler route could recover it.

Suite	Construct that must remain separate
Minimum order	unary, pairwise, triadic, quartic, composition, and mixed-order tasks under matched resources
Typed role binding	exact participant identity, co-membership, relation schema, role assignment, symmetry, and multiplicity
Latent topology	entity count/identity, relation proposal, changing hyperedges, event boundaries, and alternative structures
Counterfactual fidelity	observational fit versus role-specific deletion, substitution, and intervention effects
Dynamic identity	persistence, split, merge, retirement, invalidation, and branch-local state
Reversible contraction	coarse-query fidelity, compression, expansion triggers, lineage, and failure under changed queries
Geometry and fields	equivariance, discretization transfer, object–field coupling, and conservation or boundary constraints
Branch discipline	simulated-versus-actual separation, false actualization, adoption events, and horizon calibration
Compilation and reuse	slow-path equivalence, monitoring, shift detection, rollback, and net cost saving

Every suite needs a complete proposal denominator. Candidate recall is reported beside relation precision and recall; otherwise a sparse model can appear precise by never proposing hard structures. Role permutations, lower-order rescues, latent-mediator alternatives, oracle structure/order upper bounds, exact symbolic controls where applicable, and strong fixed-order systems are mandatory. Parameters, data, accessible information, context, compute, memory, tuning, retry, evaluator, and repair opportunity are matched or explicitly priced.

The primary result is a vector: useful task behavior, exact role accuracy, order regret, intervention consistency, branch leakage, contraction fidelity, expansion precision/recall, calibration, topology transfer, active and total resources, rejected candidates, verifier/human burden, repair, rollback, and hidden residuals. A nominally higher-order model has not won if proposal, qualification, sparse data movement, or monitoring consumes its local gain.

RODIE is presently a proposed benchmark program. No generator, held-out split, evaluator, preregistration, model run, or result exists in this repository. Before implementation, the suite needs an anti-self-confirmation review: its generators and metrics cannot simply reward the relation ontology and minimum order assumed by the RDC design.

71.7.12.1 What a credible RODIE campaign would have to show

Synthetic structure is useful for isolating order and role binding, but it can also build the answer into the generator. A credible campaign therefore needs two linked tracks. The first uses controlled tasks with independently known entities, roles, branches, interventions, and contraction structure. The second uses natural domains in which useful relational structure is not chosen by the benchmark author—for example event extraction with role changes, scientific systems with independently measured interventions, program or workflow traces with exact identities, or physical simulations whose fields and discrete objects have separate ground truth. Success on the controlled track qualifies the instrument; success on the natural track tests transfer.

The final split stays sealed until candidate proposal, schema choice, role labels, evaluator sensitivity, and lower-order rescues pass development gates. Evaluation separates at least four questions: whether the correct participants were ever proposed; whether the correct schema and roles were assigned; whether the resulting structure improved prediction, intervention, or useful action; and whether the total route beat a competent simpler alternative. A system can pass one and fail the next. High relation precision with poor proposal recall is selective omission. Correct co-membership with swapped roles is semantic failure. Better structural reconstruction without useful or causal gain is a representation result, not a capability result. Faster kernel time with worse end-to-end qualification cost is a hardware-local result, not an efficiency result.

Negative results need the same discipline. Failure of one naive triadic layer cannot refute higher-order computation. Failure after matched architecture engineering, method-specific tuning, adequate candidate recall, evaluator sensitivity, lower-order and fixed-order rescue, multiple seeds, complete failure denominators, and transfer can narrow the exact mechanism and regime. Conversely, a positive result does not establish one true ontology: alternative latent mediators, equivalent schemas, and simpler computation plans remain live explanations until interventions distinguish them.

71.7.13 KERC as a full-system representation campaign

Kernel English cannot exit argument by showing that hand-written controlled sentences use fewer custom tokens. Its campaign begins with a frozen Kernel grammar, stable concept registry, codebook, tokenizer, packet and residual schemas, exact-object contract, compiler/renderer/verifier versions, migration policy, and public-safe corpus. The corpus must include paraphrases and minimal semantic contrasts; dialect and code switching; rare and confusable names; numbers, units, modality, attribution, quotations, code, and exact-form tasks; long interactions; registry updates; and adversarial residual, object, macro, authority, and state transitions.

The serious baseline set includes strong surface BPE and unigram models, byte-level and dynamic-chunking systems, learned discrete compression, controlled-language or semantic-IR systems, copy-aware entity-handle systems, compact or latent reasoning languages, semantic cache/shared glossary alternatives, and deliberately simpler combinations that omit the full Kernel language. A KERC core is compared at equal underlying raw bytes, equal total training FLOPs, and equal end-to-end inference budget. Method-specific tuning and rescue happen before a negative mechanism claim.

Evaluation has separate representation, core-modeling, task, fidelity, interaction-amortization, robustness/fairness, security, and migration tracks. Compiler preservation is judged independently from renderer quality through bidirectional entailment, question preservation, targeted entity/value/scope/ modality/attribution probes, exact-object checks, and adjudicated counterexamples. Semantic, faithful, lexical, and lossless modes report Kernel, global, segment, local, exact-object, registry, and codec rates separately.

The denominator retains every source item, compilation candidate, correction, abstention, fallback, dropped or strengthened claim, initially correct form corrupted by the compiler, render mismatch, verifier miss, state update, checkpoint, desynchronization, migration, retry, human judgment, and discarded macro. Measures include task quality, rare-name and exact-value fidelity, calibration, bytes, tokens, FLOPs, KV cache, peak memory, latency distributions, energy where instrumented, parameters, residual burden, privacy and governance cost, and benefit as a function of interaction length.

Causal ablations remove protection, uncertainty lattices, sense-aware canonicalization, concept capsules, each residual level, Kernel BPE, dual vocabularies, round-trip verification, state hashing, and macro promotion. Explicit falsifiers include no realistic end-to-end compute or memory regime, worse task accuracy at matched bytes/FLOPs, unacceptable semantic corruption, residual rates erasing savings, no shared-residue benefit, worse exact-object fidelity, or greater engineering/governance cost without a better frontier. A narrow result—such as entity handles and terminology locks helping while the full internal language fails—is a valid scientific outcome, not a reason to hide the denominator.

71.8 Interfaces

The Benchmark Ratchet Record is the interface between evaluation, readiness gates, Appendix C, prototype phases, and the changelog. It carries enough evidence metadata for downstream layers to distinguish a reproduced result, a source-reported result, a synthetic fixture, a failed run, and an unattempted target.

Minimum fields:

• ratchet_id
• capability_ref
• benchmark_ref
• benchmark_state
• run_refs
• baseline_refs
• mastery_threshold
• saturation_state
• residual_escrow
• regression_refs
• anti_goodhart_checks
• promotion_decision
• support_state_effect
• negative_results

Those compact fields join two separate ledgers rather than hiding benchmark and system change in one row. The Benchmark Ledger additionally records the claimed construct, task population and sampling frame, instrument role, version, evaluator, saturation vector, contamination/public-exposure state, label or test reliability, transfer evidence, cost, last refresh, retirement criteria, and capability narrative. The Model Ledger records (architecture, parameters/checkpoint, data, training process, inference process), complete portfolio results, residual map, inherited floors, cost and latency profile, safety profile, selection lineage, and current limiting hypothesis. Their join identifies which model-system actually faced which instrument. Neither ledger substitutes for raw runs or evaluator records.

The operational packet around those fields should also name the evidence class, benchmark ownership, public-calibration boundary, contamination surface, hidden/transfer status, fixture-vs-run distinction, exact claim that could move, exact claim that cannot move, and the reviewer or automation that made the promotion decision. If any of those are unknown, the packet can still be useful, but it cannot move a support state.

Routing and SCFs consume readiness. Appendix C records support movement only when evidence permits it. The changelog records evidence changes. Prototype roadmap phases cannot use benchmark success as evidence without the run record.

Procedural memory consumes benchmark ratchets as retirement pressure. If a generated tool fails a preserved floor, it should be revised, quarantined, or retired. If a benchmark saturates, it may still guard the tool as a regression even when it no longer justifies a stronger claim.

Twelve owner interfaces keep that connection honest. Task/Construct owners define the behavior; Data/Supply Chain owns provenance, rights, contamination, retention, deletion, and hidden custody; Runtime/Tools/Artifacts owns the exact model-to-output trace; Evaluator Governance owns scorer validity and independence; Routing/SCFs/Training/Policy/Self-Improvement consume only an accepted lease; Claims/Evidence owns support movement; Readiness/Thresholds/ Safety Cases/Release owns safeguards and deployment; Security/Privacy/Rights owns attacks, leakage, affected-party constraints, and erasure claims; Resource Economics owns complete cost; Proof Envelope owns only finite ratchet-record claims; Unlearning/Continual Learning owns influence, privacy, storage, retained utility, and descendant-state semantics; and Living-Book/Steward/Change Governance owns synchronized surfaces, retirement, and correction.

71.9 Invariants

• No test success is claimed without execution.
• Synthetic and empirical evidence remain distinct.
• Regressions are preserved.
• Saturated benchmarks become regression floors, not permanent definitions of intelligence.
• Negative and inconclusive results remain visible.
• Source-reported, synthetic, fixture, reproduced, and empirical results never collapse into one evidence class.
• Public-calibration boundaries are preserved even when a benchmark is useful for orientation.
• Every metric exposes its origin class; measured, derived, modeled, fixed, proxy, vacuous, and unknown values remain distinct.
• Descendant benchmarks inherit prior floors, exclusions, contamination findings, and residuals unless an explicit override is recorded.
• A metric cannot be admitted when the architecture cannot express the target or the evaluated checkpoint cannot be bound to the output.
• Equivalent retries cannot escape a lineage ceiling through renamed runs.

Claim specificity is the benchmark ratchet’s strongest invariant. A benchmark can support only the claim it actually measured, under the recorded conditions. Broader readiness, deployment safety, model quality, or ASI-progress claims require their own evidence.

Ratchets must preserve old baselines and failed attempts so benchmark progress cannot erase regression history.

Evaluator updates also need versioned admission rules before old scores and new scores are compared.

The full invariant set also requires exact instrument scope; construct and target-capacity validity; traceable metric origin and checkpoint-to-output binding; versioned data exposure, rights, contamination, and hidden custody; matched baseline opportunity; complete selection and failure denominators; saturation-to-floor conversion; descendant inheritance; visible subgroup, tail, adversarial, privacy, retained-capability, and cost effects; separately budgeted holdout and contamination checks; no score-to-causality or score-to- deployment inference; review before authority; complete cost; change-triggered expiry; independent error separation; and owners for every metric dispute, proxy gap, failed transfer, retirement, and residual.

71.10 Failure modes

• Benchmark overfitting.
• Hidden regression deletion.
• Claim support inflation.
• Source-reported results presented as reproduced results.
• Public-calibration leakage.
• Evidence-class collapse, where a schema fixture, book-build check, synthetic probe, or source-reported command is narrated like empirical capability evidence.
• Floor/frontier inversion, where a saturated regression floor is used to justify broad readiness while the new frontier remains untested.
• Metric-origin laundering, where a constant, formula, fixture, proxy, source-reported, or vacuous value is presented as measured.
• Architecture-inexpressible evaluation, where the scoring target is impossible for the evaluated output path.
• Retry laundering, where materially identical failed runs are renamed until a favorable number appears.
• Checkpoint/output disconnect, where a trained artifact is scored but the returned behavior bypasses it.

Each failure blocks promotion or downgrades support state until the evidence boundary is repaired.

A subtler failure is benchmark nostalgia. A once-useful benchmark may remain in the book because it was historically important, even after it stops applying pressure. The ratchet should preserve it as a regression floor or historical note while making clear that it no longer defines the frontier.

The matched failure set includes undefined constructs; architecture- inexpressible targets; fixture or source-report laundering; obscured metric origin; checkpoint/output disconnect; public exposure and hidden-holdout reuse; weak or asymmetric baselines; retry and selection-lineage laundering; validation or archive overfitting; saturation overclaim; deletion of floors, failures, and residuals; averages hiding subgroups, tails, privacy, safety, retained-capability, or cost damage; evaluator capture and reward gaming; imported external results; unsupported transfer; hidden complete cost; automatic routing/training/readiness/release/support movement; and prestige or sunk cost blocking quarantine, narrowing, refutation, or retirement.

71.11 Minimum Viable Implementation

A benchmark ratchet starts as a benchmark-evidence decision record. The repository fixture captures the fields a later benchmark must supply before it can affect support state, while making clear that no benchmark result is being reported here.

The mined-project minimum adds schemas/metric_provenance_record.schema.json, one blocked six-project lineage record, and ten expected-invalid mutations. python3 scripts/validate_metric_provenance.py rejects source-reported, constant, proxy, and vacuous metrics marked admissible; measured metrics without observations; contaminated or architecture-inexpressible metrics; retry-ceiling violations; complete-looking checkpoint bindings with missing refs; and closure records that drop inherited regressions. The valid record remains blocks_promotion because its metrics were not reproduced and its checkpoint/output route is incomplete.

A toy ratchet ledger is the smallest concrete extension: one passing run record, one failed neighboring case, one saturated benchmark moved to regression, and one blocked promotion. Such a ledger would demonstrate support-state restraint while leaving empirical capability unclaimed.

The toy ledger should include one intentionally contaminated case and one source-reported-only case. Those cases are useful because they force the harness to preserve evidence boundaries: contamination creates a blocked or quarantine decision, and source-reported status remains source-reported until the exact command, environment, data boundary, and artifacts are imported or rerun.

71.11.1 Concrete Fixture Bridge Boundary

The current repository bridge is deliberately narrower than a toy benchmark ledger. python3 scripts/validate_benchmark_fixture_bridge.py recomputes the existing benchmark anti-Goodhart fixture set and checks experiments/benchmark_antigoodhart/results/2026-07-02-fixture-bridge.json. Lean separately proves quantified ratchet-decision consequences; it no longer copies the consumer’s counts and flags.

The bridge records 2 valid fixtures and 5 expected-invalid controls. The valid side contains one promotion-ready synthetic path and one saturated-benchmark regression-floor path. The invalid controls reject missing anti-Goodhart checks, policy promotion from blocked ratchet evidence, reward-as-truth confusion, saturated benchmark promotion, and steward release without approval. Those executable facts remain fixture evidence only; the retained Lean theorems instead derive readiness, saturation-floor, and contamination consequences for arbitrary finite ratchet reviews.

The bridge makes the anti-Goodhart record discipline operational at the record boundary: a synthetic success path and five nearby failure paths are checked together instead of narrated separately. It is not a benchmark run, hidden-holdout validation, contamination detector, policy-training result, steward-agent execution, release-safety review, Theseus replay, model-quality result, or support-state transition.

71.11.2 Diagnostic scenarios and provider qualification

ext_elizaos_agent_runtime_2026 supplies a current implementation comparator for this boundary. Its scenario runner can drive a real runtime through message, action, API, and scheduler turns with deterministic or live-model paths and per-turn assertions. Its documentation explicitly distinguishes this useful in-process diagnostic lane from provider-qualified evidence. The qualification design adds an external controller, a closed content-hashed manifest, authenticated ingress, independent observers, durable readback, isolated trajectory hashing, and signed verdicts.

That distinction is a valuable benchmark-ratchet pattern. A diagnostic can exercise real code while a separate qualification lane supplies the independent observation boundary needed for stronger evidence. The documented idempotency and readback mechanisms provide additional delivery evidence while the qualification record conservatively keeps exactly-once false. The book adopts this separation and keeps the effect denominator open when external reality is not observed. No elizaOS scenario, provider, observer signature, or qualification result was executed or reproduced here.

The exact current minimum is one benchmark-ratchet schema and valid record fixture; one metric-provenance schema with one valid blocked six-project- lineage record and ten rejecting mutations; one synthetic anti-Goodhart harness with two valid records and five expected-invalid controls; one persisted executable fixture result aligned to that same set; three proof targets; and a 29-declaration BenchmarkRatchets surface with an independent lifecycle consumer. The consumer recompiles the module, executes six clean transitions, checks all seven trace splits, explores 19 reachable states through 114 transitions, checks 12 quarantine suffixes, reaches saturated-floor and contamination-quarantine outcomes, and rejects 15 lifecycle plus 11 semantic mutations. The Lean surface proves arbitrary-run custody and stage coherence, exact accepted-trace receipt accounting and composition, persistent quarantine, and that aggregate pass counts cannot exactly classify modeled promotion admissibility. Executable fixture totals are not copied into Lean. There is no natural benchmark workload, hidden holdout, contamination audit, public-calibration budget, strong empirical baseline matrix, model or policy training run, independent evaluator, capability gain, safety result, readiness transition, deployment, reproduction, transfer, unlearning result, chapter-core transition, or SOTA result.

71.12 Mature Research Target

The benchmark endpoint is an operating system for evaluation, not a leaderboard appendix. Every evaluation would become a governed evidence packet with enough history to tell whether the system learned a capability, learned a test, leaked a public calibration surface, or merely improved a proxy.

A mature benchmark OS manages benchmarks as instruments with lifecycle state. A task can begin as a candidate metric, become a frontier probe, graduate into a regression floor, enter contamination review, be retired, or remain blocked. Success would never move a claim by itself; it would open a promotion review that compares baselines, residuals, negative results, holdouts, transfer checks, mutation checks, public-calibration limits, and the exact claim being considered.

A benchmark ratchet surface needs:

• A ratchet ledger that records evidence class, command, environment, data boundary, baseline, mastery threshold, saturation state, residual escrow, regression refs, negative results, and promotion decision.
• Automatic floor/frontier management so solved tasks become regression obligations and new frontier tasks are generated only after the prior evidence boundary survives review.
• Anti-Goodhart controls for contamination, hidden-transfer failure, benchmark-answer exposure, proxy drift, metric gaming, and over-optimization for speed or presentation.
• Direct links to SCF readiness, routing decisions, procedural-memory retirement, Appendix C support movement, and changelog entries.
• Separate treatment for book-build checks, schema fixtures, synthetic probes, source-reported results, locally reproduced runs, and empirical capability results.
• Failure closure that preserves regressions, quarantines contaminated results, blocks claim inflation, and turns benchmark saturation into a floor rather than a story about general intelligence.

The completed evaluation surface would not call a capability “better” because a single score improved. It would be better only within the recorded evidence boundary, with regressions preserved and residuals visible. The benchmark operating system remains a target architecture until ratchet records, contamination reviews, transfer checks, regression-preservation tests, and accepted evidence transitions show that benchmark movement changed the permitted claim boundary.

71.12.1 Argument-exit campaign

The full attempt preregisters natural task distributions and adversarial, temporal, hidden, paraphrase, mutation, transfer, contamination, privacy, and tool-use cases for every load-bearing construct. It validates target capacity; freezes versioned data, exposure, rights, harnesses, model/checkpoint/output binding, evaluator identities, strong ordinary baselines, random/no-skill and wrong-target controls, budgets, retries, selection lineage, thresholds, uncertainty, subgroup and tail analyses, critical failures, falsifiers, and promotion rules before decisive scoring.

Runs preserve all candidates and failures while measuring useful performance, calibration, robustness, retained capabilities, safety, privacy, rights, latency, throughput, memory, energy, human and verifier burden, complete cost, fallback, recovery, and residuals. Causal ablations identify the claimed mechanism; metric meta-evaluation checks response to benign and harmful changes; independent harnesses, evaluators, data custody, implementations, and organizations reproduce the result; and transfer covers tasks, models, scaffolds, languages, domains, hardware, attacks, rights regimes, and time.

The terminal result may qualify a narrow claim, preserve a null, expose a proxy, refute a mechanism, quarantine contamination, retire an instrument, or record blocked_after_full_attempt. Until the required lanes pass and an exact transition is accepted, the core remains argument; score increases, green fixtures, and growing benchmark inventories are non-terminal.

71.12.2 The Portia/Spider lineage is a benchmark-authenticity test

SpiderSynapse reports an extended plateau; PortiaSynapse reports a replacement that compiles, trains with finite losses, and passes a test suite. Keeping both records prevents successor bias, but neither record by itself establishes the causal repair. Portia’s own document gives three incompatible test counts—15, 22, and 24—and leaves full integration and comparative benchmarking incomplete. The contradiction is not clerical noise: it is a concrete reason to bind every claim to an exact commit, test manifest, command, environment, and result.

The benchmark must also target the right object. Mean bit accuracy over a 128-bit HLSH coordinate can look good while every full coordinate is wrong; coordinate recovery can improve while route selection or downstream task success does not. The evidence packet therefore reports bit accuracy, exact coordinate identity, prefix/depth error, graph distance, edge accuracy, route success, task success, calibration to a named release event, abstention, fallback, latency, memory, and batch-composition invariance separately.

Matched baselines include the previous Synapse, one-path linear and MLP heads, Portia without Focus, without mutable memory, without each refinement, and Spider with controlled branch count and depth. Random walks and known paths are partitioned by semantic object and graph lineage to prevent structural leakage. A passing deterministic fallback cannot mask a failed learned path, and a stable optimization trace cannot substitute for held-out improvement. Only factorial ablation and replication can support the proposed causal story; otherwise the honest outcome is an improved implementation candidate with an unresolved mechanism.

71.13 Codex test plan

Test	Purpose	Status
Benchmark ratchet record fixture validation	Validate that a ratchet record names benchmark state, run refs, baselines, saturation, residual escrow, anti-Goodhart checks, promotion decision, support effect, and negative results.	implemented; passing via python3 scripts/validate_protocol_examples.py
Metric-provenance and closure harness	Validate one blocked six-project lineage record and reject measured-without-observation, source-reported/constant/proxy/vacuous laundering, contamination, architecture-inexpressible targets, retry-ceiling escape, incomplete checkpoint/output binding, and erased inherited regressions.	implemented by python3 scripts/validate_metric_provenance.py; record semantics only; no benchmark reproduction or support-state promotion
Saturation detection test	Check that saturated benchmarks move to regression rather than broad promotion.	implemented by benchmark anti-Goodhart harness for synthetic saturated/regression-only records; no benchmark run
Hidden benchmark transfer test	Check that transfer and contamination fields exist before support promotion.	implemented by benchmark anti-Goodhart harness for synthetic holdout, contamination, and mutation-or-transfer gates; no hidden benchmark run
Benchmark anti-Goodhart fixture bridge	Check the synthetic harness result record, 2 valid fixtures, and 5 expected-invalid controls without importing those counts into Lean or creating benchmark or policy evidence.	implemented by python3 scripts/validate_benchmark_fixture_bridge.py; copied Lean summary mirror retired; no empirical benchmark, policy-training run, steward release, support-state transition, or chapter-core promotion
Contamination audit test	Check that suspected leakage, public-calibration spending, or benchmark-answer exposure blocks or quarantines promotion.	planned; not run
Floor/frontier split test	Check that a saturated benchmark can become a regression floor while a different frontier remains required for stronger claims.	planned; not run
Source-reported boundary test	Check that source-reported commands or benchmark summaries cannot be marked locally reproduced without imported artifacts and replay metadata.	planned; not run
Regression preservation test	Check that promoted capabilities keep prior regression refs.	implemented by benchmark anti-Goodhart harness for synthetic regression refs and negative-result retention; no empirical regression suite run

71.13.1 Formalization hooks

Tag	Module	Target	Status
lean:benchmarks.ratchet.operational_invariant	AsiStackProofs.BenchmarkRatchets	Every arbitrary finite ratchet run preserves instrument, dataset, harness, claim, authority, evidence-policy, support, and effect custody plus stage/outcome coherence; every accepted trace accounts for exactly one receipt per event and composes across batches; clean evidence reaches only a non-authorizing independent-review candidate.	implemented
lean:benchmarks.ratchet.failure_blocks_promotion	AsiStackProofs.BenchmarkRatchets	The finite ratchet lifecycle quarantines suspected contamination, preserves quarantine across arbitrary finite suffixes, routes saturated clean instruments to regression-floor status, rejects missing transfer or preserved-evidence records without state change, makes closure absorbing, and proves that an aggregate pass count cannot exactly classify modeled promotion admissibility.	implemented
lean:benchmarks.ratchet.fixture_bridge	AsiStackProofs.BenchmarkRatchets	An independent benchmark anti-Goodhart consumer computes two valid fixtures and five rejected controls, recompiles the exact 29-declaration Lean surface, executes clean, saturated, and contaminated witnesses, checks all seven clean-trace splits, explores 19 reachable states through 114 transitions, checks 12 quarantine suffixes, and rejects 15 lifecycle plus 11 semantic mutations; executable totals remain separate from quantified Lean semantics.	implemented

These Lean hooks now combine three retained decision consequences with a finite registered-to-closed instrument lifecycle. Arbitrary accepted runs preserve exact instrument, dataset, harness, claim, authority, evidence-policy, support, and effect custody together with stage/outcome coherence; accepted traces add exactly one receipt per event and compose across batches. Missing transfer or preserved evidence rejects without state change, suspected contamination remains quarantined across arbitrary suffixes, saturation routes to a regression floor, and a closed disposition is absorbing. A same-pass-count witness with opposite admissibility proves that no classifier using only the modeled aggregate pass count can recover the exact decision. The independent consumer recompiles the exact surface, executes clean, saturated, and contaminated witnesses, checks all seven clean-trace splits, explores 19 reachable states through 114 transitions, checks 12 quarantine suffixes, and rejects 15 lifecycle plus 11 semantic mutations. These formal targets do not prove benchmark validity, transfer quality, contamination detection, regression-suite quality, anti-Goodhart effectiveness, policy quality, steward release safety, or empirical capability gains.

The formal audit counts 29 declarations: three retained finite decision cases and 26 lifecycle, trace, witness, persistence, Boolean-correspondence, and information-loss results. The three former summary-mirror declarations remain retired. None proves construct validity, target capacity, metric causality, benchmark adequacy, contamination resistance, hidden-holdout integrity, model capability, policy quality, safety, unlearning, deployment, reproduction, or transfer. Useful future formal work must constrain real lifecycle and inheritance semantics, selection-lineage conservation, information exposure, revocation, and implementation refinement; empirical obligations stay empirical.

71.14 LocusBench and Placement Regret

The proposed LocusBench program tests a failure ordinary task benchmarks miss: two episodes can have the same visible outcome while requiring different durable repairs. Its first tier should seed matched software-maintenance cases whose latent defects belong in different persistence loci, then compare a cross-surface adjudicator with outcome-only, fixed-locus, human-rule, and simpler routing baselines.

The denominator must include every proposed portfolio, denial, unknown route, repair, retry, guard, stale lease, rollback, escaped descendant, and governance cost. Measures should include held-out task quality, placement regret, recovery, useful throughput, latency, compute, human burden, and lifecycle carrying cost. LocusBench is presently a benchmark proposal, not a reported result or support transition.

71.15 Source crosswalk

Source ID	Title	Layer	Planned use	Readiness
reflexive_router_whitepaper	The Reflexive Router	pre_deliberative_reflexive_routing_control_plane	ReflexBench tracks and ingress modes, matched comparator matrix, independent route/outcome/effect evaluation, adversarial controls, complete denominators, and joint usefulness/safety/effect/cost measures.	source note available
benchmaxxing	Benchmaxxing: The Performance Ratchet	benchmarks_evidence	Benchmark lifecycle, wall diagnosis, anti-Goodhart safeguards, and regression/frontier discipline.	source note available; local raw cache available
rmi	Ratcheting Modular Intelligence	capability_ratchet	Benchmark pressure, residual escrow, verified modular capability, regression preservation.	source note available; local raw cache available
cognitive_loop_closure	Cognitive Loop Closure	procedural_memory	Repeated cognition should become procedural memory / verified tools.	source note available; local raw cache available
uat	Unified Adaptive Tribunal	evaluation_refinement	Retrieval-bounded verification, adversarial review, and proposition tiers.	source note available; local raw cache available
tokenmana	TokenMana	resource_economics	Resource and load-stability effects on benchmark/review capacity.	source note available; local raw cache available
moecot	MoECOT-Agent Architecture Whitepaper	implementation_reference	Runtime benchmark/readiness context; not reproduced in this repo.	source note available; connector or recovery required
road_to_agi	Road To AGI	strategic_roadmap	Source-reported benchmark and remaining-work context.	source note available; local raw cache available
project_theseus_whitepaper	Project Theseus Whitepaper	report_first_rmi_prototype	Report-first RMI implementation reference.	source note available
theseus_architecture_gate	Theseus Architecture Gate	readiness_gate_governance	Gate snapshot and architecture promotion checks; not rerun here.	source note available
theseus_self_evolution_system	Theseus Self-Evolution System	recursive_self_improvement_governance	Evidence-first self-evolution lane and outcome ledgers.	source note available
theseus_circle_transfer	Theseus Circle Calculus Transfer Lane	proof_contract_transfer	Structural fixture transfer with explicit non-claims.	source note available
cca_project	Compiled Cognitive Architecture project	local_project_lineage	Closure-authority inheritance, proxy/benchmark divergence, retry-lineage ceilings, and checkpoint/tokenizer/runtime trace discipline.	source note available
moecot_manifest_project	MoECOT Manifest compiler-era project	local_project_lineage	Internal/external scoreboard separation, contamination boundaries, and external-holdout reality.	source note available
beastbrain_project	BeastBrain historical AI system project	local_project_lineage	Simulated metrics, non-updating routers, placeholder counts, and evaluator-dependence failures.	source note available
bugbrain_project	BugBrain bare-metal neuro-symbolic intelligence project	local_project_lineage	Capacity versus active state, fixed-weight cognitive paths, status workloads, host/Pi identity, and stale green reports.	source note available
corbens_trainer_project	Corben’s Trainer epistemic training and evaluation control plane	local_project_lineage	Metric authenticity, seed identity, quarantined-run inheritance, stale promotion, and checkpoint completion.	source note available
corbens_best_model_possible_project	Corben’s Best Model Possible recurrent-model and mechanism laboratory	local_project_lineage	Metric-origin algebra, representation-capacity gate, retry discipline, and checkpoint-to-output binding.	source note available
ext_mem0_2025, ext_mlcommons_croissant_1_1_2026, ext_inspect_ai_2024, ext_agentdojo_2024, ext_darwin_godel_machine_2025, ext_adas_2024, ext_dynamic_compute_recurrent_transformers_2026	Current memory, dataset, evaluation, security, self-improvement, architecture-search, and adaptive-compute comparators	external_benchmark_infrastructure	Use to expand benchmark/version/evaluator/adaptive-attack and complexity-controlled ratchets while preserving judge, contamination, evaluator-capture, transfer, and generalization limits.	source notes available; no local reproduction or benchmark promotion
portia_synapse	PortiaSynapse	routing_training_and_dkl_navigation	Successor benchmark-authenticity case: bind implementation, test manifest, phase gates, exact coordinate/route/task metrics, calibration, memory isolation, and matched ablations.	source note available; cache status: cached_existing
spider_synapse	SpiderSynapse	routing_training_and_dkl_navigation	Negative-result lineage for retaining plateaus, complete denominators, path-level diagnostics, and causal recovery rather than laundering a successor story.	source note available; cache status: cached_existing

Those failures prevent a semantic or model conclusion even though exact-object and byte-count submetrics were informative.

71.15.1 Manifest source assignment reconciliation

These rows keep Benchmark Ratchets and Anti-Goodhart Evidence’s manifest assignments visible at their recorded review boundary. Passage review does not establish local reproduction, performance, safety, deployment, or support-state movement.

Source	Intake role	Boundary
deterministic_capability_compilation	Passage-reviewed Corben architecture source: Deterministic Capability Compilation: A Capability-Preserving Ladder from Executable Scaffolds to Governed Adaptive Agents. Corben-authored July 2026 architecture and research program for compiling executable scaffolds into contract-bound experts and linked Neural Capability Objects while retaining semantic obligation mass balance, candidate-specific translation validation, fallback, residual escrow, authority ceilings, reification, and effect-complete recovery. Existing chapters are upgraded first; no foundry implementation, learned-capability result, preservation result, safety result, SOTA result, AGI, ASI, or support-state promotion is inferred.	No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
platonic_world_model	Metadata-first comparator: The Platonic World Model: A Semantic Constitution for Grounded, Proof-Carrying, Self-Editing Artificial Intelligence. Corben-authored July 2026 conceptual architecture and falsifiable research program for semantic continuity through stable Form lineages, immutable semantic versions, typed Essence Contracts, six mutually constraining planes, explicit proposition-attestation-commitment-proof separation, branch-protected world dynamics, qualified grounding, semantic transactions, runtime packet compilation, and federated mappings. Existing chapters are upgraded first; no implemented substrate, benchmark result, philosophical solution to grounding, safety result, SOTA result, AGI, ASI, or support-state promotion is inferred.	No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row.
rgs	Passage-reviewed comparator: Ratcheting Generative Systems. Supplies the precursor unified ratchet joining benchmark pressure, mastery and subgroup floors, critical-failure vetoes, stalled-effort threshold decay, frontier momentum, residual escrow, public calibration, loop closure, three execution modes, four ledgers, and a seven-level intervention ladder.	Conceptual precursor in the same author-side lineage as RMI, Benchmaxxing, CGS, and Cognitive Loop Closure; overlap is not independent corroboration. The 90/70 thresholds and portfolio percentages are illustrative, and no benchmark, intervention, tool, safety, transfer, or capability result exists. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
coherence_exchange	Passage-reviewed comparator: The Coherence Exchange. Supports contestable evidence movement through structured claim/evidence units, verification supply chains, audit, fork/exit/contestability language, and governance/accounting frames for knowledge claims.	Reviewed source note only because authenticated connector text is not published here; epistemic-liquidity and exchange metaphors are not implemented economics, external corroboration, security evidence, governance evidence, or benchmark evidence. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_claw_swe_bench_2026	Passage-reviewed comparator: Claw-SWE-Bench: A Benchmark for Evaluating OpenClaw-style Agent Harnesses on Coding Tasks. Supports binding coding-agent outcomes to a fixed model, harness, workspace, patch extraction, evaluator, runtime budget, and cost rather than attributing a harness result to the model alone.	Primary preprint comparator only; no reported task, score, harness, cost, contamination control, or safety result was reproduced here. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_muse_unlearning_2025	Passage-reviewed comparator: MUSE: Machine Unlearning Six-Way Evaluation for Language Models. Supports separating memorization, privacy leakage, retained utility, removal-scale behavior, and sequential sustainability instead of using one forgetting score.	No MUSE corpus, method, 7B model, privacy probe, scale test, or sequential deletion request was reproduced by the local small policy network. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_unlearning_benchmarks_weak_2024	Passage-reviewed comparator: Position: LLM Unlearning Benchmarks are Weak Measures of Progress. Supports treating unlearning benchmark validity, target ambiguity, forget/retain dependence, and benign perturbation robustness as separate evidence burdens.	The position paper raises the interpretation standard but does not validate the local workload or establish influence, privacy, legal erasure, or storage erasure. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_openunlearning_2025	Passage-reviewed comparator: OpenUnlearning: Accelerating LLM Unlearning via Unified Benchmarking of Methods and Metrics. Supports versioned unlearning methods, standardized execution interfaces, public checkpoints, diverse evaluations, and meta-evaluation of metric faithfulness.	The framework, methods, checkpoints, evaluations, and meta-evaluations were not run; standardized record shape does not establish semantic validity or erasure. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
kernel_english_residual_compiler	Metadata-first comparator: Kernel English with Hierarchical, Interaction-Amortized Residuals: A Dual-Vocabulary Cognitive Compiler for Efficient Language-Model Reasoning. Corben-authored July 2026 architecture proposal for KERC: protected-object capture, uncertainty-aware normalization, sense-aware Kernel IR, dual surface/core vocabularies, a four-level interaction-amortized residual ledger, exact object storage, grammar-aware macro fusion, structured answer packets, rendering, round-trip verification, versioned migration, and complete rate-compute-fidelity evaluation. Existing chapters are upgraded first; no implementation, benchmark, novelty, efficiency, fidelity, safety, transfer, SOTA, AGI, ASI, or support-state result is inferred.	No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row.
ext_swe_rebench_v2_2026	Passage-reviewed comparator: SWE-rebench V2: Language-Agnostic SWE Task Collection at Scale. Adds a current multilingual natural repository-task construction funnel, full-suite executable oracle, per-instance pathology metadata, and setup/clarity ablations that expose why benchmark membership is not construct validity.	The released tasks and automated labels are not locally validated en masse; post-snapshot filtering reduces but does not eliminate contamination, and no local benchmark score is established. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_mlperf_training_v6_2026	Metadata-first comparator: MLPerf Training v6.0. Official current measurement comparator for fixed datasets and quality targets, repeated time-to-quality, system metadata, divisions, variance, and corrected results. No MLPerf run is performed and the benchmark does not establish safety or complete run integrity.	No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row.
relational_dimension_compiler	Passage-reviewed comparator: The Relational Dimension Compiler: Adaptive Polyadic Cognition with Bounded Computational Arity and Unbounded Semantic Structure. Adds the RODIE evaluation program: minimum relational order, exact role binding, latent topology, interventions, dynamic identity, reversible contraction, object-field coupling, branch separation, compilation reuse, proposal recall, no-regret routing, transfer, and complete lifecycle cost.	RODIE is a proposed suite with no corpus, independent evaluator, benchmark run, result, construct-validity receipt, or evidence transition. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_cultural_alignment_llms_2024	Metadata-first comparator: Investigating Cultural Alignment of Large Language Models. Empirical study of cultural alignment patterns in selected language models and measurements. It supports explicit population, language, and instrument scope; it does not establish stable national values or a universal measure of cultural alignment.	No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row.
ext_multilingual_evaluation_state_2026	Metadata-first comparator: The State and Fate of Multilingual Contextual Evaluation in the NLP World. Research survey and analysis of multilingual contextual evaluation. It motivates language-by-task coverage and measurement reporting; it does not establish equivalent capability or safety across languages, dialects, or sociocultural settings.	No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row.
capability_ratchet_whitepaper	Passage-reviewed comparator: The Capability Ratchet. Full authenticated connector text section-audited. Synthesizes benchmark, procedural, and structural ratchets; benchmark and tool lifecycles; an intervention ladder; interpreter/compiled/reflex runtime modes; total-cost tool compilation; and anti-Goodhart controls. Same-author synthesis, not independent evidence for Benchmaxxing, Cognitive Loop Closure, RGS, or RMI.	No independent benchmark campaign, tool compiler, architecture-selection study, or measured capability improvement. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
attd	Passage-reviewed comparator: Assembly-Theoretic Technical Debt: A Deterministic Outer Loop for Self-Improving Codebases. Full authenticated connector text section-audited. Adds historical, vector-valued structural-debt governance: artifact-class separation, intrinsic assembly burden, reuse failure, role entropy, lineage, rolling residue, debt pressure, verified simplification credit, local caps, growth guards, deterministic GREEN/YELLOW/RED admission, bounded maintenance packets, abstention, and four-arm long-horizon evaluation.	Assembly theory is design inspiration; no universal debt law, analyzer, threshold calibration, causal ablation, long-horizon maintenance result, or safe self-modification claim. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
temporal_coil_research	Passage-reviewed comparator: Temporal Coil Research. Full experiment note reviewed. Reports 11 variants, three seeds, six rounds per variant, split winner frequency, small mean deltas, flat pass/reward/holdout lanes, separation dominated by the collapse composite, and one narrow threshold-tuned seed. Preserved as an inconclusive source-reported result and placement-confounding lesson, not proof of benefit or general failure.	Source-reported live-endpoint runs only; no local reproduction, independent evaluator, causal component effect, transfer, or general coil conclusion. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
regret_engine	Passage-reviewed Corben architecture source: The Regret Engine: Governed Counterfactual Learning Signals for Continual Adaptation, Prospective Risk Control, and Self-Correction in Artificial Agents. Corben-authored August 2026 conceptual architecture and research program for decision-time-fair Governed Counterfactual Regret, immutable Decision Capsules, admissible comparator contracts, sparse Regret Tensors, append-only Regret Packets, prospective regret control, regret-aware replay, regret-to-rule compilation, three update clocks, root-cause adjudication, and bounded update leases. Existing chapters are upgraded first; no implementation, experiment, reproduction, causal-identification result, formal proof, safety result, support transition, SOTA, AGI, or ASI is inferred.	The bibliography and Markdown figure companions were not supplied; the DOCX embeds its visual material. All propositions, algorithms, experiments, and architecture claims remain proposed rather than independently validated. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_elizaos_agent_runtime_2026	Passage-reviewed comparator: elizaOS Agent Runtime and Scenario Runner. Supplies a current implementation comparator for deterministic and live runtime scenarios, per-turn and final assertions, trajectory export, and an explicit separation between in-process diagnostics and externally observed provider qualification.	Official software and documentation review only; no scenario, model, provider, observer signature, trajectory, benchmark, cost, failure, or qualification result was executed or reproduced. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
assurance_shift_learning	Passage-reviewed comparator: When Success Stops Teaching: Assurance-Shift Learning and Governed Residual Boundary Learning for Mature AI Systems. Adds selection-gap diagnosis, the benchmark-saturation versus capability-maturity distinction, and SaturationShiftBench as an equal-total-cost falsification program with attack injections and ablations.	The benchmark is proposed only; no condition, crossover, endpoint, or superiority result has been executed. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
adjudicated_persistence	Passage-reviewed comparator: Adjudicated Persistence: Governing the Transition from Experience to Durable Structure in Adaptive Systems. Contributes LocusBench: matched outcomes with different latent defects, per-locus and cross-locus baselines, placement regret, and whole-lifecycle commitment costs.	Conceptual author framework and benchmark proposal; no local implementation, empirical result, independently checked proof, safety result, or support movement. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
forward_transfer_program_synthesis	Passage-reviewed comparator: From Compression to Forward Transfer: Evaluating Reusable Knowledge in Program Synthesis. Adds causal controls for reusable-knowledge evaluation: frozen rounds, absence and matched placebo conditions, removal and necessity tests, a library-search factorial, exact verifier outcomes, and complete task-lineage denominators.	Experimental blueprint only; no benchmark condition, ablation, transfer effect, or promotion decision was executed. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.

71.16 The language-by-task denominator

Multilingual coverage is not the count of translated benchmark rows. A ratcheted evaluation records a matrix over language or dialect, task, domain, interaction mode, population, evaluator competence, translation provenance, and risk class. Every empty or low-quality cell stays visible. Macro averages are accompanied by the worst material cell, coverage-weighted results, and explicit exclusions.

Translation-derived items are marked separately from naturally authored items. Cross-language contamination, shared source passages, cultural-instrument validity, script and tokenization effects, answer-format bias, and evaluator disagreement receive their own controls. A benchmark cannot claim cultural alignment by assigning one national score, and it cannot claim multilingual safety because a dominant-language judge approved translated outputs.

The ratchet condition is claim-specific: expansion to a new language, community, or task reopens the evidence boundary. Improvements in high-resource cells cannot compensate for regression in a consequential low-resource cell without a prospectively justified decision rule and affected-population accounting.

71.17 Summary

Benchmarks are pressure surfaces, not truth machines. They become useful when every result has a run record, baseline, origin class, target-capacity check, inherited floors, retry lineage, checkpoint/output binding, residuals, regressions, anti-Goodhart checks, and a conservative support-state effect.

That pressure changes future behavior only if policy optimization preserves provenance, rollback, and governance gates while learning from rewards, preferences, verifier outputs, and benchmark signals.

The durable rule is that benchmarks should improve memory, not erase it. A good ratchet makes the next system harder to fool and easier to compare; a bad one gives the next system a number to optimize and a story to overclaim. Benchmark ratchets therefore treat each run as a governed evidence event: environment, seed, task, baseline, metric, failure cases, residuals, and downgrade triggers all matter. Anti-Goodhart discipline is not pessimism. It is how the system keeps learning signals useful after models, planners, and policies begin adapting to them. Once benchmarks become ratchets, policy optimization can update behavior without turning every reward into truth or every score into permission.

71.18 Evidence reconciliation (2026-07-16)

The invariant protocol, field meanings, and inference limits are stated once in Living Book Methodology. This packet contains only the chapter-specific projection; its authoritative per-atom rows are the benchmark-ratchets-and-anti-goodhart-evidence slice of experiments/claim_family_terminal_coverage/results/result.json.

The core remains blocked after full attempt at argument support. The strongest family attempt was Full-state update and unlearning causal campaign. Its exact boundary is: Broad unlearning claim is narrowed with no support promotion; behavioral removal is not influence, privacy, legal, storage, backup, or descendant erasure. Across 74 atoms, the terminal ledger records 74 blocked_after_full_attempt.

Chapter-specific field	Value
Family / atom denominator	CF-07 / 74 atoms
Terminal dispositions	74 blocked_after_full_attempt
Core	benchmark-ratchets-and-anti-goodhart-evidence.core: blocked_after_full_attempt at argument
Core attempted / missing lanes	causal, empirical, executable, formal, source-synthesis / normative, transfer
Attempted local lanes	causal, empirical, executable, formal, source-synthesis
Missing or unproved lanes	normative, transfer
Strongest family bundle	Full-state update and unlearning causal campaign (end_to_end): One adequate five-seed, seven-arm terminal campaign preserving two prior instrument failures and separate behavioral, influence, privacy, lineage, storage, backup, and descendant axes.
Negative controls	deletion retrain comparator; approximate mitigation arms; 15 rejecting mutations; failure-lineage preservation.
Accepted transitions	v1_0_pilot.benchmark_ratchets.no_change
Maximum inference	Broad unlearning claim is narrowed with no support promotion; behavioral removal is not influence, privacy, legal, storage, backup, or descendant erasure.
Reproduction / next burden	Replay scripts/validate_p4_m7_update_unlearning_v3.py and scripts/validate_claim_family_terminal_program.py; fill the named atom-specific lanes under a new prospective protocol.

71.19 Handoff

Benchmark Ratchets and Anti-Goodhart Evidence makes behavioral measurements harder to game, but a score cannot reveal which internal mechanism produced the result. White-Box Evidence, Interpretability, and Activation Governance receives exact model/checkpoint and behavioral-instrument identity and decides whether model-internal observations or interventions are admissible, bounded, stable, and non-authorizing evidence. Capability Thresholds and Deployment Commitments can then consume both behavioral and white-box packets without turning either into an automatic release decision.