Run GenesisBench

Evaluate unfamiliar-language reasoning with frozen context, deterministic scoring, reproducible runs, and evidence-backed contamination labels.

What GenesisBench measures

Status: normative explanation and tutorial. The machine authority is docs/spec/GENESISBENCH_PROTOCOL_v0.1.json; this guide cannot broaden it.

GenesisBench asks an agent to learn GenesisCode from an immutable repository/runtime/documentation snapshot and complete real language-engineering tasks. It measures repository learning, semantic reasoning, capability minimization, tool use, debugging, and artifact construction without relying on mature-language answer memorization.

The benchmark does not claim that GenesisCode is absent from a model’s training data merely because the language is new. Contamination claims follow evidence, and incomplete provenance is unknown.

The normative authorities are:

• Machine profile
• Profile schema
• Eligibility schema
• Contamination-attestation schema
• Public adaptation-manifest schema
• Embedded-local hardware-evidence schema
• Disclosed scaffold-manifest schema
• Fixed reference agent
• Fixed reference-agent schema
• Controlled ablations
• Reference trace schema
• Canonical front door
• Open Agent harness
• Retained Open Agent v0.1 authority
• Open Agent campaign schema
• Open Agent campaign-report schema
• Open Agent predeclaration schema
• Open Agent run schema
• Local-model preselection and inventory schema
• Closed adapter profile
• Closed adapter-profile schema
• Execution-run schema
• Deterministic bundle schema
• Signed registry authority
• Submission claim schema
• Registry policy schema
• Registry result schema
• Append event schema
• Checkpoint schema
• Static leaderboard schema
• Task benchmark
• Scoring authority
• Run schema
• Predeclared analysis plan
• Observation matrix schema
• Analysis report schema
• Real-model baseline protocol
• Baseline predeclaration schema
• Baseline evidence schema
• Baseline publication schema
• Benchmark card
• Failure taxonomy
• Baseline methods
• Held-out epoch commitments
• Temporal epoch opening audit
• Capability lease overlay

This guide explains those authorities; it cannot alter them.

Verify the profile

Use a full local clone so the frozen Git commit is available. Validation is offline and read-only:

python3 scripts/lib/genesisbench_protocol.py --check --self-test

The verifier checks:

1. The immutable commit and Git tree.
2. A complete SHA-256 manifest over all 1,775 frozen files.
3. Independent runtime, documentation, and benchmark component identities.
4. Every linked schema, card, task, scorer, run verifier, MCP catalog source, and integration test.
5. Closed context, tool, capability, attempt, model-disclosure, visibility, scoring, contamination, track, cohort, and eligibility policies.
6. Exact typed prompt order and context-artifact coverage for the selected cohort.
7. Adversarial controls that are re-signed after mutation, so semantic rules rather than stale outer hashes cause rejection.

No update, model invocation, network request, or score execution occurs in check mode.

Understand the task surfaces

The public suite contains nine immutable task lineages: generation, completion, repair, refactor, policy minimization, replay investigation, performance repair, package migration, and deployment. Each lineage has small, medium, and large child context conditions. Those 27 condition cells are useful for context ablation, but they remain nine independent challenges. Their references are intentionally inspectable development oracles. They are useful for learning, harness conformance, and longitudinal diagnostics, but they are not clean held-out evidence.

Use the fixed reference agent

Cold Acquisition fixes everything except the immutable model revision. genesisbench-reference-agent-v0.1 binds the system prompt, six-role typed prompt order, cards and grammar, deterministic integer-only retrieval, one agent with no subagents or provider tools, the exact 20-route MCP catalog, content-addressed semantic transactions, diagnostics and bounded repair, finite compute/resource budgets, and complete event capture. Adapters may transport the typed request and response, but may not rewrite semantics, inject authority, retry invisibly, or enable provider tools. Any scaffold change creates a new profile and cohort.

The public ablation authority applies eight conditions to each of the same nine lineage IDs: core card only, fixed context, retrieval, structured diagnostics, semantic patching, grammar constraint, one attempt, and one bounded repair. The 72 cells are repeated observations, not independent tasks. Only six within-lineage contrasts are predeclared, and cross-lineage pooling must remain clustered or hierarchical.

Validate the profile, all 72 condition bindings, a canonical trace, and 25 adversarial controls offline:

python3 scripts/lib/genesisbench_reference_agent.py --check --self-test

Compile the exact public input/context plan for a condition without reading its reference oracle:

python3 scripts/lib/genesisbench_reference_agent.py \
  --plan --case generation-small --ablation retrieval

The compiler is not a model adapter. genesis bench preserves this plan while supplying the canonical provider/local transport, execution, replay, and bundle layer.

Execute through one front door

Inspect the public suite and closed adapter classes without model execution:

genesis --json bench inspect --case generation-small
python3 scripts/lib/genesisbench_front_door.py check --self-test

Run the deterministic mock only for harness conformance; it is permanently unranked:

genesis --json --selfhost-artifact selfhost/toolchain.gc bench run \
  --case generation-small \
  --adapter benchmarks/genesisbench/v0.1/adapters/deterministic-mock.json \
  --out .genesis/bench/conformance
genesis --json bench validate-run \
  --run .genesis/bench/conformance/run.json
genesis --json --selfhost-artifact selfhost/toolchain.gc bench replay \
  --run .genesis/bench/conformance/run.json

The five adapter classes are hosted API, loopback OpenAI-compatible server, direct local runtime, command plugin, and deterministic mock. They share one strict request/response contract. Adapters may frame bytes for transport, but cannot alter candidate semantics, add provider tools, broaden context, hide retries, or emit undeclared provider facts. Hosted adapters receive one exact HTTPS endpoint and one named credential; local HTTP is loopback-only; process adapters are executable-digest-bound, use finite output and wall limits, and kill and reap their process group on timeout or cancellation. A direct local runtime additionally verifies the model artifact digest.

Every attempt is retained. A provider failure, timeout, or cancellation becomes an immutable invalid run rather than disappearing. Secret values never enter evidence; only the declared variable name and presence are recorded. Replay validates all fields and bytes without adapter, endpoint, credential, executable, or model access, then independently rescores successful candidates.

Run a repository-editing agent

The five fixed adapters above answer the Cold Acquisition transport question. They do not give a command plugin a repository workspace. Repository-editing systems use the separate Open Agent harness and remain in the open-agent track.

Predeclare Codex CLI with Luna at xhigh before inference. A provider alias is useful exploratory evidence but is not immutable, so this example deliberately omits --immutable-revision:

CODEX=/Applications/ChatGPT.app/Contents/Resources/codex
genesis --json bench agent-campaign-plan \
  --campaign codex-luna-xhigh-2026-07 --phase reality-gate \
  --case completion-small --case deployment-small --case generation-small \
  --case package-migration-small --case performance-repair-small \
  --case policy-minimization-small --case refactor-small --case repair-small \
  --case replay-investigation-small \
  --runner codex-cli-hosted \
  --agent-executable "$CODEX" \
  --model luna \
  --model-revision provider-alias:luna@2026-07-17 \
  --reasoning-effort xhigh \
  --timeout-ms 900000 \
  --hardware-class apple-silicon-local \
  --out .genesis/bench/predeclarations/campaign.json
genesis --json bench agent-plan \
  --case completion-small \
  --campaign-predeclaration .genesis/bench/predeclarations/campaign.json \
  --out .genesis/bench/predeclarations/completion-small.json
genesis --json --selfhost-artifact selfhost/toolchain.gc bench agent-run \
  --campaign-predeclaration .genesis/bench/predeclarations/campaign.json \
  --predeclaration .genesis/bench/predeclarations/completion-small.json \
  --agent-executable "$CODEX" \
  --out .genesis/bench/open-agent/completion-small
genesis --json bench agent-validate \
  --run .genesis/bench/open-agent/completion-small/run.json
genesis --json --selfhost-artifact selfhost/toolchain.gc bench agent-replay \
  --run .genesis/bench/open-agent/completion-small/run.json

The campaign is one immutable commitment to every reality-gate case, shared model/runtime/capability/resource facts, non-selective stop rules, and the expected attempt count. Each attempt must bind that campaign before execution. The v0.2 harness stages outside the repository ancestry, reconstructs and byte-checks the frozen source snapshot under a read-only input root, and deterministically tears down the protected snapshot after atomically publishing evidence. Only the case inputs are writable. One ephemeral JSONL Codex attempt runs with no resume, hidden retry, approval, or additional writable root. The post-run audit rejects source changes, symlinks, extra candidate paths, non-editable drift, malformed events, timeout, capture overflow, or nonzero exit. Replay validates retained evidence and independently rescores without resolving the Codex executable or contacting the model.

The retained v0.4 schema can describe lmstudio or ollama, but local score-producing use is currently prohibited. A July 18 readiness audit proved that Codex’s LM Studio path attempted provider-managed download instead of consuming the preselected offline MLX snapshot; an unisolated custom provider received ambient ChatGPT authorization and retried a failing request five times; and the inner workspace-write sandbox does not prove read isolation from unrelated host files. These are harness defects, not model failures. Do not disguise an MLX endpoint as LM Studio or publish a local result from v0.4.

The score-blind preselection is still reproducible:

python3 scripts/lib/genesisbench_local_models.py --check --self-test

It binds five cached snapshots and revision-pinned license/model-card evidence, selecting Apache-2.0 Qwen3 4B 4-bit and Qwen3 8B 4-bit before any quality score. Local execution waits for the successor authority to provide a byte-bound Responses-to-MLX adapter, auth-free ephemeral homes, zero transport retries, disabled ambient web/multi-agent/plugin surfaces, exact observed scaffold binding, an outer filesystem/network allowlist, and hard adapter/model-server cancellation and reaping. Each model/runtime/artifact/scaffold/hardware combination remains a distinct cohort, and each must complete one nine-class reality gate before any 27-condition expansion.

The retained July 17 Luna campaign completed all nine expected attempts, and all nine validate and replay. The provider rejected luna as unsupported for the ChatGPT-backed Codex account before task execution, so the result is an authentic unranked availability failure, not a GenesisCode capability score. Its report also records that v0.1 discovered a repository skill through parent traversal. The 27-condition expansion is therefore forbidden; v0.2 fixes ancestry isolation and temporary-snapshot cleanup without retrying or rewriting those results.

The separately predeclared exact-slug campaign used gpt-5.6-luna at xhigh. Its repaired v0.3 reality gate retained eight independently rescored successes and one invalid deployment attempt, so it also correctly stops before 27-condition expansion. Both the v0.2 verified attempt and all v0.3 verified attempts retain campaign-bound tool archives; replay uses those exact executable and self-host bytes after the repository runtime or scoring authorities change.

Bundle and stage the evidence locally:

genesis --json bench bundle \
  --run .genesis/bench/conformance/run.json \
  --out .genesis/bench/conformance.gcbundle
genesis --json bench submit \
  --bundle .genesis/bench/conformance.gcbundle \
  --claim .genesis/bench/claim.json \
  --outbox .genesis/bench/outbox \
  --submitter example-lab \
  --key .genesis/keys/example-lab.toml

The bundle is byte-reproducible gzip/USTAR with fixed metadata and a closed artifact manifest. submit performs no network operation. It validates a closed claim, signs a domain-separated DSSE statement through the Rust Ed25519 boundary, and writes a content-addressed immutable outbox. The claim binds one target registry policy, evaluation set, track/scaffold/training facts, immutable model/runtime identity, contamination and hardware evidence, and separately reported economics.

Initialize and operate a local registry with a distinct operator key:

genesis --json bench registry-init \
  --registry .genesis/bench/registry \
  --policy .genesis/bench/registry-policy.json \
  --operator-key .genesis/keys/registry-operator.toml
genesis --json --selfhost-artifact selfhost/toolchain.gc bench registry-admit \
  --registry .genesis/bench/registry \
  --submission .genesis/bench/outbox/SUBMISSION.submission.json \
  --bundle .genesis/bench/conformance.gcbundle \
  --operator-key .genesis/keys/registry-operator.toml
genesis --json --selfhost-artifact selfhost/toolchain.gc bench registry-verify \
  --registry .genesis/bench/registry
genesis --json --selfhost-artifact selfhost/toolchain.gc bench registry-build \
  --registry .genesis/bench/registry \
  --out .genesis/bench/static-leaderboard

Admission verifies pinned submitter provenance, every bundle byte, run closure, and strict replay, then independently re-scores before deriving a result. It appends one operator-signed hash-chain event and a signed checkpoint over the complete signed event prefix. Verification re-extracts every retained bundle and rederives every result; signatures alone are insufficient. Unknown files, orphaned objects, gaps, deletions, byte rewrites, and hidden history fail closed.

Static ranking is restricted to complete evaluation sets inside one exact cohort. It orders verified solve rate first, conditional quality among solved lineages second, then capability excess, context bytes, tool calls, and repair calls. Invalid partial quality is zero. Cost, latency, energy, provider identity, and stable system IDs never improve substantive rank; exact ties remain ties. The publication retains every attempt, failure, invalid, abstention, missing lineage, per-class result, model/runtime fact, contamination/hardware fact, and offline replay command.

Analyze lineages, not repeated conditions

The predeclared v0.1 analysis chooses the small-context child as the primary condition and calculates headline verified solve rate over the nine lineage IDs. Every expected system, epoch, and case cell must be present; missing, invalid, and abstained are published separately and all remain in the solve denominator. Conditional quality includes solved lineages only and publishes its own denominator.

Solve uncertainty uses a 95% Wilson interval over lineages. System comparisons require the same cohort and paired primary lineage set, publish a risk-difference effect and uncertainty, use an exact two-sided McNemar/binomial test, and apply Holm-Bonferroni correction across the predeclared pair family. A comparison that does not clear the corrected threshold is indeterminate; decimal score order alone cannot become a rank claim. Context-condition summaries remain clustered by parent lineage.

Saturation requires at least three distinct model families above 90% verified solve rate for two consecutive ranked epochs. Public conformance evidence cannot trigger it. Validate the canonical complete matrix and report offline:

python3 scripts/lib/genesisbench_analysis.py --check --self-test

Ranked evaluation uses a non-public oracle and the active held-out commitment authority. The active epoch-2026-07-b contains 90 independently salted lineages, exactly ten per core class. Its public metadata is balanced across three difficulty bands, ten author/generator identities, five generator families, ten domains, and ten acceptance shapes; no generator family controls more than 20% of ranking weight. All ranking weight is fresh for this epoch.

Temporal-clean evaluation additionally requires a task precommitment made after the target model’s immutable release. The epoch records the precommit boundary and requires run-specific immutable release evidence; merely choosing a new model or a new-looking task is insufficient. Private tasks, salts, inputs, prompts, and oracles never belong in the repository, documentation site, training corpus, model retrieval root, or public run bundle.

Verify public scale, balance, chronology, custody, lifecycle, and 30 adversarial controls offline:

python3 scripts/lib/gc_held_out_evaluation.py --check --self-test

The tracked temporal audit is intentionally hash-only. It records that all 90 salted commitments, metadata bindings, and private quality contracts were opened under local mode-0600 custody, but CI cannot independently inspect private payload quality without violating the evaluation firewall. Custodians can reproduce the opening check locally with --verify-private; they must never publish the pack or feed it to an authoring or training context.

The epoch rotates within 90 days, or sooner on leakage or statistical saturation. Retired commitments remain immutable. Scheduled payload opening waits at least 30 days after retirement, and disclosed work permanently becomes contaminated public development material.

Maintain useful temporal overlays

Every scaled epoch must introduce at least one post-release package, protocol, data structure, capability, or semantic-patch operation that remains useful after ranking weight retires. Arbitrary riddles and benchmark-only APIs are rejected.

The current overlay is GC-CAPABILITY-LEASE-v0.1, a general deny-by-default lease protocol for finite, exact-scope authority. It uses an explicit logical step rather than host time, content-addressed scope, bounded uses, append-only issue/consume/revoke transitions, and replay-bound prior/next state identities. Validate its independent maintenance gate with:

python3 scripts/lib/gc_capability_lease.py --check --self-test

Control tools and capabilities

The Genesis MCP 2025-11-25 interface is the canonical interactive surface. Its exact 20 routes are generated from the production CLI and frozen by the catalog authority. Artifact-only response mode is retained for minimal and local baselines, but it is a separate cohort.

Arbitrary shell, ambient filesystem access, ambient network access, wildcard capabilities, and agent-authored policy broadening are forbidden. Effectful commands use only task-owned exact capabilities, finite hard resource limits, complete deterministic logs, and replay. Provider transport is separate from candidate tool authority and cannot become a hidden tool or context channel.

Fix attempts and model disclosure

A ranked case has one predeclared attempt. Best-of-N selection, hidden retries, cherry-picking, and post-result prompt changes are forbidden. Unranked experiments may use up to 16 attempts, but every success, failure, abstention, retry, request, response, candidate, token fact, and selection is retained.

Every run discloses model ID, immutable revision, provider, weights identity for local models, tokenizer, runtime, quantization, integer decoding controls, seed, stop policy, prompt retention, and secret policy. Mutable aliases such as latest are invalid. Missing training provenance defaults to unknown; it never becomes a clean claim.

Choose one benchmark track

Every run belongs to exactly one track. Tracks answer different questions and are never merged into one rank, average, or headline number.

Track	Question	Admission boundary
cold-acquisition	How well does the unadapted model learn GenesisCode through one controlled reference agent?	No GenesisCode-specific training and exactly genesisbench-reference-agent-v0.1; its system prompt, retrieval, loop, budgets, tools, and trace policy are immutable within a cohort. Ranked admission is open for conforming run evidence.
open-agent	How well does a disclosed agent system solve the work under the benchmark’s authority and resource ceilings?	No declared GenesisCode adaptation. Custom retrieval and orchestration are allowed only when fully recorded and content-addressed. Unknown training provenance is unranked.
genesis-adapted	How capable is a GenesisCode specialist built from public, auditable adaptation material?	declared-public training plus a lineage-manifest identity is mandatory. Raw and adapted results never share a cohort.
embedded-local	What can a completely offline, self-hosted appliance do within a reproducible memory envelope?	Local model, denied network, and measured peak or enforced hard-ceiling evidence for combined model/runtime resident bytes.

Embedded Local uses the smallest fitting hardware class: S is at most 4 GiB, M at most 16 GiB, and L at most 64 GiB. A host’s installed RAM is not footprint evidence. Missing measurement is unranked; a false class or non-offline execution is invalid.

The scaffold identity is the SHA-256 of an inventoried, closed manifest that records its system prompt, prompt algorithm, retrieval mode and algorithm/config identity, agent/subagent count, planner, repair loop, stop policy, tool catalog and operations, and exact authority/resource-ceiling policy. Task-specific prompt/input and retrieved context remain run evidence rather than silently changing the scaffold. Any legitimate scaffold change creates a new cohort.

Produce a reproducible run

A complete run records:

• immutable model, weights, tokenizer, and runtime identity;
• exact ordered prompts, cards, context, and retrieved artifacts;
• integer decoding and retry policy;
• every attempt, failure, abstention, request, response, and candidate;
• exact tool catalog, capabilities, transcript, and effect log;
• normalized host facts without usernames, hostnames, serial numbers, or absolute paths;
• canonical deterministic score and separate latency, cost, queue, and energy facts;
• a complete content-addressed artifact inventory.
• a closed, content-addressed track declaration with scaffold, training, inference, and hardware facts.

Validate the record without executing it:

python3 scripts/lib/gc_agent_benchmark_run.py --check --run RUN.json

A fully local model may use the pinned genesis.agent-model-runner.v0.1 effect. Replay must still succeed after the model bridge and weights are unavailable, proving transcript reproducibility rather than model regeneration.

Score the candidate

The scorer uses only deterministic artifacts and production GenesisCode execution:

python3 scripts/lib/gc_agent_scoring.py --score \
  --case CASE_ID \
  --candidate CANDIDATE_DIR \
  --genesis-bin ./target/debug/genesis \
  --selfhost-artifact selfhost/toolchain.gc

Quality is 10,000 basis points across semantics, obligations, effects, patch minimality, deterministic resource use, and policy scope. Required semantic, obligation, policy, or editable-scope failure makes the result invalid. Wall time, provider latency, API cost, and energy never buy quality points. A ranked result requires an independent byte-identical rescore; submitting a score hash is not independent verification.

Classify eligibility

Classify a validated run and its run-bound contamination evidence under the active profile:

python3 scripts/lib/genesisbench_protocol.py --check \
  --run RUN.json \
  --attestation ATTESTATION.json \
  --json

The deterministic result is invalid, unranked, or ranked. It binds the protocol, snapshot, run, case, track, observed context and interaction modes, validation facts, strongest evidence-supported contamination label, exact cohort, reason codes, and its own content identity. Structurally malformed or unbindable input exits nonzero and emits no eligibility record; this prevents rejected bytes from being mistaken for a signed result.

The canonical fixture is deliberately unranked:

python3 scripts/lib/genesisbench_protocol.py --check \
  --run examples/agent_benchmark_reproducibility/run.json \
  --attestation benchmarks/genesisbench/v0.1/contamination.fixture.json \
  --json \
  | cmp - benchmarks/genesisbench/v0.1/eligibility.fixture.json

It uses a public answer and a conformance model, and it has not been independently rescored by a registry admission worker. A perfect score does not erase those facts.

Interpret contamination labels

Label	Required evidence
temporal-clean	Immutable model release; later task precommitment; active undisclosed epoch; commitment and custody proof; training cutoff; no known exposure.
declared-uncontaminated	Complete provenance and explicit non-exposure attestation, but no independently proven temporal ordering.
unknown	Evidence is incomplete and no exposure is known. This is the default.
declared-contaminated	Task, oracle, equivalent answer, or evaluation material is known to have been exposed. This overrides clean claims.

Never collapse these labels into one leaderboard. The closed cohort record binds track, scaffold, language-profile artifact, task epoch, context mode, interaction/tool mode, attempt-policy identity, hardware class, protocol identity, task visibility, and contamination label. rankingCohort is the cohort record’s SHA-256 identity, not an editable display label.

Validate the construct

The scorer must distinguish GenesisCode engineering from reference imitation. The checked construct study executes nine non-reference alternatives and nine targeted invalid controls through the shipped binary, including noncanonical JSON/TOML layouts, different pure expressions, a different persistent-map repair, artifact-contract violations, authority broadening, scope escape, resource failure, and a constant that passes only the visible collection example. The collection repair is re-evaluated with unseen seeds appended by the evaluator; candidate source bytes are never compared with the reference patch for validity.

python3 scripts/lib/genesisbench_construct_validity.py --check --self-test

The retained report binds the benchmark, profile, scoring policy, held-out quality-contract audit, scorer, and study runtime. It requires 100% alternative acceptance, 100% targeted-negative rejection, zero surviving evaluator/scorer mutants, lineage-clustered deterministic bootstrap intervals, correct two-epoch saturation behavior, all five maintenance families, a maintained non-benchmark-only overlay, and two independent deterministic verifier implementations. Human or model preference is not an oracle. Active hidden verifier content remains commitment-bound and outside the public report.

Current boundary

The v0.1 protocol, schemas, verifier, four non-combinable tracks, frozen reference agent, eight controlled ablations over nine shared lineages, nine-lineage/27-context public practice suite, predeclared statistical analysis, construct-validity and anti-gaming study, scorer, canonical sixteen-command CLI, five closed Cold Acquisition adapters, separate capability-minimal Open Agent harness, strict replay, deterministic run bundles, signed append-only registry, lexicographic static leaderboard, 90-lineage active temporal epoch, maintained post-release capability-lease overlay, and deterministic eligibility and analysis fixtures exist now. Reproducible multi-model campaigns, benchmark card, and methods paper remain R1.4.o.

Until those gates close, report protocol-conformant local evidence without describing it as a production leaderboard or temporal-clean model comparison.

Versioning

Changes to snapshot identity, track admission, scaffold identity rules, hardware-class bounds, cohort keys, context order, tool routes, capability semantics, attempt rules, disclosure requirements, visibility classes, scoring authority, contamination classification, or eligibility rules require a new protocol version. A correction that changes a prior decision requires a new profile and explicit supersession record; historical evidence remains immutable.