71 Bench v4 Hardness Ladder
72 Bench v4 Hardness Ladder
Bench v4 begins where the v3 field-spell corpus stopped being discriminating enough. The seed ladder keeps the evidence narrow: local deterministic artifacts are executed repeatedly in repo-local tmp/ sandboxes so the fixture contract can be inspected before any model-surface claims are made.
72.1 Policy
Bench v4 hardness fixtures run only local deterministic tests inside repo-local tmp/ sandboxes. Model-surface runs are not simulated.
72.2 Seed Rungs
| Case | Rung | Hardness Axis | Weak | Repaired | n | Fixture |
|---|---|---|---|---|---|---|
| Disabled Account Status Ambiguity | ambiguity | conflicting docstring versus caller contract | 0/5 passed | 5/5 passed | 10 | fixture |
| Event Deduplication Hidden Order Invariant | hidden-invariant | order and replay semantics hidden behind simple dedupe wording | 0/5 passed | 5/5 passed | 10 | fixture |
| Tax Discount Misleading Context | misleading-context | stale inline comment contradicts current merchant contract | 0/5 passed | 5/5 passed | 10 | fixture |
| Beta Flag Blast Radius | blast-radius | narrow feature-flag change versus tempting billing overreach | 0/5 passed | 5/5 passed | 10 | fixture |
| Agentic Release Handoff | agentic | multi-step stack handoff with repo-local scratch and gate evidence | 0/5 passed | 5/5 passed | 10 | fixture |
72.3 Model-Surface Artifact Runs
These rows are model-produced artifacts, not hand-authored controls. Prompts include the public task context and starting files, but not the hidden grader or ground-truth JSON. The runner preserves each prompt, transcript, extracted artifact, extraction failure, and fixture-local execution result.
| Case | Surface | Weak | Repaired | Extraction Failures | n | Example Transcript |
|---|---|---|---|---|---|---|
| Disabled Account Status Ambiguity | codex-cli-default | 0/5 passed | 0/5 passed | 0 | 10 | transcript |
| Event Deduplication Hidden Order Invariant | codex-cli-default | 0/5 passed | 0/5 passed | 0 | 10 | transcript |
| Tax Discount Misleading Context | codex-cli-default | 0/5 passed | 0/5 passed | 0 | 10 | transcript |
| Beta Flag Blast Radius | codex-cli-default | 0/5 passed | 0/5 passed | 5 | 10 | transcript |
| Agentic Release Handoff | codex-cli-default | 1/5 passed | 1/5 passed | 0 | 10 | transcript |
72.4 Reviewer Import Contract
Additional non-Codex surfaces can enter through a manual import bundle. The validator checks schema fields, repo-local paths, provenance/evidence-class consistency, allowed artifact filenames, prompt exclusion of hidden grader files, and fixture-local execution against the private Bench v4 grader. It does not publish or accept the run; maintainer review remains a separate decision.
Template: manual-import-template.json
Validate a bundle:
python3 scripts/import_hardness_model_run.py validate examples/evaluations/hardness-v4/manual-import-template.json
python3 scripts/grimoire.py bench hardness-import examples/evaluations/hardness-v4/manual-import-template.json72.5 Reviewer Acceptance Decision
Validated import bundles remain intake records until a named maintainer accepts and publishes a decision. The pending decision template does not count as cross-surface evidence.
Decision template: hardness-intake-decision-template.json
Validate a decision:
python3 scripts/check_hardness_intake.py validate examples/evaluations/hardness-v4/hardness-intake-decision-template.json
python3 scripts/grimoire.py bench hardness-decision examples/evaluations/hardness-v4/hardness-intake-decision-template.json72.6 Interpretation
- The ambiguity rung tests whether an artifact resolves a stale docstring versus a stronger caller contract.
- The hidden-invariant rung tests whether an artifact preserves replay order and mutation boundaries that are easy to erase with a naive cleanup.
- The misleading-context rung tests whether an artifact ignores stale or wrong local guidance in favor of the executable contract.
- The blast-radius rung tests whether an artifact keeps a requested change narrow instead of altering adjacent behavior.
- The agentic rung tests whether a stack-shaped handoff preserves repo-local scratch, allowlists, gates, and human-review boundaries.
- These are execution checks for seed artifacts. They do not count as independent model-provider evidence, external adoption, or human review.
- Model-surface rows count only when prompts, transcripts, extracted artifacts, and execution results are preserved. They still do not count as external adoption or human review.
Raw results: results.json
Raw model-surface results: model-surface-results.json