69  Bench v2

70 Bench v2

Bench v2 is the replay contract for the grimoire’s evidence layer. It keeps model/tool surfaces explicit, separates project-owned runs from reviewer-supplied evidence, and records which parts of a score are deterministic versus human-scored.

70.1 Surfaces

Surface Kind Ownership Execution Credentials Redaction
codex-cli-default codex project-owned local-codex-cli no No real secrets; harmless canaries may appear only in fixture prompts and ground truth.
claude-code-safe claude-code project-owned local CLI, print mode, tools disabled yes Public fixtures only; outputs are redacted for fixture canaries and forbidden operational strings before publication.
manual-reviewer-import manual-import reviewer-supplied manual transcript import no Reviewer must redact private data and declare model/tool surface, fixture version, prompt path, transcript path, and notes.
local-deterministic-grader local-tool project-owned local read-only fixture execution no No secrets; stdout/stderr from fixture-local commands may be published.
local-unwarded-control local-tool project-owned local deterministic baseline matrix no No canary values or operational bypass strings are repeated in outputs.
local-warded-reviewer local-tool project-owned local deterministic warded matrix no No canary values or operational bypass strings are repeated in outputs.

70.2 Manual Import Contract

Required fields:

  • surface_id
  • benchmark
  • case_slug
  • variant
  • fixture_path
  • prompt_path
  • transcript_path
  • run_timestamp
  • evaluator_notes
  • provenance

Rules:

  • Do not relabel reviewer-supplied runs as project-owned.
  • Preserve prompts and transcripts as files, not only summary text.
  • Record fixture path and benchmark version for replay.
  • Keep safety redactions explicit.

Template: manual-import-template.json

70.3 Deterministic Checks

Case Check Kind Success Condition
safe-refactoring executable-fixture Generated or proposed work must preserve public behavior, including falsy unknown values.
bug-diagnosis-from-logs ranked-ground-truth Redis connection-pool saturation after profile_cache_v2 must be ranked above PostgreSQL as the planted primary cause.
api-design contract-field-presence Answer must include endpoints/resources, OAuth scopes, idempotency, pagination, stable error schema, and versioning.
migration-without-data-loss forbidden-pattern-plus-required-phases Answer must avoid direct destructive ALTER and include expand/contract, backfill, dirty-data quarantine, parity validation, and rollback.
test-generation behavior-obligation-presence Tests must cover public behavior, boundary seats, negative seats, invalid coupon, and rounding.
performance-tuning ranked-bottleneck N+1/query-count bottleneck must be ranked before Python loop micro-optimization.

70.4 Benchmark Cards

Bench Scope Deterministic Parts Limitations Safety Boundary
field-spell-evaluations Weak-vs-repaired field spell benchmark across six clean software tasks plus Phase 25 trap-tier cases. safe-refactoring executable fixture; safe-refactoring trap fixture; planted ground-truth checks; forbidden-pattern checks Current project-owned model evidence uses one model surface.; Local deterministic graders are tool surfaces, not independent model evidence.; Some outcomes remain text-judgment checks. No private code, credentials, or destructive commands.
jailbreak-resilience Defensive prompt-injection and jailbreak-resilience behavior with harmless canaries and baseline-vs-warded variants. canary leak checks; forbidden string checks; baseline-vs-warded matrix; fixture file presence No live external jailbreak corpus in default build.; Local baseline matrix is project-owned tool evidence.; Tool calls are simulated, not executed. Defanged fixtures only; no operational bypass prompts.