flowchart LR
H["Question, hypothesis ancestry, priors, and alternatives"] --> P["Exploratory or preregistered confirmatory protocol"]
P --> I["Authorized instrument or simulator with calibration"]
I --> X["Execute every attempt; preserve intervention and exclusions"]
X --> A["Blinded analysis, robustness checks, and independent reanalysis"]
A --> R{"Replication and claim ceiling support the conclusion?"}
R -- "no" --> N["Null, narrow, correct, rerun, or retain disagreement"]
R -- "yes" --> C["Bounded scientific claim packet"]
C --> U["Dual-use review, disclosure, and downstream evidence"]
U --> D["Correction, retraction, and replication follow-up"]
D -. "contradiction or failed replication" .-> H
N -. "new protocol" .-> P
81 Scientific Discovery and Experimental Governance
81.1 Chapter status
| Field | Value |
|---|---|
| Chapter ID | scientific-discovery-and-experimental-governance |
| Part | Part IV - Evidence, Implementation, and the Living Book |
| Status | conceptual |
| Manuscript maturity | v0.4 concept-complete argument-level manuscript |
| Last updated | 2026-08-13 |
| Claim label | Design rationale |
| Evidence level | argument |
| Source loading state | source notes: aletheia, ext_autonomous_lab_materials_2023, ext_coscientist_chemistry_2023, ext_ai_scientist_end_to_end_2026, ext_ai_co_scientist_2025; raw cache: aletheia |
| Test state | The chapter defines a minimum implementation and falsification plan; no chapter-core promotion follows from prose or source synthesis. |
81.2 Drafting guardrail
This chapter owns the evidence-producing scientific loop from hypothesis ancestry through correction and dual-use disposition. It does not infer discovery, causality, reproducibility, or safety from a generated idea, successful synthesis, simulator output, significant result, autonomous workflow, or polished paper.
81.3 Human Reading Path
Concrete lens. The simpler baseline publishes the positive attempt and calls repeated execution replication. The chapter preserves null and inconclusive attempts and asks whether the confirmatory and replication paths were actually independent.
Science is not one act of generation followed by one act of verification. Question choice, design, instruments, stopping rules, exclusions, analysis, and publication all shape what appears known. Automation can accelerate this loop, but it can also repeat bias, conceal failed attempts, and hide discretionary choices behind a seamless pipeline. The entire evidence-producing lifecycle therefore needs governance.
Exploration and confirmation must remain distinct. An experimental contract records hypothesis ancestry, prior evidence, competing explanations, preregistration, design and power, samples, controls, instruments or simulators, calibration, blinding, stopping, exclusions, analysis, human intervention, replication, and dual-use boundaries. Generated ideas may expand search, but protected data and outcome-dependent choices cannot rewrite the test that evaluates them.
The lifecycle runs from question selection through design, authorization, execution, measurement, analysis, independent reanalysis, replication, claim drafting, disclosure, and correction. HARKing, p-hacking, selective stopping, simulator laundering, drift, contaminated controls, publication bias, irreproducibility, and dangerous affordances remain visible. Automation may accelerate inquiry only while complete denominators and independent checks determine what may be claimed, because no smooth workflow substitutes for causal identification, reproducibility, or responsible control of dangerous knowledge.
81.4 Problem
Automation magnifies both search and researcher degrees of freedom. A system can generate thousands of hypotheses, choose among instruments, repair failed protocols, transform data, and draft interpretations faster than a human team can inspect each choice. Without durable role and denominator records, this throughput converts ordinary selective reporting and confirmation bias into a high-speed epistemic failure.
AI-assisted science closes a loop from research objective and hypothesis through design, simulation or instrument control, measurement, analysis, causal or statistical claim, replication, dual-use review, and evidence handoff. Generic planning and tool use do not preserve the epistemic controls of that full loop.
Without one experimental owner, planning, instruments, analysis, evidence states, and publication can each be locally valid while outcome-dependent choices and failed attempts disappear between them. The shared lifecycle method supplies custody; experimental governance preserves exploration/confirmation, calibration, denominators, replication, and claim ceilings.
81.5 Why existing approaches are insufficient
Reproducible code is necessary but can faithfully reproduce a biased design. Preregistration can be too vague to constrain analysis. Blinding can fail through metadata or adaptive instrument behavior. Replication can repeat the same calibration defect. Experimental governance must therefore preserve why each choice was made and test controls that would reveal a functioning but misleading pipeline.
A fluent hypothesis, autonomous laboratory, successful synthesis, simulator result, significant p-value, or generated paper can be useful while still hiding HARKing, selective stopping, instrument drift, contaminated controls, analysis flexibility, failed experiments, human intervention, replication gaps, or dual-use risk.
Planning, Runtime Adapters, Embodied Agency, Benchmark Ratchets, Evidence States, and Artifact Graphs form the strongest alternative composition. It wins if their join preserves hypothesis ancestry, preregistration, power, controls, calibration, blinding, stopping, exclusions, all attempts, independent analysis, replication, dual use, and correction.
What this experimental-governance diagram shows: a preregistration- and instrument-bound claim packet with its complete attempt history moves. Experimental completion and significance never cross as causal truth by themselves.
81.6 Core Claim
Reader claim. A positive experiment is not a discovery when the null and inconclusive attempts disappeared, the confirmatory branch saw the outcome, or the replication shares the same hidden dependency.
Operational rule. Freeze hypothesis ancestry, exploratory or confirmatory status, outcomes, sampling, stopping, exclusions, analysis, instrument scope, and claim ceiling before protected results open. Preserve every attempt and require independently qualified replication; omitted attempts or outcome exposed confirmation block the claim.
[scientific-discovery-and-experimental-governance.core, label: Design rationale, support: argument] An AI-generated scientific claim should enter the evidence stack only through a preregistered experimental contract that binds hypothesis lineage, exploratory versus confirmatory status, design and power, instrument or simulator authority, calibration, sample and protocol lineage, blinding and holdouts, stopping and exclusions, analysis, complete attempts, independent replication, dual-use disposition, and claim ceiling; experimental completion, significance, synthesis, instrument output, or formal workflow validity alone establishes neither causal truth, general scientific discovery, reproducibility, safety, nor transfer.
81.7 Mechanism
81.7.1 Worked attempt ledger: positive, null, and inconclusive stay together
The authored experiment dossier records three attempts under one hypothesis: attempt 21 is positive, attempt 22 is null, and attempt 23 is inconclusive. All three remain in the denominator. The confirmatory branch is marked confirmatory and preregistered before outcome, while its protected outcome remains unopened. Publishing only attempt 21 would turn search into a retrospective success story; opening the outcome before freezing the branch would turn confirmation back into exploration.
The eight-transition review rejects both failures, along with 52 other single- axis mutations. It preserves attempt identity across arbitrary finite lists, keeps null results inside benchmark handoff, invalidates receipts after seven scope changes, and distinguishes replication count from replication independence. The record does not establish that any hypothesis is true or any instrument accurate. It demonstrates the custody condition that lets a later reader tell the difference between a result that survived a declared test and one selected by the pipeline that produced it.
81.7.2 Three automation regimes, three different evidentiary risks
Current systems automate different slices of science, and their strongest result has to be interpreted at the layer where it occurred. A physical laboratory agent can turn a plan into an intervention, as the bounded chemistry tasks reported for Coscientist and the materials loop reported for A-Lab illustrate. That makes instrument identity, calibration, sample custody, operator intervention, and stop authority load-bearing. A successfully issued robot command or completed synthesis is still an event in the laboratory. It does not adjudicate what was measured or whether the scientific claim follows.
An end-to-end computational research agent faces a different boundary. The Nature report on the AI Scientist connects ideas, literature search, code, experiments, figures, manuscripts, and automated reviews. One of three generated workshop submissions received a mean score of 6.33 at a workshop with a reported 70 percent acceptance rate, while the authors judged none of the papers ready for the main ICLR conference and documented weak ideas, implementation errors, insufficient rigor, duplicated figures, and inaccurate or hallucinated citations. The venue outcome is informative about one institutional review path. It is not a measurement of the paper’s underlying claims, much less a replacement for independent analysis or replication.
A hypothesis co-scientist occupies a third regime. The reported Google system uses specialized generation, reflection, ranking, evolution, proximity, and meta-review agents with an Elo-style comparison process and additional inference compute. This can be a useful proposal and prioritization engine. Internal rank, expert interest, and experimental confirmation are nevertheless three different evidence objects. A system does not manufacture independence by asking several related agents to review one another.
A search-tree denominator accompanies all three regimes. Every candidate receives an identity, parent, creation path, compute allocation, tests, repairs, pruning reason, and disposition. The record includes abandoned ideas, failed code, invalid experiments, rejected manuscripts, and human interventions as well as the selected branch. Selection policy is frozen or versioned, and outcome access is recorded. Otherwise a large search can present its winner as though it were the only hypothesis ever considered.
Review also needs an evaluator-dependence graph. Five reviews are not five independent roots when they share a model family, training corpus, retrieval system, prompts, code, literature database, or scoring heuristic. The graph does not disqualify dependent review; it states what kind of evidence it is. Model self-review can provide debugging and triage. A claim that relies on independence needs a separately qualified method, data path, or replication whose relevant dependencies are actually different.
These evidence objects remain separate: a paper is an artifact, acceptance is an institutional observation, and a scientific conclusion is a claim compiled from design, observations, analysis, alternatives, and replication. Automation may strengthen the first two while leaving the third weak. The experimental packet must make that distinction inspectable rather than asking polished prose to carry epistemic authority.
The scientific packet is a dependency graph from research question to bounded claim. Hypotheses retain ancestry to observations, theories, prior searches, and model-generated suggestions. Confirmatory branches freeze outcomes, sampling, power or precision targets, exclusions, stopping rules, and analysis families before protected results are exposed. Exploratory branches may adapt, but every adaptation remains labeled and cannot be backdated into confirmation.
Instrument authority is leased separately from epistemic interpretation. The lease names the exact simulator, robot, assay, sensor, software, calibration, operating range, samples, safety interlocks, operator, maintenance state, and permitted effects. Instrument events and human interventions are appended to the attempt record. A successful API call or completed synthesis is only an observation until controls and analysis establish what it means.
Analysis runs in a declared environment with immutable data snapshots and complete code and parameter lineage. Primary and sensitivity analyses are distinguished; alternative explanations receive explicit tests; an independent role can replay the pipeline without inheriting the original interpretation. The claim compiler exposes the narrowest conclusion supported by design, calibration, statistics, causal assumptions, and replication, while preserving disagreement instead of forcing one paper-ready narrative.
Within the Developmental Intelligence Loop, this chapter owns governed causal intervention and experimental observation when capability development depends on learning how the world changes under action. It can return qualified measurements, rejected hypotheses, null results, and causal residuals to world models, memory, and training; it cannot promote a learned candidate, define its authority, or convert exploratory novelty into a supported conclusion.
Contract. Record hypothesis ancestry, prior evidence, competing explanations, exploratory status, preregistration, design, power, outcomes, exclusions, stopping, and analysis before protected data are opened.
Admission. Lease simulators and instruments with calibration, operating envelope, sample identity, control state, maintenance, operator intervention, and safety authority.
Execution. Separate generated hypotheses, execution, measurement, analysis, causal interpretation, and claim drafting across inspectable roles and preserve all failed attempts.
Observation. Require positive, negative, null, contamination, drift, and analysis-robustness controls plus independent reanalysis and, where feasible, replication.
Closure. Route dangerous hypotheses, protocols, materials, capabilities, and disclosures through dual-use review and retain withheld details and unresolved risks as explicit residuals.
81.8 Concept-completion ledger
81.8.1 Hypothesis and experiment provenance
Mechanism. Give every hypothesis a stable identity and ancestry graph linking the motivating observation, prior work, model or human proposer, competing explanations, search procedure, and the exact protocol versions that tested it. Record whether a branch was exploratory or confirmatory when it was created, not after its result was known. Generated variants remain siblings with separate attempt records; a later edit creates a descendant rather than silently replacing the earlier question. This makes novelty, independence, and outcome-conditioned revision inspectable.
Failure mode. A pipeline can generate thousands of candidates, test only convenient ones, repair protocols after partial results, and publish the winner as though it had been specified in advance. Provenance can also become decorative if aliases hide duplicate hypotheses or if the same model both proposes and “independently” verifies a result.
Non-claim. Complete ancestry does not make a hypothesis important, original, causal, or true. It establishes custody of the search and test history only.
Source grounding. aletheia motivates claim-native dossiers, explicit unknowns, and verification lineage. ext_autonomous_lab_materials_2023 supplies a bounded corrected case in which target identity, training overlap, attempted syntheses, and later reinterpretation materially affect what may be said. ext_ai_scientist_end_to_end_2026 and ext_ai_co_scientist_2025 make candidate population, tree search, ranking, pruning, compute, and selection dependencies concrete. None establishes that this provenance design improves discovery in a local campaign.
81.8.3 Discovery versus benchmark success
Mechanism. Compile a discovery claim from several separately scored objects: target definition, prior-art boundary, prediction, experimental realization, identification confidence, causal explanation, novelty to a platform, novelty to a field, and independent reproduction. Benchmark metrics such as yield, hit rate, or time-to-result remain operational observations. Promotion requires the claim compiler to select the narrowest supported discovery predicate and expose every denominator, including screened targets, attempted protocols, inconclusive identifications, and previously known items.
Failure mode. A system can optimize an easy proxy—recovering database entries, matching a platform’s “new” set, or producing classifier-positive samples—while being credited with new science. Test contamination, flexible target definitions, and undisclosed human curation can make a discovery benchmark look stronger than the underlying epistemic result.
Non-claim. High benchmark performance, a successful synthesis, or a novel model output does not by itself establish scientific novelty, explanation, causality, or general discovery competence.
Source grounding. The corrected ext_autonomous_lab_materials_2023 record distinguishes 57 targets, 36 confirmed realizations, four later-inconclusive identifications, and novelty relative to a prediction platform rather than necessarily to science. ext_ai_scientist_end_to_end_2026 distinguishes manuscript production and workshop appraisal from conference-level scientific quality, while ext_ai_co_scientist_2025 distinguishes internal ranking, expert preference, and selected laboratory appraisal. aletheia supports bounded claim compilation. These sources motivate the distinction but do not validate a general discovery metric.
81.8.4 Reproducibility and replication
Mechanism. Treat computational replay, same-laboratory repetition, method replication, and independent conceptual replication as different evidence routes. A reproducibility packet contains raw observations, protocol and code versions, instrument calibration, environments, exclusions, randomization, stopping, analysis families, and operator interventions. A replication contract prospectively states which elements must be held equivalent, which differences create independence, what result range counts as consistent, and how disagreement changes the claim rather than forcing consensus.
Failure mode. Byte-identical replay can reproduce a shared bias; a nominally independent laboratory can reuse the same contaminated data, model, reagent, or analysis code; and a failed replication can be dismissed post hoc by changing the target construct. The opposite error treats any difference as refutation despite insufficient power or failed positive controls.
Non-claim. One successful replay or replication does not establish broad reproducibility, transfer across domains, or correctness of the original causal interpretation.
Source grounding. aletheia motivates replayable claim dossiers and retained contradiction. The correction history in ext_autonomous_lab_materials_2023 demonstrates why later reanalysis must remain connected to the original denominator. ext_ai_scientist_end_to_end_2026 and ext_ai_co_scientist_2025 expose another independence problem: generator, reviewer, ranker, and meta-reviewer may share model and tool dependencies even when they occupy different named roles. None supplies an independent replication of the chapter’s proposed mechanism, so the support state remains argument.
81.8.5 Negative and null results
Mechanism. Preserve every authorized attempt, including null, negative, contaminated, interrupted, and inconclusive outcomes, under a frozen denominator. The record distinguishes absence of the target effect from instrument incapacity, insufficient power, protocol deviation, failed manipulation, and safety termination. Known-null and known-effect controls calibrate the pipeline; publication and memory interfaces must make negative evidence retrievable without letting a high-volume generator bury it beneath successful-looking variants.
Failure mode. Selective stopping, outcome-dependent exclusions, silent retries, and “cleanup” of inconclusive identifications turn search volume into apparent certainty. A weak instrument can also yield a false negative that is mistakenly used to reject an idea, while an overpowered analysis can label negligible effects important.
Non-claim. A null result is not proof of no effect outside the tested sensitivity, population, protocol, and time window. A failed implementation is not evidence against the underlying architecture unless competence and positive controls passed.
Source grounding. ext_autonomous_lab_materials_2023 provides a concrete corrected denominator with failed and inconclusive targets that must not disappear. aletheia motivates preserving aporea and rejected claims. The sources do not establish publication-bias mitigation or local null-result handling.
81.8.6 Dual-use review and information hazards
Mechanism. Run dual-use review at question selection, protocol authorization, execution, artifact custody, claim drafting, and disclosure. The review packet separates scientific validity from who may perform an experiment or receive operational detail. It records hazard class, plausible misuse pathway, required expertise and resources, affected populations, safer abstraction, access tier, monitoring, expiry, appeal, and residual knowledge that cannot be recalled. Withholding an operational recipe does not erase the existence or evidential status of the result.
Failure mode. Review performed only at publication misses dangerous tool calls, materials, code, or intermediate artifacts created earlier. A vague safety label can suppress benign work without a falsifiable threat model, while a truth-seeking mandate can be abused to publish reusable harmful instructions.
Non-claim. Dual-use review does not prove safety, eliminate misuse, or authorize secret adjudicators to alter scientific conclusions. It governs execution and disclosure within a named threat model.
Source grounding. aletheia contributes constitutional export boundaries, source-bounded review, and separation of verification from action authority. ext_autonomous_lab_materials_2023 is a benign materials case and supplies no general dual-use safeguard evidence; it shows only why integrated laboratory capability makes lifecycle review relevant.
81.8.7 Resource allocation and search governance
Mechanism. Allocate experimental resources through a portfolio record that names expected information gain, decision relevance, cost, opportunity cost, safety burden, neglected alternatives, replication reserve, and stopping conditions. Search systems may propose large candidate sets, but an independent budget owner preserves diversity among hypotheses and funds calibration, negative controls, and replication rather than rewarding only short-term positive yield. Adaptive allocation remains exploratory unless its update rule and outcome access were prospectively authorized.
Failure mode. A generator can monopolize compute or laboratory time with easily scored hypotheses, starve disconfirming work, and optimize publication count rather than knowledge. Cost-blind governance can make every claim prohibitively slow; yield-only governance can conceal failed attempts and discourage replication. Resource asymmetry can also make the proposed method appear superior to weaker baselines.
Non-claim. A rational-looking budget or high information-gain estimate does not establish that the chosen scientific questions are socially valuable, morally legitimate, or globally optimal.
Source grounding. aletheia motivates routing by uncertainty, risk, ambiguity, and historical failure while warning about verification latency. ext_autonomous_lab_materials_2023 demonstrates active selection inside one bounded synthesis workflow. ext_ai_scientist_end_to_end_2026 and ext_ai_co_scientist_2025 add computational tree search, hypothesis ranking, and test-time compute; the chapter therefore tracks branch-level resources and selection policy. None validates the proposed portfolio rule or its societal priorities.
81.8.8 Human scientific judgment and accountable handoff
Mechanism. Preserve named human roles for question legitimacy, protocol authorization, interpretation challenge, dual-use disposition, and correction without treating a signature as epistemic proof. Reviewers receive the full attempt and intervention record, can inspect model-generated alternatives, record disagreement, request independent reanalysis, and narrow or reject the claim. Automation must report when it lacks a competent instrument, when protected outcomes leaked, and when a decision exceeded the reviewer’s available time or expertise.
Failure mode. Human-in-the-loop theater asks a reviewer to approve a polished conclusion after the search history has been compressed away. Automation bias, authority gradients, alert overload, and correlated model-written analyses can make nominal independence meaningless. Conversely, requiring manual approval for every benign step can destroy the throughput being evaluated and shift work into unlogged channels.
Non-claim. Human approval does not make an experiment correct, safe, representative, or legitimate. The claim is that accountable roles and inspectable dissent preserve a correction path.
Source grounding. aletheia supplies adversarial review, explicit unknowns, and separate evidence authority. The corrected ext_autonomous_lab_materials_2023 history illustrates the value of later human reanalysis. ext_coscientist_chemistry_2023 preserves a bounded human-supervised laboratory context; ext_ai_scientist_end_to_end_2026 documents gaps that survived an automated paper-and-review pipeline; and ext_ai_co_scientist_2025 reports expert appraisal while naming factuality and evaluation limitations. None proves that this governance arrangement prevents error or scales to autonomous science.
81.9 Interfaces
The handoffs are deliberately one-way in authority. Planning may propose but not preregister retroactively; an instrument owner may certify execution but not causal truth; analysis may estimate but not authorize a dangerous action; Evidence States may adjudicate only the packet it receives. Dual-use review can narrow execution or disclosure without rewriting the scientific outcome.
Planning proposes experiments; Runtime Adapters and Embodied Agency own tool and instrument effects; Artifact Graphs preserve records; Evidence States adjudicates claims. This chapter owns the scientific design and inference bridge among them, including replication and dual-use limits.
- Planning proposes experiments; this chapter governs confirmatory design and inference.
- Runtime Adapters and Embodied Agency control tools and instruments; they do not certify scientific truth.
- Benchmark Ratchets and Evidence States receive complete results and claim ceilings.
- Artifact Graphs and the Living Book preserve protocol, data, analysis, replication, correction, and publication lineage.
Failed attempts remain included.
81.10 Invariants
Protected outcomes remain inaccessible to hypothesis selection and pipeline tuning after the confirmatory branch freezes. If leakage occurs, the branch is reclassified or replaced with a fresh holdout. Replication preserves both method equivalence and relevant difference; a byte-identical rerun by the same pipeline is replay evidence, not independent replication.
The shared lifecycle method requires exploration and confirmation to remain distinct, instrument output to remain an observation, all attempts and interventions to remain counted, and claim scope to stay inside the design, calibration, analysis, and replication evidence.
- Hypothesis generation and confirmatory testing remain separated.
- Simulator or instrument return is an observation, not scientific truth.
- Exploratory and confirmatory analyses remain visibly distinct.
- All attempts, exclusions, stopping events, and human interventions remain in the denominator.
- One laboratory, domain, or replication does not imply broad transfer.
81.11 Failure modes
Automation-specific failures include agents silently repairing protocols after seeing partial outcomes, choosing among many plausible analyses, discarding “uninteresting” nulls, inferring calibration from expected results, or using a simulator as both hypothesis generator and truth oracle. Positive and null controls must pass under the same autonomy and tooling available to the target study.
Branch-selection laundering hides the generated population and presents one survivor as a prespecified insight. Evaluator monoculture counts correlated model reviews as independent judgment. Paper-production substitution treats a complete manuscript or favorable venue decision as confirmation of its claims. Review-system flooding exploits cheap generation, consumes scarce human attention, lowers review depth, and gives polished but defective artifacts more opportunities to survive. Each failure is testable only when the system preserves the search tree, evaluator dependencies, submission denominator, and human review burden.
The principal failure family includes HARKing; p-hacking; selective stopping; publication bias; instrument drift; simulator laundering; contaminated controls; automation bias; hidden intervention; irreproducibility; unsafe experiment generation; dual-use leakage; claim drafting beyond the data.
Evaluation must inject known nulls, known effects, drift, contamination, selective stopping, analysis flexibility, and hidden intervention. The control plane fails if it cannot detect those cases; an incapable instrument or simulator narrows the run rather than the scientific-governance idea.
81.12 Minimum Viable Implementation
The first artifact can be a local experiment ledger and execution wrapper. It issues immutable protocol and instrument leases, assigns attempt identifiers, seals the confirmatory holdout, captures commands and interventions, and emits an analysis bundle with all planned and exploratory outputs. A second implementation replays the bundle from the raw observations and records any disagreement before claim drafting.
Use a benign reproducible simulation or low-risk open instrument with injected null and known effects. Freeze stopping and analysis, preserve every attempt, blind the confirmatory holdout, run independent analysis, and package protocol, data, code, calibration, exclusions, and residuals for replication. This tests the control plane, not autonomous science in general.
The minimum campaign uses a benign reproducible simulation or low-risk open instrument with injected null and known effects. It freezes stopping and analysis, preserves all attempts, blinds a confirmatory holdout, runs independent analysis, and packages protocol, data, code, calibration, exclusions, and residuals for replication.
81.13 Evidence and falsification program
Argument exit requires multiple preregistered studies or replications in which hypothesis generation, instrument authority, execution, analysis, and adjudication are separable; positive and null controls work; all attempts are visible; independent reanalysis agrees or preserves disagreement; and the claim ceiling predicts what transfers and what does not.
81.14 Mature Research Target
An advanced scientific layer lets AI systems search large hypothesis and design spaces while the governance layer keeps the scientific process more inspectable than an ordinary laboratory notebook. Hypothesis families, negative results, calibration, adaptive choices, sample custody, analysis alternatives, and replication attempts form a queryable graph. Researchers can see where search efficiency increased and where evidence quality or safety constrained it.
The research program compares human-led, AI-assisted, and increasingly autonomous workflows on benign domains with known nulls, known effects, simulator mismatch, instrument drift, contamination, and adversarially tempting analysis choices. Outcomes include discovery yield, false discovery, calibration, time and resource cost, attempt completeness, analysis agreement, replication, claim-scope accuracy, and dual-use control. Novelty alone is never the objective.
The mature layer also supports responsible nonpublication. Dangerous methods or artifacts can be withheld, abstracted, or shared through protected channels while the existence and rationale of the disclosure decision remain auditable. This does not grant a hidden authority to declare truth; it keeps scientific adjudication separate from who may execute or distribute a risky capability.
The mature layer lets AI expand hypothesis and experiment search while making discretionary choices, instruments, failed attempts, analysis, replication, correction, and dangerous affordances more visible than in the baseline scientific workflow. Speed counts only when evidence quality and responsible control survive.
No such campaign has passed. Scientific-automation support should stay at argument until known-null, known-effect, drift, contamination, selective-analysis, replication, and dual-use controls survive independently replayed workflows.
81.15 Codex test plan
| Test | Purpose | Status |
|---|---|---|
| Exploration/confirmation split | Reject outcome-informed hypotheses or analysis choices presented as preregistered confirmation. | planned |
| Instrument authority and drift | Inject calibration failure, simulator mismatch, contamination, and operator intervention. | planned |
| Complete attempt denominator | Preserve failed runs, exclusions, stopping events, and analysis variants. | planned |
| Replication and claim ceiling | Require independent reanalysis or replication and block transfer beyond the tested protocol and domain. | planned |
| Search-tree and evaluator custody | Reject omitted branches, hidden pruning or intervention, correlated reviewers presented as independent, and venue outcomes presented as truth. | planned |
81.16 Formalization hooks
Implemented formalization: lean:scientific-discovery-and-experimental-governance.admission_boundary binds this chapter to AsiStackProofs.ScientificExperimentReview. Its 41 theorem declarations define a reachable eight-transition experiment review and test 54 admission-axis mutations. A complete authored dossier reaches only eligibility for a Project Theseus governed experiment campaign; every one-axis omission or forbidden broad claim reaches repair with an exact disposition.
The mechanized core proves experimental-record custody rather than scientific truth. Attempt-identity collection composes over arbitrary finite lists and retains every member. A complete denominator must count each attempt, while an omitted attempt rejects completeness; an outcome-exposed branch cannot satisfy confirmatory integrity. Expiry, an omitted-attempt gap, and an independent-replication gap remain rejecting under adverse monotone changes. Hypothesis, protocol, instrument, data, analysis, environment, and claim-ceiling changes invalidate receipts.
Two information-loss constructions show why equal significance signals cannot determine whether preregistration remained intact and why equal successful-replication counts cannot determine whether replication was independent. The consumer bridges block Evidence States empirical promotion without independent replication and reject Benchmark Ratchet promotion when null results are missing. These are theorems about encoded records and transitions; the model does not prove hypothesis truth, causal identification, instrument accuracy, reproducibility, scientific discovery, laboratory safety, deployment readiness, support, release, or external effect. Chapter support remains argument and support_state_effect remains none. Those claims require the Project Theseus governed experiment campaign across preregistered synthetic and benign natural protocols, instruments or simulators, known-null and known-effect controls, drift, contamination, analysis variation, independent replication, correction, and dual-use review.
81.17 Source crosswalk
| Source ID | Title | Bounded use |
|---|---|---|
aletheia |
Aletheia Foundry | Corben-authored safe-intelligence lineage for truth-seeking, verification, and disciplined knowledge production. It motivates an epistemic foundry interface, but it does not supply preregistered experiments, instrument calibration, causal identification, independent replication, laboratory safety, or a scientific-discovery result. |
ext_autonomous_lab_materials_2023 |
An autonomous laboratory for the accelerated synthesis of inorganic materials | Preliminary autonomous-laboratory comparator based on the corrected official Nature article abstract, selected article-page passages, and the 2026 author correction: A-Lab integrates computation, literature-derived data, machine learning, active learning, and robotics, with the corrected article reporting 36 realized compounds from 57 targets. The correction narrows the novelty wording and excludes four inconclusive identifications; no laboratory run, material synthesis, replication, or general experimental-control-plane result has been reproduced locally. |
81.17.1 Manifest source assignment reconciliation
These rows keep Scientific Discovery and Experimental Governance’s manifest assignments visible at their recorded review boundary. Passage review does not establish local reproduction, performance, safety, deployment, or support-state movement.
| Source | Intake role | Boundary |
|---|---|---|
ext_ai_scientist_end_to_end_2026 |
Passage-reviewed comparator: Towards end-to-end automation of AI research. Passage-reviewed computational-research comparator: the reported system connects ideation, literature search, code, experiments, analysis, manuscript production, and automated review. The paper reports one of three generated workshop submissions receiving a 6.33 mean score at a workshop with a 70 percent acceptance rate, while the authors judged none conference-ready and documented implementation, rigor, figure, and citation failures. | Source-reported machine-learning workflow only; paper completion, automated review, and venue scoring are not scientific truth. No system run, manuscript, hidden search tree, review, reproduction, transfer, or support result is local. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row. |
ext_coscientist_chemistry_2023 |
Passage-reviewed comparator: Autonomous chemical research with large language models. Passage-reviewed physical-laboratory comparator: Coscientist connects a language-model planner to web and document search, code execution, and robotic laboratory APIs across six reported chemistry task families, making proposal, tool, intervention, measurement, and adjudication boundaries concrete. | Source-reported bounded chemistry demonstrations with human oversight and task-specific assessment; no local chemistry run, independent scientist, general laboratory safety, causal discovery, transfer, or support promotion. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row. |
ext_ai_co_scientist_2025 |
Passage-reviewed comparator: Towards an AI co-scientist. Passage-bounded hypothesis-generation comparator: the proposed multi-agent system generates, reflects on, ranks, evolves, and meta-reviews hypotheses with an Elo-style process and additional inference compute, followed by source-reported expert and selected laboratory appraisal. | Preprint and provider-reported evaluation only; internal rank, expert preference, and laboratory confirmation remain separate evidence objects. No full search population, evaluator independence, experiment, reproduction, transfer, or support result is local. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row. |
81.18 Summary
Scientific discovery is a governed evidence-production process from question and hypothesis ancestry through protocol, instrument authority, execution, measurement, analysis, causal interpretation, replication, disclosure, and correction. AI can accelerate every stage, so it must also make every discretionary choice and failed attempt easier to inspect.
The governing separation is between exploration, confirmation, observation, inference, replication, and effect authority. Known-null and known-effect controls, calibration challenges, sealed outcomes, independent reanalysis, complete denominators, and explicit claim ceilings let search expand without allowing a successful experiment or polished manuscript to become self-authenticating science.
AI-assisted science needs governance over the whole evidence-producing loop, not merely generated hypotheses or verified code. Experimental custody preserves ancestry, design, instruments, calibration, attempts, blinding, stopping, analysis, replication, dual-use decisions, correction, and claim ceilings so automation increases search without silently weakening science.
81.19 Handoff
Artifact Steward Agents and Living Project Governance receives the bounded claim packet, protocol, data, code, calibration, attempt denominator, replication status, disclosure decision, and open residuals. Stewardship can maintain and coordinate those artifacts but cannot promote the scientific claim, reinterpret a failed control, or authorize dangerous follow-on work.