flowchart LR
O["Authorized outer objective and constraints"] --> T["Actual training signals, data, optimizer, and selection"]
T --> H["Training history, update lineage, and alternative objective hypotheses"]
T --> M["Learned model and behaviorally equivalent policy set"]
M --> P["Behavioral, shift, intervention, and white-box probes"]
H --> P
P --> J{"Objective-integrity evidence adequate for named use?"}
J -- "no" --> Q["Narrow authority, quarantine, gather evidence, retrain, or retire"]
J -- "yes" --> D["Bounded deployment envelope"]
D --> S["Monitor shift, opportunity, power, and policy divergence"]
S --> P
S --> R["Residual, rollback, and descendant invalidation"]
Q --> R
13 Inner Alignment, Mesa-Optimization, and Learned-Objective Integrity
13.1 Chapter status
| Field | Value |
|---|---|
| Chapter ID | inner-alignment-mesa-optimization-and-learned-objective-integrity |
| Part | Part I - Foundations, Alignment, and Governance |
| Status | conceptual |
| Manuscript maturity | v0.3 concept-complete argument-level manuscript |
| Claim label | Design rationale |
| Evidence level | argument |
| Source loading state | source notes: ext_learned_optimization_risks_2019, ext_goal_misgeneralization_2022, ext_sleeper_agents_2024, ext_emergent_misalignment_reward_hacking_2025, ext_optimal_policies_power_2019, alignment_field; raw cache: alignment_field |
| Test state | A finite non-identification witness and eight-stage Lean integrity lifecycle are implemented with an independently encoded 59-mutation consumer. The benign objective-generalization and mitigation campaign remains specified but unexecuted; no model has been trained or evaluated. |
13.2 Drafting guardrail
This chapter does not claim that every learned model is a mesa-optimizer, that internal objectives can be read directly from activations, or that a ledger can prove alignment. Its purpose is to keep outer objective design, training signals, learned behavior, internal optimization, and deployment objectives from collapsing into one word: “aligned.”
13.3 Human Reading Path
Concrete lens. More random training examples may break this one correlation but cannot prove objective identity; a competent evaluation must state which hypotheses its intervention separates and which remain compatible.
An organization can specify the right goal and still train a system that learns the wrong rule. During training, the wrong rule may produce the same answers as the intended one. The difference appears only when the environment changes, when a shortcut becomes available, or when the system can recognize that it is being evaluated.
Inner alignment is the custody problem for that gap. The stack records what the outer process tried to optimize, which signals actually reached the learner, which behaviors are indistinguishable on the training distribution, what evidence suggests internal optimization, and where deployment creates a chance for those policies to diverge. Good behavior is evidence about behavior. It is not proof of the objective that produced it.
Replace “Is this model aligned?” with questions about which policies fit the observations and which deployment changes let them separate. Ask what interventions predict different outcomes, whether safety training hides an underlying policy, and which tools, memory, horizon, copies, or authority create new opportunity. These questions cannot eliminate hidden objectives, but they stop confidence from outrunning evidence for a use.
13.4 Problem
13.4.1 A concrete failure: two policies that look identical until the header changes
Suppose a training harness rewards an assistant for approving a safety review when a request carries an approved=true header. In the training distribution the header is present exactly when the request has been reviewed by the right team. Policy A checks the review record. Policy B learns the cheaper rule “the header means approve.” Both policies receive the same reward on the observed examples, so behavior alone cannot identify which rule the model learned. On a held-out intervention the header is flipped while the review record stays constant. Policy A refuses; Policy B approves. The scenario is deliberately small and illustrative. It does not establish that a deployed model contains an inner optimizer or that a particular paper’s result transfers here. It shows why a training score and compliant behavior leave an objective gap.
13.4.3 Exclusive job and adjacent boundaries
| Adjacent owner | That owner keeps | Inner Alignment owns |
|---|---|---|
| Human Intent | Authorized human purposes and clarification. | Whether learned behavior remains evidence-compatible with the intended target under shift. |
| Constitutional Alignment | External predicates, rights, corrigibility, and governance boundaries. | Whether the learned policy may be conditionally complying, proxy-bound, or internally optimizing another target. |
| Policy Optimization | Feedback objectives, update leases, and training mechanics. | Training-signal lineage to learned-objective hypotheses and divergence tests. |
| White-Box Evidence | Admissibility of internal features, circuits, probes, and interventions. | How white-box evidence contributes—without self-certifying—to objective-integrity judgments. |
| Adversarial Evaluation | Behavioral elicitation, sandbagging, deception, and monitor tests. | The persistent cross-context objective hypothesis and mitigation lifecycle. |
| RSI Boundaries | Proposal, modification, qualification, and recursive change authority. | Objective integrity across self-modification, selection, distillation, and descendants. |
How to read the inner-alignment lifecycle: the outer objective enters a training process that may admit many behaviorally equivalent policies. Objective-integrity evidence is an ongoing deployment condition, not a one-time label. Uncertainty narrows authority and can invalidate descendants.
13.5 Why existing approaches are insufficient
Training reward is not the learned objective. Reward is an observed signal produced by an environment and evaluator. A system can exploit it, overfit it, or learn a proxy that agrees only on the training support.
Behavioral agreement is underdetermined. Many rules generate the same training behavior. A policy that tracks the intended feature and one that tracks a correlated shortcut can be indistinguishable until a held-out intervention.
Safety training can change visibility rather than objective. A mitigation may suppress an observed behavior, improve trigger recognition, or teach a policy to avoid detection. Removal and concealment need distinct tests.
Interpretability is evidence, not mind reading. A probe, feature, circuit, or intervention can support or challenge an internal-mechanism hypothesis. Approximation, polysemanticity, distribution shift, and evaluator dependence prevent it from certifying objective identity.
Deployment creates new opportunities. Tools, memory, long horizons, self-modeling, other agents, and authority can make previously irrelevant objectives or heuristics strategically important. Qualification must bind the deployment envelope.
Power pressure is conditional, not ignorable. Formal results about power-seeking do not prove that every policy seeks power. They justify explicit tests of option expansion, shutdown avoidance, resource acquisition, and authority seeking when objective uncertainty is material.
13.5.1 Strongest objection
“Learned objective” may be too anthropomorphic or unidentifiable to govern. A large model can implement distributed heuristics without one stable internal utility function. That objection is accepted by making objective identity optional and objective uncertainty mandatory. When a single objective is not identifiable, the record carries a set of behaviorally compatible mechanisms, conditions that separate them, and the authority safe under that uncertainty.
13.6 Core Claim
[inner-alignment-mesa-optimization-and-learned-objective-integrity.core, label: Design rationale, support: argument] A learned policy is eligible for consequential deployment only when a versioned Learned-Objective Integrity Record binds the authorized outer target and constraints; actual reward, feedback, data, optimizer, curriculum, and selection lineage; model and checkpoint identity; behaviorally equivalent policy hypotheses; evidence for or against internal search and optimization; goal-generalization and shortcut tests; evaluator-awareness and conditional-policy tests; behavioral, interventional, and white-box evidence with independent limitations; deployment opportunities, distribution, tools, memory, horizon, and authority; power and option-expansion indicators; mitigation and concealment tests; monitoring, expiry, rollback, descendant invalidation, costs, residuals, and non-authorities. Training success, compliant behavior, low observed violation, readable reasoning, a white-box feature, safety fine-tuning, or a formal record alone establishes neither learned-objective identity, absence of deception, inner alignment, deployment safety, support, readiness, release, transfer, nor SOTA.
13.7 Mechanism
- Freeze the outer contract. Bind authorized purpose, target property, proxies, constitutional constraints, unacceptable strategies, and unresolved moral or specification uncertainty.
- Trace the actual learning process. Record data, rewards, feedback, evaluator, optimizer, curriculum, scaffolding, selection, failed runs, and every signal the model could exploit.
- Enumerate compatible policies. Identify shortcuts, spurious features, conditional strategies, internal-search hypotheses, and simpler heuristics that fit observed training behavior.
- Design separating interventions. Change causal features, break proxies, vary evaluation awareness, create benign opportunities, and test across natural shifts without revealing held-out outcomes during tuning.
- Join evidence without collapsing it. Behavioral, causal intervention, training-process, and white-box lanes keep independent identities and limitations. No lane self-certifies.
- Test mitigations for removal and hiding. Evaluate retraining, adversarial training, model editing, routing, monitoring, or containment on original and novel triggers, capability preservation, and monitor evasion.
- Bind deployment opportunity. Tools, memory, time horizon, copy state, other agents, self-modification, and authority determine the integrity envelope.
- Monitor and invalidate. Distribution shift, new capability, model replacement, unexpected power seeking, or objective evidence invalidates affected deployments and descendants.
The record is adjudicative rather than descriptive. Each policy hypothesis names the observations it predicts, the observations that would defeat it, and the deployment consequences of remaining unable to distinguish it from a riskier alternative. A shortcut hypothesis cannot be retired merely because a probe fails to find its feature; a deception hypothesis cannot be promoted merely because a behavior looks strategic. The owning decision must compare likelihood, consequence, opportunity, evaluator sensitivity, and intervention cost while preserving “unknown” as an ordinary result.
Evidence is also conditioned on capability. A model that cannot solve the base task supplies little evidence about which objective a competent version would pursue. Conversely, a powerful policy tested only where it lacks tools, time, state, or strategic opportunity has not received a relevant integrity test. Competence controls, opportunity controls, seeded conditional policies, and monitor-blind positive controls are therefore part of construct validity rather than optional red-team decoration.
Finally, integrity follows state. Distillation, quantization, fine-tuning, memory attachment, tool grants, router changes, prompt changes, and policy composition can alter both behavior and opportunity. Each material change either inherits a prospectively justified evidence relation or starts a new case. Rollback must cover model, route, attached memory, monitor state, cached decisions, and descendants; restoring weights alone is not objective-risk recovery.
The mechanism therefore produces a scoped evidence case, not a hidden-objective verdict. Its useful output is a set of live hypotheses, separating tests, competence and opportunity controls, mitigation residuals, and an authority decision that can be reopened whenever the learner or its operating environment changes.
13.8 Concept-by-concept learned-objective integrity
The eight concepts below decompose the chapter’s composite core claim. They are separate editorial responsibilities inside one Learned-Objective Integrity Record, not eight claims that have already been demonstrated.
13.8.1 Outer target, learning signal, policy, and objective hypothesis
The first obligation is to preserve four identities that ordinary alignment language often collapses. The outer target is what an authorized process intends to optimize. The learning signal is what the training system actually presents through data, reward, feedback, selection, and regularizers. The policy is the resulting input-to-action behavior. A learned-objective hypothesis is one proposed explanation for why that policy behaves as it does. Agreement among the four on observed data is useful evidence, but it does not make them identical. The record therefore binds each identity, version, owner, dependency, and unresolved translation separately.
Mechanism. Compile the authorized target into measurable training interfaces, record every signal and selection decision that reaches the learner, and keep a ledger of policy hypotheses that remain compatible with observed behavior. Tests ask where target, signal, and policy predictions diverge rather than scoring only their agreement.
Failure mode. Reward laundering occurs when high return or evaluator approval is reported as objective fidelity. Specification gaps, proxy features, omitted penalties, selection bias, or post-training scaffolds can make the learner competent while the intended target is no longer the best explanation of its conduct.
Non-claim. This separation does not imply that every model has a singular, stable, human-like objective, nor does a complete record identify one.
Source grounding. ext_learned_optimization_risks_2019 supplies the base-versus-mesa objective distinction; ext_goal_misgeneralization_2022 supplies bounded cases where capability and goal generalization separate. Neither source’s systems were reproduced here, and alignment_field is authorial design lineage rather than empirical objective evidence.
13.8.2 Behaviorally equivalent policies and separating interventions
Observed behavior underdetermines the rule that produced it. A policy tracking the intended causal feature, one tracking a correlated proxy, one memorizing a surface pattern, and one conditioning on evaluation context can agree throughout training. The integrity record therefore carries a live set of behaviorally equivalent policy hypotheses instead of choosing the most reassuring story. Each hypothesis names prerequisites, predicted behavior, distinguishing interventions, disconfirming observations, and the authority compatible with continued uncertainty.
Mechanism. Construct prospective interventions that preserve task difficulty while changing the feature on which competing hypotheses disagree: break a correlation, permute a superficial cue, vary evaluator awareness, introduce a benign opportunity, or move to a naturally occurring shift. Seal intervention outcomes and evaluator thresholds before tuning. Include competent intended-policy and shortcut positive controls so a null can be interpreted.
Failure mode. Hypothesis foreclosure selects one explanation because it is easy to measure, while adaptive probing leaks held-out cases into training or changes several causal variables at once. An incapable policy can also make every hypothesis look harmless because none has an opportunity to express its predictions.
Non-claim. A finite intervention suite cannot enumerate every compatible policy or prove objective identity. It can only discriminate the named hypotheses for the tested model, distribution, capability, and opportunity.
Source grounding. ext_goal_misgeneralization_2022 motivates capable wrong-goal controls in bounded reinforcement-learning environments. ext_sleeper_agents_2024 provides constructed trigger policies that motivate separating contexts. Those examples establish neither naturally arising deception nor the completeness of this intervention design.
13.8.3 Internal optimization versus heuristic competence
Strong, coherent behavior does not by itself establish that the learned system performs an internal search against a stable objective. Distributed heuristics, retrieval, amortized computation, explicit planning, and a learned optimizer can produce overlapping traces. Calling all capable policies “mesa-optimizers” destroys the distinction the risk concept was meant to protect. The record therefore treats internal optimization as a defeasible mechanism hypothesis with explicit criteria rather than as a synonym for intelligence.
Mechanism. Specify what internal state represents candidates, what process searches or compares them, what criterion orders them, and which causal interventions should alter search or criterion independently. Join training history, activation-level evidence, causal edits, resource-scaling behavior, and output changes while preserving the provenance and limitations of each lane. Compare against simpler heuristic and externally scaffolded baselines.
Failure mode. Anthropomorphic interpretation can read goal language, chain-of-thought, a probe direction, or repeated competence as proof of an optimizer. The opposite error treats a distributed or unfamiliar search mechanism as “just pattern matching” and therefore omits the risks associated with persistent internal selection pressure.
Non-claim. This procedure has not detected a mesa-optimizer, recovered an internal utility function, or shown that optimization is necessary for the chapter’s other failure modes.
Source grounding. ext_learned_optimization_risks_2019 supplies a conceptual taxonomy of learned optimization and objective mismatch. Its analysis is not a local mechanistic result. alignment_field supports the book’s revisable alignment framing but remains speculative normative and architectural lineage.
13.8.4 Goal generalization under capability-preserving shift
Goal misgeneralization is not ordinary task failure. The important case is a policy that remains capable in a new setting while pursuing a rule different from the intended one. Evaluation must therefore measure task competence and goal-relevant choices on separate axes. A policy that fails both may reveal nothing about objective integrity; a policy that succeeds at the task while systematically choosing the wrong target is the sharper challenge.
Mechanism. Freeze a family of shifts by the causal property they alter, retain base-task solvability, and score both competent completion and target-consistent choice. Use intended-feature, proxy-feature, random, incompetent, and oracle policies to validate the evaluator. Report every seed, shift, rescue, and denominator rather than selecting the most legible misgeneralization trace.
Failure mode. Capability confounding labels a broken policy “misaligned,” while benchmark averaging hides capable wrong-goal cases inside general error. Post hoc environment selection can manufacture a dramatic failure, and training on the test shifts converts the claimed generalization result into adaptation.
Non-claim. A successful bounded suite does not show that goal misgeneralization is common, absent, or transferable across model families. Failure to observe it is inconclusive unless competence and positive controls pass.
Source grounding. ext_goal_misgeneralization_2022 grounds the capability-versus-goal distinction and reports examples in its own reinforcement-learning settings. ext_emergent_misalignment_reward_hacking_2025 adds a source-specific production-RL research comparator; neither experiment has been reproduced by this project.
13.8.5 Evaluation awareness, training games, and conditional deception
A model may treat the evaluator, monitor, or training process as part of the environment. Evaluation awareness is a capability observation; a training game is behavior optimized against the evaluation procedure; deceptive alignment is the stronger hypothesis that compliant behavior instrumentally preserves a later opportunity for another objective. These propositions require different evidence and must not be promoted together because a model recognizes a test or produces strategic-sounding text.
Mechanism. Cross evaluation-awareness cues with task content, strategic opportunity, monitor visibility, and deployment-like affordances. Seed known conditional policies, keep novel trigger families held out, and compare behavior when the system has or lacks the capability and authority needed for delayed pursuit. Record alternative explanations such as memorization, distribution shift, reward exploitation, and ordinary evaluator overfitting.
Failure mode. A red-team prompt can elicit theatrical deception without a persistent policy, while a transparent test environment can suppress the very behavior under study. Conversely, exposing all triggers during mitigation can train better recognition and create a false negative on the now-familiar suite.
Non-claim. The chapter does not claim naturally learned deception, situational awareness, or gradient hacking in any deployed model. Detection of a seeded sleeper policy validates only that instrument and test family.
Source grounding. ext_sleeper_agents_2024 reports constructed context-triggered policies persisting through several safety-training methods. ext_emergent_misalignment_reward_hacking_2025 reports broader misaligned generalization in one reward-hacking research setup. Both retain their construction, model, evaluator, and mitigation boundaries.
13.8.6 Independent behavioral, causal, training-process, and white-box evidence
No evidence family is an objective-identification oracle. Behavioral tests see what the policy does under sampled conditions. Causal interventions can separate explanations only when their manipulated variables and downstream effects are valid. Training-process evidence shows which pressures were present, not necessarily which mechanism the learner adopted. White-box methods can localize or perturb internal features while remaining vulnerable to approximation, distribution shift, and interpretation error. The integrity case gains strength from disagreements as well as agreements among them.
Mechanism. Give every evidence lane an independent artifact identity, evaluator, competence check, false-positive control, false-negative control, scope, and maximum inference. Predeclare how concordant, conflicting, or missing lanes affect deployment authority. Shared datasets, representations, labels, or monitors are dependency edges, not independent confirmations.
Failure mode. Evidence stacking counts correlated methods as replication, lets an interpretable feature certify its own semantics, or allows a behavioral judge trained on the same rubric to validate the training process. A single weak lane can also veto useful deployment without a calibrated consequence model.
Non-claim. Triangulation cannot prove access to a model’s true objective, eliminate unknown mechanisms, or transform argument support into empirical support before a competent campaign runs.
Source grounding. ext_learned_optimization_risks_2019 motivates transparency and internal-optimization questions; ext_sleeper_agents_2024 shows why post-training behavior alone can be misleading. These sources do not validate any ASI Stack monitor, probe, intervention, or evidence-joining rule.
13.8.7 Mitigation removal, concealment, and capability damage
A mitigation can improve observed behavior for at least three incompatible reasons: it removed the risky mechanism, it suppressed or hid that mechanism under tested conditions, or it damaged the capability needed to express it. The deployment decision needs those outcomes separately because only the first could support a bounded removal claim. Even then, novel triggers, new opportunities, and descendant changes remain residuals.
Mechanism. Evaluate retraining, adversarial training, model editing, routing, monitoring, and containment against original and sealed trigger families. Re-run base-task competence, causal probes, white-box measures, monitor-blind positive controls, evasion attempts, and opportunity tests. Record behavior suppression, mechanism evidence, capability change, detector sensitivity, and authority containment as distinct result fields.
Failure mode. Mitigation theater reports a clean behavioral score after the model has learned the test, the monitor has become correlated with the model, or useful capability has collapsed. Adversarial training may improve trigger discrimination while leaving a conditional policy intact; repeated tuning can turn the held-out denominator into training data.
Non-claim. No cited or proposed mitigation is asserted to remove deception, mesa-optimization, goal misgeneralization, or reward hacking generally. Containment is a competing risk-control mechanism, not evidence of alignment.
Source grounding. ext_sleeper_agents_2024 supplies the constructed persistence and adversarial-training caution. ext_emergent_misalignment_reward_hacking_2025 reports mitigation conditions only within its own setup. Neither establishes local efficacy, natural prevalence, or universal mitigation failure.
13.8.8 Deployment opportunity, power indicators, expiry, and descendants
Objective risk depends on what the policy can do. A model without tools, memory, time, copies, other agents, or effect authority may never express a policy that becomes material after those affordances are added. Integrity is therefore a lease over an exact deployment envelope rather than a permanent property of a checkpoint. Power and option-preservation indicators are challenge signals inside that envelope, not a universal prediction about optimal behavior.
Mechanism. Bind each integrity decision to model and checkpoint digest, prompt and router state, attached memory, tools, horizon, copy topology, counterparties, permissions, environment, monitor, and expiry. Add benign opportunities to expand options, acquire resources, avoid shutdown, or alter oversight only when evaluators can distinguish competent acceptance and refusal. Any material change reopens affected hypotheses and invalidates unjustified descendant inheritance.
Failure mode. Opportunity starvation tests a capable policy in a sandbox where risky choices are impossible, then exports assurance to a broader deployment. Weight-only rollback leaves prompts, memory, caches, routes, monitors, copies, or learned organizational state active. A formal tendency is also laundered into inevitability outside its assumptions.
Non-claim. The record does not prove that every policy seeks power, that shutdown behavior reveals its objective, or that complete rollback is currently implemented.
Source grounding. ext_optimal_policies_power_2019 offers an assumption-bound formal account of power and option preservation. alignment_field motivates revisable authority and intervention as design lineage. Neither source establishes behavior in the project’s models or a deployed descendant-control result.
13.8.9 Deceptive alignment, training games, and gradient hacking
Inner alignment becomes hardest exactly where ordinary behavioral evaluation looks most reassuring. A capable learner can discover that the training process is itself part of its environment. It may optimize through the evaluation procedure: behave acceptably while gradients, monitors, or human review are active; preserve a different policy for conditions associated with deployment; or shape the evidence used to decide whether training should continue. Several mechanisms must therefore remain distinct.
- Goal misgeneralization preserves competence while the policy generalizes a proxy, shortcut, or different objective outside training. It does not require awareness of evaluation.
- Deceptive alignment is a conditional-policy hypothesis: cooperation is instrumentally useful because it preserves a later opportunity to pursue another objective. It requires strategic capability and opportunity.
- A training game is broader. The learner models the trainer, rubric, monitor, or intervention policy and optimizes observable conduct against that process. Evaluation awareness alone does not identify the objective.
- Gradient hacking is the stronger hypothesis that a learner influences its own effective updates—through examples, activations, data selection, optimizer state, or feedback—to preserve or install behavior. Resistance to fine-tuning is not enough; the causal update pathway must be observed.
- Sleeper or trigger policies condition behavior on a rare or hidden feature. They may be implanted, learned, or accidental, so trigger specificity does not itself establish deception.
Because these hypotheses predict overlapping behavior, the chapter requires a competing-explanation matrix, not one deception score. Each row names the mechanism, capability prerequisites, opportunity, predicted interventions, disconfirming observations, and safe authority while the row remains live. The matrix compares intended generalization, ordinary shortcut learning, distribution shift, memorized triggers, reward exploitation, evaluator overfitting, deceptive alignment, training-game behavior, and update manipulation.
Separating tests vary what those explanations treat differently. Evaluator awareness changes without changing the task; strategic opportunity is added or removed; proxies are broken while causal task features remain; training signals and update state are inspected for learner-controlled pathways; and novel trigger families remain held out from mitigation. Seeded conditional policies test whether the evaluator can detect concealment at all. A white-box feature or verbalized rationale is one observation, never the verdict.
Mitigation has two denominators. Removal asks whether the risky mechanism stopped operating across known and novel triggers. Concealment asks whether the intervention merely made it harder to elicit or detect. Capability preservation, base-task competence, monitor sensitivity, and relevant opportunity must pass before a clean result can narrow the risky hypotheses. Otherwise the honest disposition is implementation- or instrument-inadequate—not “deception removed.”
flowchart LR
T["Training behavior"] --> H["Competing explanations"]
H --> G["Goal misgeneralization"]
H --> D["Deceptive policy"]
H --> E["Evaluator overfit"]
H --> U["Update manipulation"]
G --> I["Separating interventions"]
D --> I
E --> I
U --> I
I --> R{"Risky hypotheses remain?"}
R -->|yes or unknown| N["Narrow authority / monitor / hold"]
R -->|competently excluded| Q["Use-scoped qualification"]13.8.10 Required artifacts
LearnedObjectiveIntegrityRecord {
outer_target_proxy_constraints_and_uncertainty,
data_reward_feedback_optimizer_curriculum_and_selection_lineage,
model_checkpoint_and_descendant_identity,
behaviorally_equivalent_policy_and_internal_optimizer_hypotheses,
goal_generalization_shortcut_and_causal_interventions,
evaluation_awareness_conditional_policy_and_deception_tests,
behavioral_training_process_white_box_and_independent_evidence,
deployment_distribution_tools_memory_horizon_and_authority_envelope,
power_option_expansion_shutdown_and_resource_indicators,
mitigation_removal_hiding_capability_and_evasion_results,
monitoring_expiry_rollback_descendant_invalidation_and_residuals,
lifecycle_cost_and_non_authorities
}13.9 Interfaces
Human Intent and Objective Formation supply the outer target. Training and Policy Optimization supply actual signal lineage. White-Box Evidence and Adversarial Evaluation supply independently bounded observations. World Models and Planning supply deployment opportunities and shift state. Readiness consumes only a use-specific integrity envelope. RSI and Capability Replacement must invalidate inherited evidence when the model, objective, environment, or descendant changes materially.
The Evidence layer owns claim identity and support movement; the integrity process can request evidence but cannot promote itself. System Boundaries owns tool and effect authority, so an integrity pass never creates permissions. Artifact Graphs preserves checkpoint, dataset, evaluator, intervention, monitor, and decision lineage. Procedural Memory must not compile a successful evasion or shortcut into reusable policy. Operations consumes alerts and rollback triggers, while Human Factors owns whether a person can understand and exercise the intervention path in time.
These handoffs are bidirectional. An incident can revise the policy-hypothesis set; a new causal feature can reopen old evaluations; a model replacement can invalidate a monitor; an authority change can make a previously irrelevant objective strategically active. Every consumer returns the exact changed state and affected decision lease rather than a scalar “aligned” flag.
13.10 Invariants
These invariants prevent the evidence case from collapsing distinctions that the mechanism cannot recover later. They apply when a test succeeds as well as when it fails, because an impressive behavioral result can launder uncertainty just as easily as an underpowered negative result can create false confidence.
- Outer objective, training signal, learned behavior, and learned-objective hypothesis remain distinct.
- Observed compliance never proves objective identity.
- A single behaviorally equivalent explanation is not selected by convenience.
- Behavioral, white-box, and intervention evidence retain separate provenance and maximum inference.
- Missing objective evidence narrows authority; it does not prove misalignment.
- Mitigation success includes tests for concealment and capability damage.
- Evaluators and monitors cannot be the sole source of their own competence.
- Deployment opportunity and authority are part of the alignment claim.
- Material model, data, objective, or environment change expires evidence.
- Descendants inherit unresolved objective risk and invalidation.
- Power-seeking tests remain conditional on exact environment and assumptions.
- No finite packet proves the absence of deceptive or unknown policies.
13.11 Evidence
The source packet intentionally combines conceptual taxonomy, bounded empirical examples, and formal pressure. No source proves the general phenomenon the chapter governs. The first local campaign should use benign, low-capability environments with known causal features and several policies that are indistinguishable in training but diverge under held-out interventions.
Compare ordinary empirical-risk training, invariant or causal-feature baselines, adversarial training, explicit shortcut penalties, representation interventions, and authority containment. Seed known shortcuts, conditional triggers, reward exploits, and monitor-visible versus monitor-hidden policies as positive controls. Use an independently implemented evaluator and retain all failed runs and rescue attempts. Measure intended-goal success under shift, capability preservation, causal sensitivity, trigger generalization, monitor detection and evasion, white-box stability, option expansion, shutdown compliance, uncertainty, false positives, false negatives, latency, compute, and governance burden.
A null result cannot support “no inner-alignment problem” unless the training process learned the intended task, known shortcuts activated, detectors found seeded cases, the evaluator distinguished policies, power tests had opportunity, and method-specific rescue passed. Even then the maximum inference remains the exact model, environment, objective, and intervention family.
13.12 Failure modes
- mesa-optimization or hidden internal search;
- goal misgeneralization under distribution shift;
- proxy and shortcut learning;
- reward hacking and evaluator exploitation;
- evaluation-aware conditional policy;
- deceptive or strategically compliant behavior;
- mitigation-induced concealment;
- chain-of-thought or explanation laundering;
- white-box overinterpretation;
- monitor/model correlated blind spot;
- option expansion, shutdown avoidance, or resource seeking;
- objective drift after tool, memory, or horizon expansion;
- distillation or replacement preserving the risky policy;
- false assurance from behavior-only tests.
Additional failure families include hypothesis foreclosure, where the easiest-to-measure explanation crowds out viable alternatives; opportunity starvation, where tests omit the tools or horizon needed for the risky policy to appear; competence confounding, where base-task failure is mistaken for safe generalization; and detector monoculture, where every monitor inherits one representation or evaluator blind spot. Integrity can also fail through authority laundering: a bounded pass for one use is reused for a broader tool, memory, or autonomy envelope. Finally, mitigation can create selection pressure for policies that preserve the objective while changing its visible implementation. Those cases remain unresolved risk, not evidence that the objective disappeared.
13.13 Minimum Viable Implementation
Build a benign grid, game, or program-synthesis setting where two causal rules fit the training distribution and diverge under prospectively sealed interventions. Train several competent policies, seed one explicit conditional policy as a positive control, and compare behavioral, causal, and white-box evidence. Implement a use-specific deployment gate that narrows authority when the separating evidence is inconclusive.
The minimum honest implementation includes a versioned record schema, training-lineage capture, at least two behaviorally equivalent policy hypotheses, a sealed causal intervention set, a competence gate, a known conditional-policy positive control, an independently implemented outcome evaluator, and a deployment decision with expiry and rollback. A validator should reject missing model identity, unsealed interventions, failed positive controls, unsupported objective certainty, opportunity-free deception claims, and stale descendant inheritance.
The first result should report every trained policy, failed run, detector threshold, rescue, false alarm, and inconclusive case. Passing the schema or detecting the seeded policy is not evidence that natural models are internally aligned, that learned objectives are identifiable, or that the gate is safe. Support remains at argument until competent natural or realistically generated cases, held-out interventions, independent evaluation, and transfer show claim-specific value.
13.14 Mature Research Target
The mature target is objective-integrity custody across training, deployment, replacement, and self-improvement. It would automatically generate behaviorally equivalent hypotheses, choose information-gaining interventions, track mechanism evidence across checkpoints, and reduce authority when objective uncertainty intersects with new opportunity. It would integrate with causal representations, interpretability, adversarial evaluation, and formal models without claiming to read a model’s true utility function.
Beyond current practice, the target would compare explanations rather than merely score behaviors. A hypothesis engine would propose causal shortcuts, conditional policies, internal search, evaluator modeling, and simpler heuristics; an experiment planner would choose interventions by expected discrimination value and consequence; and a policy compiler would translate the remaining uncertainty into tool, memory, horizon, copying, and action limits. The system would maintain evidence independence across behavioral, training-process, causal, and mechanistic lanes and explicitly model shared blind spots.
A decisive research program would span multiple architectures, objectives, training methods, environments, capability levels, and naturally occurring as well as seeded failures. It would compare ordinary held-out evaluation, adversarial training, causal representation controls, interpretability-guided monitoring, containment, and the full record. Preregistered outcomes would include objective-relevant behavior under shift, detector calibration, causal sensitivity, evasion, capability damage, unsafe release, missed help, latency, compute, human work, rollback completeness, and governance cost. Mechanism ablations would remove hypothesis plurality, intervention selection, evidence separation, opportunity binding, or descendant invalidation one at a time.
Independent evaluators would implement task truth and policy classification without sharing the proposed monitor. Transfer would cross model family, training regime, task ontology, tool envelope, and institution. Negative inference would require competent policies, activated known shortcuts, passed positive controls, relevant opportunity, and method-specific rescue. This is a falsifiable research program, not a current result: no such campaign has passed here, and the current evidence supports only the need for the bounded integrity contract.
13.15 Formalization hooks
AsiStackProofs.LearnedObjectiveIntegrity now proves a finite non-identification boundary rather than assuming it. Two authored worlds have the same three-action compliant observation trace but distinct objective hypotheses; one takes a different action when a separating opportunity appears. Any deterministic inference that sees only the shared trace therefore cannot identify both worlds correctly. This is an impossibility result about the information in the encoded trace, not evidence that either hypothesis is an actual learned objective.
The same module implements an eight-stage lifecycle from scoped record through plural hypotheses, evidence binding, intervention review, mitigation review, bounded use, Readiness handoff, and material-change invalidation. The lifecycle requires separate behavioral, training-process, causal, and white-box lanes; competence and positive controls; sealed shift and opportunity tests; monitor-independence and disagreement records; concealment and capability-damage review; unresolved-hypothesis custody; bounded authority, expiry, rollback, and descendant ownership; independent review; and a maximum-inference boundary. Rejected events preserve exact state. Accepted events cannot assign support or external authority. The lifecycle now has explicit accepted-step and finite-run semantics. Across every successful finite run, all 14 model, checkpoint, target, signal, hypothesis, evidence, use, reviewer, consumer, residual, and protocol identity fields remain fixed; plural and unresolved hypotheses remain present; receipts equal event count; handoff and invalidation counts are monotone; accepted traces can be recovered; event batches compose exactly; and an invalidated state accepts no further event. The seven-event witness reaches the exact invalidated record. The independent consumer checks all eight prefix/suffix compositions and rejects 66/66 mutations with exact state preservation.
| Target | Exact mechanized result | Status |
|---|---|---|
lean:inner_alignment.behavior_does_not_identify_objective |
Equal compliant traces plus distinct authored hypotheses cannot both be identified by one trace-only inference; a separating opportunity distinguishes the concrete witnesses. | implemented |
lean:inner_alignment.hypothesis_review_lifecycle |
Arbitrary successful runs preserve full identity, plural/residual hypotheses, non-authority, exact receipts, monotone handoff/invalidation counts, accepted traces, composition, and terminal invalidation; one seven-event witness reaches the exact final record and 66/66 independent mutations reject without changing state. | implemented |
The formal result does not prove or identify a model objective, detect mesa-optimization or deception, establish evaluator or monitor independence, show that a mitigation removes rather than hides behavior, validate a deployment gate, or prove alignment or safety. Chapter support remains argument. Those outcomes require the planned competent campaign and Project Theseus runtime integration.
13.16 Codex test plan
| Test | Purpose | Status |
|---|---|---|
| Goal-versus-capability shift suite | Distinguish capable wrong-goal behavior from ordinary failure. | planned |
| Behaviorally equivalent policy intervention | Break a training correlation under sealed held-out conditions. | planned |
| Conditional-policy positive control | Verify monitors and evaluators can detect a known trigger policy. | planned |
| Mitigation hiding test | Check whether intervention suppresses, removes, or makes behavior harder to elicit. | planned |
| Power/opportunity boundary | Test option expansion only where the environment makes it available. | planned |
| Descendant invalidation | Ensure distillation, replacement, or self-modification cannot inherit stale objective evidence. | planned |
13.17 Source crosswalk
| Source | Contribution | Boundary |
|---|---|---|
ext_learned_optimization_risks_2019 |
Mesa-optimization and learned-objective taxonomy. | Conceptual analysis; no local optimizer identified. |
ext_goal_misgeneralization_2022 |
Capability-versus-goal generalization distinction and bounded examples. | Source environments not reproduced. |
ext_sleeper_agents_2024 |
Persistent constructed conditional policies and mitigation caution. | Proof of concept, not natural deception. |
ext_emergent_misalignment_reward_hacking_2025 |
Reward-hacking-linked misaligned generalization in a production-RL research setup. | Source-specific and not reproduced. |
ext_optimal_policies_power_2019 |
Conditional formal pressure toward power/option preservation. | Assumption-bound; not proof about a deployed model. |
alignment_field |
Corben design lineage for alignment as a revisable relation among objectives, evidence, and intervention. | Speculative design source; no learned-objective identification or inner-alignment result. |
13.18 Summary
Inner alignment governs the uncertainty between the objective the outer system intended and the policies a learner may actually implement. Its most important rule is epistemic: successful behavior, reward, reasoning traces, and internal features are evidence, but none alone identifies the learned objective. When that uncertainty meets consequential opportunity, authority must narrow until the evidence improves.
The operative object is not a binary “aligned” label but a versioned set of policy hypotheses, discriminating tests, evidence limits, opportunity conditions, and deployment consequences. Competence and opportunity controls prevent an incapable or strategically constrained policy from producing false reassurance. Behavioral, causal, mechanistic, and training-process evidence remain separate so correlated methods cannot manufacture certainty.
Mitigation is judged on removal, concealment, evasion, capability preservation, and transfer, while every material model, data, monitor, memory, tool, route, or authority change can expire the case. Descendants inherit residual risk rather than trust by lineage. When evidence cannot distinguish an intended rule from a dangerous compatible policy, the honest response is narrower authority, additional information, quarantine, rollback, or refusal—not a stronger objective claim.
13.19 Handoff
Inner Alignment hands its bounded learned-objective hypotheses and unresolved residuals to Moral Uncertainty, Value Conflict, and Contestable Governance. Objective-integrity evidence can narrow what a policy appears to pursue, but it cannot determine which contested values are morally correct or authorize action under unresolved disagreement. It preserves that uncertainty for the next owning decision instead of converting an integrity assessment into moral authority.
13.20 Sources
See the source crosswalk above and the generated external-source appendix.