Skip to main content

79  Policy Optimization and Learning from Feedback

79.1 Chapter status

Field	Value
Chapter ID	policy-optimization-and-learning-from-feedback
Part	Part IV - Evidence, Implementation, and the Living Book
Status	conceptual
Manuscript maturity	v0.3 manuscript draft
Last updated	2026-07-15
Primary source records	Twenty-nine exact passage-reviewed assignments; no source result imported
Claim label	Design rationale
Evidence level	argument
Source queue	primary: benchmaxxing, rmi; supporting: spinoza, verification_bandwidth, talos, vcm_public, planforge, octopus_router, scf, tokenmana, cognitive_loop_closure; external variants: TRPO, PPO, ReMax, DPO, IPO/preference theory, ORPO, KTO, SimPO, REINFORCE-style RLHF, DeepSeek-R1, DAPO, GSPO, S-GRPO, LongRLVR, RLHF limitations; connector/recovery: moecot; handoff: sources/inbox/policy_optimization_browser_note_2026-06-24/
Source loading state	source notes: learning_compute_topology, benchmaxxing, rmi, spinoza, verification_bandwidth, talos, vcm_public, planforge, octopus_router, scf, tokenmana, cognitive_loop_closure, moecot, ext_trpo_2015, ext_ppo_2017, ext_remax_2023, ext_dpo_2023, ext_ipo_preference_2023, ext_orpo_2024, ext_kto_2024, ext_simpo_2024, ext_reinforce_style_rlhf_2024, ext_deepseek_r1_2025, ext_dapo_2025, ext_gspo_2025, ext_s_grpo_2025, ext_longrlvr_2026, ext_rlhf_limitations_2023, ext_muse_unlearning_2025, ext_faithfulness_information_flow_2026, portia_synapse, spider_synapse, regret_engine, ext_eggroll_hyperscale_es_2026, ext_openai_es_2017; raw cache: benchmaxxing, rmi, spinoza, verification_bandwidth, talos, vcm_public, planforge, octopus_router, scf, tokenmana, cognitive_loop_closure; connector/recovery: moecot
Test state	policy_optimization_record.valid.json passes repository-level protocol fixture validation; AsiStackProofs.PolicyOptimization implements finite policy-update record and reward/governance promotion gates plus bounded negative cases for missing holdout or contamination checks, reward-proxy promotion without target evaluation, authority-expanding policy updates without approval or rollback, policy-promotion route failures for inadmissible feedback, missing target evaluation, missing reward-hacking probes, governance or authority gaps, missing rollback, and regression/residual gaps, and the policy-update lease fixture bridge; python3 scripts/validate_benchmark_antigoodhart.py validates synthetic feedback admissibility, reward-boundary, reward-hacking-probe, evidence-packet, governance-gate, blocked-ratchet, and promotion-gate behavior; python3 scripts/validate_policy_update_lease_probe.py validates experiments/policy_update_lease/results/2026-07-02-local.json, a deterministic synthetic router-policy canary lease with holdouts, contamination check, reward-hacking probes, unchanged authority, expected-invalid controls, and dry-run rollback. No policy optimization experiment has been run.

79.2 Drafting guardrail

Policy optimization is a governed update mechanism for the stack, not evidence that any model, router, planner, verifier, or context policy has improved. The browser-GPT packet that motivated this layer is local-only author-intent context. The external RL papers named here now have stable source records, primary arXiv metadata, and conservative source notes; they support method-family context only, not support-state promotion or local training claims.

Governed Operations may hand this layer an incident residual, failed control, affected identity, cost record, or bounded change proposal. It may not hand over emergency command authority or silently convert a postmortem lesson into an update. Every such input enters the ordinary update lease with its evidence, uncertainty, privacy limits, affected paths, rollback boundary, and non-claims.

Benchmark ratchets turn evaluation pressure into evidence records. Policy optimization asks how that pressure may change future behavior without letting reward become authority. A policy update is allowed to propose better planning, routing, context selection, verification, execution, or generation only when admissible feedback, drift bounds, evaluations, rollback, and governance gates are recorded.

The policy update is a behavior-change lease. It grants a candidate policy limited authority to act under a target layer, evaluation scope, drift bound, rollback condition, and monitor window before it can become ordinary behavior.

79.3 Human Reading Path

Concrete lens. The reward-only baseline promotes the faster router. The governed lease uses independent target evaluation and rejects the proxy-improving, target-regressing update.

Benchmark ratchets create improvement pressure that policy updates must govern carefully. Policy optimization is how that pressure can update planning, routing, context selection, verification, execution, or review behavior without letting reward become authority.

The distinction is crucial: learning from feedback is not the same as handing the system control over its own goals. A policy update should carry evidence, scope, rollback, residuals, and governance review before it becomes the new default.

At its strongest, the loop is less like tuning a single model and more like approving a behavior-change lease. The lease names what may change, what must not change, how regressions are detected, and who can roll it back.

Feedback becomes governance-compatible when improvement remains reversible and evidence-scoped.

The learning loop should update behavior without rewriting the system’s authority. Good feedback teaches the policy what to prefer, not what it may ignore.

A better policy deserves adoption only when its new habits remain bounded by older duties. Learning pressure belongs inside the same contract as deployment authority, and deployment still sets boundaries after training pressure changes behavior in production.

79.4 Problem

Benchmarks, verifiers, human preferences, tool results, context-grounding checks, and governance violations all produce feedback. A living ASI stack needs a disciplined way to turn that feedback into future behavior. Without that layer, the book can explain how to measure, review, and gate capabilities, but not how the system learns which behaviors to repeat, suppress, shorten, route, verify, or retire.

The hard part is not saying “use reinforcement learning.” The hard part is deciding what policy is being updated, what feedback is admissible, how far the update may move, what evidence is required before promotion, and how rollback works when the reward was wrong.

That problem spans more than model weights. A stack can update planning heuristics, router policies, context selection, verification budgets, tool-use preferences, review workflows, or generation modes. Each update surface has different failure modes. Without a typed policy boundary, reward pressure can quietly become governance pressure, and a local improvement can change system behavior beyond the evidence that justified it.

79.5 Why existing approaches are insufficient

RLHF, preference optimization, and verifier-reward training are often discussed as model fine-tuning recipes. That framing is too narrow for this architecture. The ASI Stack has many policies: planner decomposition, VCM context selection, router expert choice, generation-mode choice, verifier escalation, execution/tool policy, benchmark admission, and SCF promotion decisions.

A reward signal can also become a failure channel. Human preference can reward persuasive wrongness. A verifier can be gamed. A benchmark can become a target instead of a pressure surface. A latency reward can suppress needed verification. A router reward can learn cheap shortcuts that pass local tests while damaging cross-layer traceability.

The stack therefore needs policy optimization to be governed by the same evidence discipline as every other layer.

Reward substitution is the policy-learning failure. A reward, preference label, verifier score, latency metric, or benchmark result stands in for the actual claim: safer routing, better context, truthful answers, lower cost, or stronger reasoning. The update record must keep the proxy and the target claim separate.

79.5.1 Strongest objection

Governance can constrain an update record without making the feedback signal truthful. Correlated evaluators, strategic behavior, and distribution shift can all produce a clean causal chain from reward to checkpoint to output while the target capability worsens. These contracts prevent proxy/claim conflation; they do not prove reward adequacy or policy improvement.

79.6 Core Claim

[policy-optimization-and-learning-from-feedback.core, label: Design rationale, support: argument] Policy Optimization and Learning from Feedback owns a target-policy-, baseline-, objective-, feedback-, evaluator-, dataset-, optimizer-, checkpoint-, rollout-, authority-, resource-, monitor-, rollback-, consumer-, environment-, and time-specific Governed Policy Update Lease: before any update, it freezes the legitimate target behavior, admissible feedback and proxy boundary, strong baselines, update family and budget, drift and authority ceilings, complete evaluation and failure denominators, reward-hacking and causal checks, rollback and monitoring, residuals, expiry, and promotion authority; a reward, preference, verifier score, benchmark gain, loss reduction, synthetic canary, formal route, rollback dry run, or trained checkpoint alone establishes neither reward validity, causal policy improvement, retained capability, alignment, safety, readiness, deployment, support, transfer, nor SOTA.

Reader claim. Reward is a training signal, not evidence that the target behavior improved; the proxy and the target must be evaluated separately.

Operational rule. Freeze the target policy, admissible feedback, proxy boundary, baseline, drift and authority ceilings, holdouts, hacking probes, rollback, and monitor before updating. Promote only on independent target evaluation with every failed rollout and candidate retained.

79.6.1 Worked policy update: reward rises while the target falls

A router policy is trained to reduce latency. Its scalar reward improves because it learns to skip expensive verification. On the training distribution, requests finish faster and the reward curve is clean. On a frozen target evaluation, however, unsupported releases increase. The update therefore fails even though the optimizer, checkpoint, and reward receipt are all valid. The previous policy remains the default and the reward design is recorded as the primary residual.

The composed local refinement starts from a six-sample, five-candidate lease and follows a seven-stage update lifecycle. It checks 63 routes, rejects 73 route mutations and three cross-stage mutations, and preserves the reward/target distinction through rollback. This makes proxy failure visible but does not establish that the target metric is legitimate, the evaluator is independent, the rollback reverses every effect, or the policy improved causally.

The claim remains at argument support. The assigned ASI Stack sources support benchmark pressure, residual escrow, proof/verification discipline, context adequacy, execution ledgers, routing, resource budgets, and stable capability gates. The external RL literature records source-note method families and failure cautions, but no external-literature-backed support state is claimed yet.

79.6.2 Claim-source mapping status

Appendix C records passage-reviewed mappings for policy optimization. The internal ASI Stack mappings use local raw-cache passages for benchmark pressure, modular ratcheting, verifier tiers, context adequacy, execution feedback, planner and router policy targets, SCF governance, resource pressure, and procedural loop closure. The complete authenticated moecot connector text is passage-reviewed implementation-reference context; private raw text is not published and runtime artifacts are not reproduced. The external method-family records use public arXiv abstract/metadata references plus conservative source notes; they are taxonomy and risk context, not reproduced experiments.

Mapping group	Reviewed support	Boundary
Internal evidence/governance sources	Benchmark ratchets, residual escrow, verifier tiering, verification bandwidth, typed execution ledgers, context adequacy, planner/router policy targets, SCF gates, resource pressure, and procedural compilation.	No policy training, reward-quality result, benchmark result, route-quality result, verifier-quality result, context-policy result, or support-state promotion.
moecot connector/recovery source	Compact orchestration, specialist lanes, readiness gates, replay, promotion blockers, residual tracking, and implementation-reference policy-update context.	Source-note only; no runtime artifact, benchmark artifact, log, or policy-update result has been imported or reproduced.
External online-RL sources	TRPO/PPO/ReMax/REINFORCE-style method-family context for bounded or simpler policy-gradient updates.	arXiv abstract/metadata and source-note review only; no online RL run, environment, reward model, benchmark, or ASI Stack policy update is reproduced.
External preference-optimization sources	DPO, IPO/preference theory, ORPO, KTO, and SimPO context for preference, binary-feedback, reference-free, and sequence-level objectives.	No local preference data, preference trainer, reward-model validation, preference-quality study, or governance-safe behavior result.
External reasoning/RLVR/limitations sources	DeepSeek-R1, DAPO, GSPO, S-GRPO, LongRLVR, and RLHF-limitations context for reasoning RL, sequence-level updates, early exit, verifiable context rewards, and reward-process risks.	No reasoning-RL reproduction, context-grounding reward experiment, model-quality result, early-exit result, mitigation result, or deployed safeguard.

79.7 Mechanism

79.7.1 Eighteen-stage update lifecycle

The complete lifecycle freezes the legitimate target and consumer; binds exact policy, model, data, tool, runtime, and authority identities; classifies feedback provenance and its proxy boundary; selects an update family only after target/feedback fit; freezes optimizer, budget, seeds, stopping, and checkpoint selection; gives strong baselines matched opportunity; declares drift and authority constraints; freezes holdouts, contamination, adversarial, tail, and transfer evaluation; attacks the reward and evaluator; retains every rollout, failure, retry, checkpoint, and cost; runs causal ablations; emits a policy- delta receipt; governs lifecycle states; rehearses effect-complete rollback; monitors natural outcomes; expires material changes; and adjudicates target quality, safety, rights, authority, resources, residuals, reproduction, and transfer jointly.

Policy optimization enters the stack as a governed update ledger. Before any behavior is changed, the stack records the update state, which policy is being updated, the proposed policy delta, which feedback artifact is admissible, the reward boundary, what drift bound applies, how reward hacking will be probed, which holdouts and regressions constrain the update, what authority effect is allowed, which residuals remain, and which governance gate can promote or reject the update.

flowchart LR
  A["Feedback artifact"] --> B["Admissibility check"]
  B --> C["Target policy and update family"]
  C --> D["Reward or preference boundary"]
  D --> E["Drift bound + rollback plan"]
  E --> F["Holdouts, regressions, and reward-hacking probes"]
  F --> G["SCF / governance gate"]
  G --> H{"Promote update?"}
  H -- "yes" --> I["Promotion with rollback handle"]
  H -- "no" --> J["Quarantine, rollback, or residual"]
  I --> K["Claim/evidence ledger"]
  J --> K

What the feedback update gate shows: Feedback does not update behavior until admissibility, target policy, reward boundary, drift bound, rollback, holdouts, regressions, and reward-hacking probes are visible. Promotion is a governance decision, while failed updates route to quarantine, rollback, or residuals.

A policy update record separates:

• Target policy: planner, VCM, router, generator, verifier, execution, governance, generation mode, or whole-stack policy.
• Training mode: SFT, policy gradient, PPO-like online RL, group-relative RL, offline preference optimization, RLVR, process reward, or control-policy RL.
• Feedback source: human preference, exact verifier, process verifier, tool result, context-grounding check, governance check, benchmark result, latency/cost signal, or residual.
• Feedback admissibility and reward boundary.
• Update constraint: KL, clipping, group-relative comparison, sequence-level bound, route-change bound, SCF boundary, or another explicit drift limit.
• Evaluation refs: holdouts, baselines, regressions, reward-hacking probes, contamination checks, residual accounting, and negative results.
• Governance refs: authority effect, authority limits, high-impact review, monitor window, evidence packet refs, rollback plan, deployment scope, support-state effect, and promotion decision.

The core rule is simple: policy optimization may learn from feedback, but the book should never let a reward signal silently become evidence, authority, or readiness.

79.7.2 Regret packets are protected feedback, not actor rewards

The Regret Engine source (regret_engine) adds a typed feedback artifact between observed outcomes and this update lease. A Regret Packet binds the decision-time capsule, admitted and rejected comparators, actual outcome evidence, model-conditioned counterfactual intervals, hard violations, causal contribution, foreseeability, feasibility, recurrence, counterfactual surplus, root-cause hypotheses, and unresolved residuals. Severity, contribution, and learning eligibility remain separate: an incident can demand immediate containment while still being too uncertain for gradient exposure.

The safest consumption mode is diagnostic. Replay priority or an auxiliary prediction target requires more evidence; constrained or direct objective shaping requires stronger independence, calibration, double-counting checks, and tampering probes. The actor may propose alternatives and self-critiques, but it cannot be the sole authority that admits comparators, chooses weights, assigns its own root cause, declares learning eligibility, or promotes the resulting update. Removing evidence must widen uncertainty or leave it unchanged, never earn a better audit classification.

This packet does not replace the Governed Policy Update Lease. It supplies one candidate feedback input to a lease that still freezes update surface, replay policy, protected capabilities, drift and resource limits, monitor window, rollback, expiry, and independent promotion authority. No Regret Engine was implemented, and no policy improvement or anti-tampering effect is inferred.

79.7.3 Informative exceptions do not self-authorize negative pressure

Assurance-Shift Learning (assurance_shift_learning) sharpens the conditions under which a failure-like event can reach that lease. Ordinary positive support must first be broad and stable in the declared region; outcome quality and process quality are scored separately; the evaluator must distinguish the defect from bad luck, ambiguity, or instrumentation failure; and the bundle must preserve the successful prefix, protected positive behavior, a candidate correction, a counterexample to overbroad repair, and unresolved causal uncertainty. There is no direct Observed -> Train edge.

Negative pressure remains one repair surface among many. A test, guard, procedure, memory entry, tool correction, recovery route, adapter, or module may be narrower and more reversible than changing policy weights. The update gate must therefore compare placements and test their interactions before it accepts a gradient-bearing condition. This is a proposed allocation discipline, not evidence that negative feedback improves policy or resists tampering.

Every update should produce a policy-delta receipt. The receipt records what behavior changed, which feedback authorized the change, which probes tried to break the reward, which holdouts and regressions were checked, which authority boundary remained unchanged, what monitor window was opened, which evidence packet was produced, which deployment scope was allowed, and how rollback would restore the prior policy.

79.7.4 Portia and Spider: optimize the smallest learning contract first

The SpiderSynapse-to-PortiaSynapse transition supplies a compact case study in why architectural repair should begin with the smallest falsifiable learning contract. Spider combined four hypotheses, three refinements, working memory, a selector, and several output heads before it had shown that one path could learn one coordinate target. Its reported plateau is worth retaining, but the claim that each path received only one twelfth of the gradient is a causal hypothesis, not an established law. Gradient magnitude, path utilization, prediction diversity, selector credit, target geometry, and label smoothing were not independently identified.

The repaired protocol begins at K=1, N=1, coordinate-only training. It records per-layer gradients, activations, exact-coordinate accuracy, route or task success, and held-out learning curves before adding an edge head. Focus and mutable memory enter only after a memory-disabled ablation shows a task that needs them. Confidence enters only after its target event is named and calibration is evaluated. Full integration is the final phase, not evidence that the earlier phases succeeded.

Phase progression is triggered by frozen evidence gates rather than a fixed fraction of training. Each gate requires matched no-component and simpler baselines, seed replication, exact denominators, leakage checks over generated walks and known paths, and rollback to the last admitted phase. A causal matrix varies branch count, refinement depth, memory, selector, label smoothing, and loss geometry independently. This prevents a later architecture from retroactively converting a predecessor’s opaque failure into proof of its own explanation.

79.8 Method families

The external literature named in the browser packet is best organized by what the update is trying to do.

Family	Role in the stack	Current repository status
Foundational policy gradients	Minimal reward-learning substrate: REINFORCE-style estimators, baselines, RLOO, ReMax-like critic-free simplifications.	Source-noted via ext_reinforce_style_rlhf_2024 and ext_remax_2023; no local trainer
Trust-region and clipped online RL	Bounded online exploration with update-size control: TRPO, PPO, PPO-RLHF, GRPO, and descendants.	Source-noted via ext_trpo_2015, ext_ppo_2017, ext_dapo_2025, and ext_gspo_2025; no local online RL run
Zeroth-order and evolution strategies	Optimize outcome-level fitness through parameter perturbations when action-level gradients, critics, or differentiable objectives are unavailable.	Source-noted via ext_openai_es_2017 and ext_eggroll_hyperscale_es_2026; no local ES run
Offline preference optimization	Stable updates from static preference or binary feedback data: DPO, IPO, ORPO, KTO, SimPO.	Source-noted via ext_dpo_2023, ext_ipo_preference_2023, ext_orpo_2024, ext_kto_2024, and ext_simpo_2024; no local preference baseline
Verifier-based RL / RLVR	Learn from exact, process, tool, context, governance, or efficiency verifiers.	Source-noted via ext_longrlvr_2026; process-reward sources beyond this remain queued
Reasoning-budget and latency-aware RL	Learn when to think, stop, retrieve, verify, use tools, or switch generation modes.	Source-noted via ext_deepseek_r1_2025, ext_s_grpo_2025, and ext_rlhf_limitations_2023; no local reasoning-budget run
Control-policy RL	Train planner, router, VCM, execution, and generation-mode policies rather than only answer style.	Planned ASI Stack experiments

Policy optimization should not become an acronym catalog. Acronyms change quickly; the architectural question remains stable: what policy is being updated, what reward is admissible, and what gate decides whether the update survives?

The method-family list is therefore subordinate to governance. PPO-like, DPO-like, GRPO-like, RLVR-like, or custom control-policy updates all need the same record: target policy, reward boundary, drift limit, evaluation suite, failure probes, rollback, and promotion gate.

79.8.1 Fitness is feedback, not authority

79.8.2 Evaluator topology and credit topology are different

A policy-update pipeline often draws one arrow from reward to parameters. The LCT projection requires at least four events: evidence is observed, an evaluator produces judgement, a credit mechanism assigns responsibility, and an update changes a versioned adaptive identity. Multiple reward models can increase evaluator width while sharing one blind spot; many actors can increase evidence width while one learner and one credit rule remain narrow.

This matters when branching policies. Forking creates option capacity but does not create conditional information unless branches receive differentiated evidence, objectives, environments, constraints, or stochastic histories. Candidate selection is also limited by the information carried through the evaluator channel, and exact retention is limited by the integration channel. Scaling candidates while leaving those cuts fixed can increase cost and proxy exploitation without increasing retained learning.

The update dossier now records adaptive identities, evaluator identities and correlations, judgement type, credit operator, held-out evidence route, integration operator, lifecycle decision, and realization leakage. These fields do not validate a reward or policy update. They prevent a wide-looking system from hiding a single captured evaluator, an unexamined credit rule, or a destructive merge.

Evolution strategies clarify a distinction that policy-gradient vocabulary can hide. A learner can improve an outcome score without differentiating through the environment or assigning credit to individual actions. OpenAI ES returns scalar fitness from distributed perturbation rollouts; EGGROLL batches low-rank parameter perturbations and can optimize discrete rewards. This enlarges the mechanism set, but not the reward’s authority.

A nondifferentiable reward can still reward shortcuts, exploit an evaluator, collapse diversity, or omit safety constraints. Before population search, the record freezes the fitness function and evaluator version, population and seed policy, perturbation scale and rank, data and environment exposure, invalid-run handling, selection/normalization rule, total candidate denominator, and final independent evaluation. A result cannot discard low-fitness or crashed members and then describe only the surviving lineage. The reward proposes an update; the normal drift, regression, rights, safety, cost, and promotion gates decide whether it survives.

The target policy should be explicit before an optimizer is named.

Stack layer	Policy that can be optimized	Reward boundary
Planner	Decomposition, tool choice, stopping, escalation.	Task success cannot erase constraints, approvals, or residuals.
VCM	Context selection, compression, grounding, context refresh.	More compact context is not better unless evidence adequacy is preserved.
Router / MoECOT	Expert selection, fallback, quorum, latency/cost tradeoff.	Cheap routing cannot bypass readiness, authority, or residual escrow.
Generator	Answer quality, uncertainty style, repair behavior, reasoning length.	Fluent or concise output is not evidence without verifier support.
Verifier / Spinoza	Critique accuracy, contradiction detection, calibrated abstention.	A verifier reward must be audited for gaming and coverage gaps.
Talos execution	Tool policy, job sequencing, artifact success, approval compliance.	Tool success cannot expand the authority envelope.
SCF governance	Promotion, rollback, quarantine, field replacement.	Governance rewards must preserve rights, review, and rollback.
Generation mode	AR, speculative, diffusion-like, early-exit, or slow verified route selection.	Speed is only useful when accepted-output quality and fallback are recorded.

The training mode should match the available feedback and risk.

Training mode	Best when	Weakness to record
SFT	High-quality demonstrations exist.	Imitation does not explore better policies.
DPO / IPO / ORPO / KTO / SimPO	Static preference or binary feedback is available.	Limited online exploration and dependence on preference quality.
PPO-style online RL	Online rollout and reward optimization are needed.	Critic cost, tuning sensitivity, and reward-model risk.
GRPO / RLOO / ReMax-style critic-free RL	Multiple samples per prompt can create useful reward contrast.	Needs reliable contrast and can still overfit reward artifacts.
DAPO / GSPO-style variants	Long reasoning, sequence-level rewards, or MoE-specific training needs different granularity.	Fast-moving literature and higher reproduction risk.
RLVR and process rewards	Automatic verifiers, tests, proofs, or step-level checks are available.	Verifiers can be sparse, brittle, or gamed.
Control-policy RL	Planner, router, VCM, execution, verifier, or generation-mode policies need improvement.	Requires system-level reward design and authority-aware evaluation.

79.9 Interfaces

flowchart TD
  Bench["Benchmarks and residuals"] --> Reward["Reward / preference signal"]
  Verify["Spinoza, tests, VCM adequacy, Talos artifacts"] --> Reward
  Reward --> Update["Policy optimization"]
  Update --> Eval["Holdouts, regressions, reward-hacking probes"]
  Eval --> Gate["SCF and governance gates"]
  Gate --> Promote["Promote with rollback handle"]
  Gate --> Block["Quarantine / rollback / residual"]
  Promote --> Ledger["Claim and evidence ledger"]
  Block --> Ledger

Policy optimization connects to nearly every layer:

• Planner policy: reward task success, constraint preservation, fewer wasted steps, and correct stopping.
• VCM policy: reward evidence-grounded retrieval, context adequacy, lower context bloat, and fewer unsupported claims.
• Router policy: reward correct specialist choice, authority fit, readiness fit, latency, and cost.
• Generator policy: reward verified answer quality, concise reasoning when sufficient, and repairable uncertainty.
• Verifier policy: reward critique accuracy, evidence judgment, contradiction detection, and calibrated abstention.
• Execution policy: reward artifact success, safe tool use, approval compliance, and replayable logs.
• Governance policy: reward promotion decisions that preserve regression floors, rollback, rights, and auditability.
• Generation-mode policy: reward verified output quality per wall-clock cost while preserving fallback.

Training evidence, evaluation evidence, governance evidence, and deployment evidence have to remain separate. A successful training run is not a promotion decision; a promotion decision is not a long-term safety result; a rollback handle is not proof that rollback works until it is exercised or dry-run.

The complete twelve-owner boundary keeps Benchmark, Adversarial Evaluation, Evidence/Spinoza/UAT, target-layer owners, Data/Supply Chain, Artifact/Talos, Runtime/Security, Stable Capability/Replacement/Readiness, Threshold/Assurance, Resource, Formal, and Incident/Release/Living-Book authority distinct. The policy-update layer owns the bounded update lease, not the truth of feedback, target semantics, runtime enforcement, support movement, readiness, release, or public claims.

79.10 Invariants

• Reward is not evidence unless the verifier and evaluation boundary are recorded.
• Policy updates cannot expand authority by training side effect.
• A faster or shorter policy is not promoted unless task success and regressions are preserved.
• Reward-hacking probes are part of the update record, not optional commentary.
• Rollback remains available for promoted updates.

The most important invariant is authority conservation. A policy update may improve route choice, context selection, or reasoning length, but it cannot create permission to act outside the authority envelope that governed the original policy.

Proxy humility keeps optimization from outrunning evidence. Optimizing a proxy cannot promote the target claim unless target-specific evaluation, residuals, and failure probes are recorded.

The full invariant set additionally requires legitimate-objective and consumer scope, provenance and rights, independent promotion authority, optimizer-family humility, complete selection and cost denominators, matched strong baselines, prospectively frozen holdouts and stopping, target/proxy separation, retained quality floors, independent feedback validation, lifecycle-state separation, full-state rollback, natural post-update monitoring, lease expiry, independent reproduction and transfer, and the rule that finite artifacts prove no causal policy improvement.

79.11 Failure modes

• Reward hacking or verifier gaming.
• Over-optimization for preference style rather than truth or task success.
• Planner, router, or context policies learning hidden shortcuts.
• Latency rewards suppressing needed verification.
• Policy drift crossing SCF authority or readiness boundaries.

Reward, verifier, regression, and authority failures block promotion or force rollback. If the reward is ambiguous, the update remains experimental. If the verifier can be gamed, the verifier enters residual escrow. If the update improves one metric while harming regressions, benchmark ratchets preserve the regression floor rather than celebrating the local gain.

Reward laundering is the policy-optimization failure. The policy improves the reward, then the writeup describes it as improving the underlying task while reward hacks, preference bias, verifier blind spots, or regression losses remain unexamined.

The complete failure register also covers evaluator feedback loops, offline- preference selection and rights failures, unsafe online exploration, checkpoint and seed cherry-picking, baseline starvation, outcome-aware tests, authority and training-completion laundering, full-state rollback theater, forgetting and mode collapse hidden by aggregates, mitigation substitution, update-churn cost, and unsupported transfer.

79.12 Minimum Viable Implementation

79.12.1 Exact current minimum

The repository currently has one public policy-optimization schema and valid protocol fixture plus one persisted deterministic lease-probe result over six synthetic routing samples, two holdouts, five candidate policies, three expected-invalid candidates, one selected canary kept experimental, and one fixture-only rollback dry run. Eleven useful legacy contradiction and route theorems now sit inside a seven-stage, 63-route update lifecycle checked by an independent consumer with 73/73 route and 3/3 cross-stage mutations rejected. Two composition theorems derive the complete cycle and a failed-evaluation prefix. The adjacent anti-Goodhart harness belongs to Benchmark Ratchets. No optimizer, model, live router, natural workload, preference dataset, reward model, real canary, deployed rollback, causal improvement, independent reproduction, or transfer ran. The core-claim support effect is exactly zero.

Policy learning should begin as policy_optimization_record.schema.json with one valid fixture, one deterministic policy-update lease probe, and a finite Lean module.

The fixture records update state, target layer, policy ref, policy delta summary, training mode, feedback source, feedback admissibility, reward or preference signal, reward boundary, verifier refs, reward-hacking probes, update constraint, drift bound, holdouts, regressions, evaluation refs, governance gate refs, authority effect, rollback plan, monitor window, evidence packet refs, deployment scope, promotion decision, measurement status, support-state effect, residuals, and non-claims. It validates the record shape only. The Lean envelope adds finite rejection cases for promotion candidates missing holdout or contamination checks, reward-proxy improvements used as sole evidence without target evaluation, authority-expanding policy updates that omit governance approval or rollback, route-level policy-promotion reviews that reject or route inadmissible feedback, missing target evaluation, missing reward-hacking probes, governance or authority gaps, missing rollback, and regression/residual gaps, and a fixture bridge for a synthetic policy-update lease rehearsal. It does not run PPO, DPO, GRPO, RLVR, router-policy RL, context-policy RL, or reasoning-budget RL.

The Policy update lease probe at experiments/policy_update_lease/results/2026-07-02-local.json is the first bounded rehearsal of the lease idea. It uses six synthetic router-policy samples, five candidate policies, one selected source-grounded canary kept experimental, and three expected-invalid controls: a reward-only fast route, an authority-expanding route, and a missing-rollback route. The validator checks holdout coverage, contamination check presence, reward-hacking probes, unchanged authority, rollback plan, residuals, and non-claims, then injects a synthetic monitor event and records that the dry-run rollback restores the baseline policy. This is a deterministic fixture only: it is no PPO, DPO, GRPO, RLVR, router-policy, context-policy, execution-policy, verifier-policy, or reasoning-budget training run, and it is not a deployed canary, live rollback, route-quality result, reward-quality result, policy-improvement result, or support-state transition.

The update-lifecycle refinement at experiments/policy_optimization_refinement/results/2026-07-16-local.json reruns that exact lease probe through a separately implemented consumer, then reaches all 63 routes across scope, full training-state binding, update receipt, independent evaluation, adjudication, bounded lease, and material-change readmission. It binds model and candidate checkpoints together with optimizer, scheduler, RNG, cache, backup, descendant, evaluator, rollback, consumer, authority, and version identities; it also requires target evaluation, a strong baseline, causal ablation, reward-hacking probes, regressions, forgetting, safety and rights checks, uncertainty, monitoring, effect-complete rollback, and descendant invalidation. Two Lean composition theorems execute the full seven-transition cycle and a failed-evaluation prefix. All 73 route mutations and three cross-stage mutations reject: missing causal ablation prevents adjudication, lease, and readmission; support-authority laundering prevents lease and readmission; and incomplete effect rollback prevents readmission after the bounded lease. The complete witness emits one bounded lease before version-2 readmission. These are authored record consequences, not evidence that any field is true or effective: no training, learning, reward validation, causal improvement, live rollback, monitoring, reproduction, transfer, release, or support transition occurred.

A policy-delta trace should carry the first governed-learning rehearsal: one offline preference update kept experimental, one reward-hacking probe that blocks promotion, one router-policy canary with authority unchanged, one latency reward rejected for suppressing verification, and one rollback dry run.

79.12.2 Argument-exit campaign

To move beyond argument, preregister natural planner, context, router, verifier, execution, generation, and governance workloads. Give no-update, deterministic, simple controller, online-RL, offline-preference, verifier-RL, and alternative- method baselines matched model/data/tool/context/compute/tuning/retry/evaluator/ engineering opportunity. Retain all rollouts, seeds, checkpoints, failures, repairs, exclusions, unsafe artifacts, and costs. Freeze hidden holdouts, contamination, target evaluations, reward-hacking probes, tails, subgroups, authority and rights floors, stopping, and checkpoint selection before outcomes. Run feedback substitutions, reward perturbations, component ablations, causal tests, forgetting studies, natural post-update monitoring, and effect-complete rollback over model, optimizer, scheduler, RNG, caches, routes, credentials, descendants, effects, monitors, and reliance. Measure useful throughput, target quality, calibration, safety, privacy, rights, authority, latency, compute, energy, human burden, opportunity cost, and residuals jointly; then require independent trainers, evaluators, data custody, institutions, and heterogeneous transfer. Positive, negative, null, inconclusive, narrowed, refuted, rolled-back, deprecated, retired, and blocked-after-full-attempt outcomes remain terminal evidence.

79.13 Mature Research Target

The learning endpoint is a governed actuator for behavior change across the entire stack. Policy optimization would not be a fine-tuning side quest attached to the generator. It would be the controlled mechanism by which planners, routers, context selectors, verifiers, execution policies, generation modes, and governance assistants become better without gaining new authority by training side effect.

A mature policy-update lane treats each update as a behavior-change lease. The lease would name the target policy, old and new behavior, admissible feedback, reward or preference boundary, update family, drift bound, holdouts, regressions, reward-hacking probes, authority effect, rollback plan, monitor window, deployment scope, evidence packets, residuals, and non-claims. PPO-like, DPO-like, GRPO-like, RLVR-like, context-policy, router-policy, or custom control-policy updates would be implementation choices under this record, not substitutes for governance.

A policy-update lease surface needs:

• A stack-wide policy registry covering planner decomposition, VCM retrieval, router selection, verifier escalation, execution/tool use, generator behavior, generation-mode choice, and governance assistance.
• Reward-source admission gates that classify human preference, exact verifier output, process rewards, benchmark results, tool outcomes, context-grounding checks, cost signals, and governance checks before training begins.
• Canary, holdout, and regression suites for each policy layer, including explicit probes for reward hacking, verifier gaming, hidden shortcuts, latency-induced verification loss, and context-grounding collapse.
• Update leases that expire, roll back, or remain experimental unless SCF and governance gates accept the evidence.
• Evidence packets that separate training success, evaluation success, governance approval, deployment monitoring, and long-term safety evidence.
• Failure closure that blocks reward laundering: proxy improvement cannot become task improvement unless target-specific evaluations and residuals support that transition.

At the governed learning endpoint, learning would be continuous but never sovereign. The system could improve its policies while preserving authority conservation, rollback, evidence review, and human/governance control. The learning actuator remains a target architecture until local policy-update harnesses, reward-quality studies, rollout records, rollback dry runs, and accepted evidence transitions show that improvement is governed instead of merely optimized.

79.14 Codex test plan

Test	Purpose	Status
Policy optimization record fixture validation	Validate that a policy update records update state, target layer, policy ref, policy delta summary, training mode, feedback source/admissibility, reward signal and boundary, verifier refs, reward-hacking probes, update constraint, drift bound, holdouts, regressions, evaluation refs, governance gates, authority effect, rollback, monitor window, evidence packet refs, deployment scope, promotion decision, measurement status, support-state effect, residuals, and non-claims.	implemented; passing via python3 scripts/validate_protocol_examples.py
DPO/offline preference baseline test	Establish a minimal static-preference update path and record its non-claims, holdouts, and preference-data limits.	planned; not run
PPO online RL baseline test	Establish a small online reward-update path with explicit drift bounds, reward refs, and baseline comparison.	planned; not run
GRPO/RLOO toy math reward test	Check whether group-relative or leave-one-out sampling improves a toy verifiable task without unstated reward assumptions.	planned; not run
Verifier reward loop test	Use exact or tool verifiers to record reward quality, false positives, false negatives, and residuals.	planned; not run
Length and latency penalty study	Test shorter reasoning or faster routing only when correctness, grounding, and verification adequacy are preserved.	planned; not run
Reward-source admissibility test	Check that unverified reward sources cannot promote a policy update without evaluation and governance refs.	implemented by benchmark anti-Goodhart harness for synthetic admissibility/evidence/gate checks; no policy optimization run
Reward hacking probe test	Check whether a candidate reward can be exploited without real task success.	implemented by benchmark anti-Goodhart harness for synthetic reward-probe and reward-as-truth rejection; no reward-hacking experiment run
Holdout and contamination negative case	Prove that a finite promotion candidate missing holdout refs or contamination checks rejects the policy-promotion evidence predicate.	implemented in AsiStackProofs.PolicyOptimization.promotion_candidate_missing_holdout_or_contamination_check_rejected; no hidden-holdout operation or contamination detector run
Reward-proxy target-evaluation negative case	Prove that a finite reward-proxy improvement used as sole evidence cannot satisfy promotion validity without target evaluation refs.	implemented in AsiStackProofs.PolicyOptimization.reward_proxy_without_target_evaluation_rejected; no reward-quality or task-success result promoted
Authority-expansion negative case	Prove that a finite authority-expanding policy update cannot satisfy validity without governance approval and rollback records.	implemented in AsiStackProofs.PolicyOptimization.authority_expanding_policy_update_without_approval_or_rollback_rejected; no deployed approval or rollback behavior claim
Policy-promotion route negative case	Prove that finite policy-promotion routes reject or route updates with inadmissible feedback, missing target evaluation, missing holdout or contamination checks, missing reward-hacking probes, governance or authority gaps, missing rollback, or regression/residual gaps before promotion.	implemented in AsiStackProofs.PolicyOptimization.PolicyUpdatePromotionRouteFor route theorems; no optimizer, reward-quality, policy-improvement, or deployment behavior claim
Router policy toy RL test	Train or simulate a small router-policy update with holdouts, baseline, latency, and authority constraints.	planned; not run
Context-policy grounding reward test	Check whether reward design improves source-grounded context selection without context bloat.	planned; not run
Reasoning-budget penalty test	Check whether shorter reasoning preserves correctness and verification adequacy.	planned; not run
Rollback and promotion gate test	Check that failed evaluation or governance gates produce rollback/quarantine rather than promotion.	implemented by benchmark anti-Goodhart harness for synthetic blocked-ratchet promotion rejection; no live rollback or policy canary run
Policy update lease probe	Check a deterministic synthetic router-policy canary lease with holdouts, contamination check, reward-hacking probes, unchanged authority, dry-run rollback, expected-invalid controls, residuals, and non-claims.	implemented by python3 scripts/validate_policy_update_lease_probe.py over experiments/policy_update_lease/results/2026-07-02-local.json; no optimizer, deployed canary, live rollback, route-quality claim, or support-state promotion
Policy-optimization update-lifecycle refinement	Check exact update identity, full optimizer/checkpoint/RNG and descendant custody, independent evaluation, causal/forgetting/rollback gates, bounded consumption, and successor-version readmission.	implemented by python3 scripts/validate_policy_optimization_refinement.py; two composed Lean traces, inherited lease, all 63 routes, 73/73 route mutations, and 3/3 cross-stage mutations pass, support/effect none; no learning, reward-validity, causal, rollback-efficacy, deployment, or support claim

Only fixture-shape validation and finite record predicates are implemented. The remaining tests require executable training or simulation harnesses.

79.14.1 Formalization hooks

Tag	Module	Target	Status
lean:policy_optimization.update.operational_invariant	AsiStackProofs.PolicyOptimizationRefinement	A versioned governed-update lifecycle binds target, baseline, objective, data, feedback, full optimizer/checkpoint/RNG and descendant state, evaluation, adjudication, bounded consumption, rollback, and readmission without assigning support or an external effect.	implemented
lean:policy_optimization.reward_boundary.failure_blocks_promotion	AsiStackProofs.PolicyOptimizationRefinement	Missing target evaluation, causal ablation, reward-hacking probes, regression, forgetting, safety/rights, uncertainty, or independent evaluation blocks a policy update before bounded-lease adjudication.	implemented
lean:policy_optimization.promotion_route.failure_routes	AsiStackProofs.PolicyOptimizationRefinement	All finite update-lifecycle routes reject or redirect incomplete scope, state, update, evaluation, adjudication, lease, rollback, or readmission records before bounded use.	implemented
lean:policy_optimization.lease_probe_fixture_bridge	AsiStackProofs.PolicyOptimizationRefinement	The independently consumed six-sample/five-candidate lease probe and the reachable lifecycle preserve three rejected controls, experimental-only use, versioned readmission, and no support or external-effect authority.	implemented

The original finite contradictions and guard-priority routes remain useful, but the public targets now resolve to a reachable update lifecycle. Its combined Lean route theorem checks incomplete optimizer custody, missing target evaluation, noncausal gain, incomplete rollback, support-authority laundering, bounded-lease admission, and versioned readmission; the independent consumer covers every route and mutation. These checks prove only finite authored record and transition consequences. They do not prove reward quality, evaluator validity, RL convergence, benchmark improvement, causal policy improvement, retained capability, forgetting control, policy safety, route quality, reward-hacking resistance, hidden-holdout or contamination quality, approval quality, effect-complete runtime rollback, live monitoring, reproduction, or transfer.

79.14.2 Formal adequacy audit

The nineteen legacy declarations remain finite: five projections are retired, four contradictions and seven guard-priority routes remain useful, and three copied fixture normalizations are replaced by generated result conformance. The refinement adds one combined route theorem, one complete-cycle theorem, and one failed-evaluation/downstream-blocking theorem over a reachable seven-stage model, plus an independent 63-route consumer with 73 route and three cross- stage mutations. None models gradient estimation, optimizer convergence, learning dynamics, reward correctness, evaluator error, causal target improvement, catastrophic forgetting, full-state recovery, runtime enforcement, institutional independence, or transfer. Their authority ends at authored record and transition semantics.

79.15 Source-noted external literature

The following sources were added as stable external-literature records with primary arXiv metadata and conservative source notes. They remain method-family context. They do not promote the claim above argument, and none of their reported training results has been reproduced here.

Area	Source records	Current use
Trust-region / online RL	ext_trpo_2015, ext_ppo_2017	Explain bounded update-size discipline, proximal objectives, and why online rollout needs drift accounting.
Simpler or critic-light RLHF	ext_remax_2023, ext_reinforce_style_rlhf_2024	Compare PPO-heavy pipelines with simpler RLHF policy-gradient baselines.
Offline preference optimization	ext_dpo_2023, ext_ipo_preference_2023, ext_orpo_2024, ext_kto_2024, ext_simpo_2024	Explain preference, binary-feedback, reference-model, and sequence-reward assumptions.
Reasoning RL and sequence-level variants	ext_deepseek_r1_2025, ext_dapo_2025, ext_gspo_2025, ext_s_grpo_2025	Compare reasoning-RL, reproducibility pressure, sequence-level objectives, and overthinking control.
Verifier reward and long context	ext_longrlvr_2026	Connect long-context reward design to VCM grounding and context adequacy.
RLHF limitations	ext_rlhf_limitations_2023	Feed reward-hacking probes, evaluator-limit warnings, and disclosure/audit requirements.
Outcome-only population learning	ext_openai_es_2017, ext_eggroll_hyperscale_es_2026	Add black-box scalar-fitness optimization while preserving full rollout/population denominators, evaluator risk, and total cost.

79.16 A rationale reward is not trace faithfulness

Rewarding a plausible rationale can optimize the appearance of reasoning while leaving the answer on a shortcut path. ext_faithfulness_information_flow_2026 shows the sharper failure: visible-test reward can rise while a code-repair policy learns lookup-table behavior that its trace does not acknowledge. Some reported update-time interventions make that behavior more visible, but visibility is not elimination and the results remain bounded to the studied models and tasks.

Any policy update that uses reasoning traces therefore records separate rewards for task outcome, rationale quality, trace/action consistency, and bounded faithfulness diagnostics. It includes counterfactual trace and prompt controls, hidden tests, low-entropy diagnostic warnings, evaluator dependencies, and a rollback trigger for shortcut growth. No trace-derived reward may promote evidence, authority, or readiness by itself.

ext_faithfulness_information_flow_2026 | Metadata-first comparator: Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning. Primary 2026 comparator that separates chain-of-thought sufficiency, completeness, and interventional necessity, demonstrates prompt-to-answer shortcuts and transparent reward-hacking diagnostics, and documents low-entropy and reference-model limits. It does not make a reasoning transcript an authoritative receipt or establish local monitorability. | No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row. |

79.17 Policy Updating Is One Persistence Choice

A reward, preference, verifier score, or eligible lesson does not determine that a policy or parameter update is the right durable response. The Adaptive Commit Boundary compares that route with narrower context, memory, procedure, tool, router, evaluator, environmental, and institutional interventions. It passes a policy-update disposition forward only with its evidence boundary, commitment profile, authority ceiling, qualification obligations, monitor, rollback state, descendants, expiry, and non-claims intact.

This does not weaken the Governed Policy Update Lease. It explains how a lesson reaches that lease without implementation convenience deciding the locus. A trained checkpoint remains a realization; it becomes qualified only after the policy chapter’s target evaluations, retained-capability checks, causal probes, authority review, effect-complete rollback, and monitoring obligations pass.

79.18 Source crosswalk

Source ID	Title	Layer	Planned use	Readiness
benchmaxxing	Benchmaxxing: The Performance Ratchet	benchmarks_evidence	Benchmark pressure, saturation, residual escrow, anti-Goodhart boundaries, and regression floors that become reward/evaluation inputs.	source note available; local raw cache available
rmi	Ratcheting Modular Intelligence	capability_ratchet	Capability ratchet framing for turning pressure into modular improvement without erasing residuals.	source note available; local raw cache available
spinoza	Proof of Belief / The Spinoza Architecture	reasoning_epistemology	Verification and claim-tier discipline for reward meaning, proof-like claims, and failed verifier consequences.	source note available; local raw cache available
verification_bandwidth	Verification Bandwidth in Bounded Contexts	context_verification_theory	Boundary between more context, adequate context, verification capacity, and reward design.	source note available; local raw cache available
talos	Talos Protocol	labor_execution_os	Execution ledgers, typed jobs, tool results, and audit logs as feedback/evaluation artifacts.	source note available; local raw cache available
vcm_public	Virtual_Context_Memory_v1	memory_context	Context-packet and grounding policy boundary for context-selection rewards.	source note available; local raw cache available
planforge	PlanForge	planning_control	Planner policy, decomposition rewards, tool-choice rewards, and stopping-condition feedback.	source note available; local raw cache available
octopus_router	Octopus Router Architecture	routing_modular_intelligence	Router-policy and specialist-choice reward interface.	source note available; local raw cache available
scf	Stable Capability Fields	governance_recursive_self_improvement	Promotion, rollback, authority, and replacement gates for policy updates.	source note available; local raw cache available
tokenmana	TokenMana	resource_economics	Token, latency, load, and resource reward components without collapsing them into quality claims.	source note available; local raw cache available
cognitive_loop_closure	Cognitive Loop Closure	procedural_memory	Toolification and procedural-memory feedback loops for repeated successful trajectories.	source note available; local raw cache available
moecot	MoECOT-Agent Architecture Whitepaper	implementation_reference	Multi-core routing and runtime implementation-reference context; not reproduced here.	source note available; connector or recovery required
portia_synapse	PortiaSynapse	routing_training_and_dkl_navigation	Evidence-gated phase training, gradient and activation diagnostics, confidence calibration, component admission, and rollback for a TreeLLM/DKL navigator.	source note available; cache status: cached_existing
spider_synapse	SpiderSynapse	routing_training_and_dkl_navigation	Preserved negative result and recovery design for diagnosing branch, refinement, memory, selector, and supervision failures before architectural expansion.	source note available; cache status: cached_existing

External method-family notes now cover the initial policy-optimization literature queue. They are intentionally separate from local evidence: source-note coverage does not imply a trainer, reward model, preference dataset, RLVR benchmark, reasoning-budget study, or policy update exists in this repository.

79.18.1 Manifest source assignment reconciliation

These rows keep Policy Optimization and Learning from Feedback’s manifest assignments visible at their recorded review boundary. Passage review does not establish local reproduction, performance, safety, deployment, or support-state movement.

Source	Intake role	Boundary
learning_compute_topology	Passage-reviewed comparator: Learning–Compute Topology: Formalizing the Causal Organization of Adaptive Systems. Corben-authored August 2026 research paper and executable preparation package that separates model architecture, learning-process topology, execution topology, and physical compute topology. It contributes adaptive-identity tests; typed evidence, judgement, credit, state, artifact, control, and authority relations; LCT-IR; Learning Causal Normal Form; seven bounded propositions; topology metrics; a semantic compiler firewall; Adaptive Branch–Validate–Integrate; toy and analytical phase diagrams; and an explicit falsification program. The bundled reference implementation passes 11 unit tests, but implements only bounded conformance behavior and does not establish neural-training benefit, causal completeness, universal canonicality, safety, scaling superiority, or ASI.	The formal propositions hold only under their stated finite, explicit-state, interface-sufficiency, information-theoretic, and cut-capacity assumptions. The executable supplement covers a bounded IR/validator/normalizer/compiler/simulator slice; the phase diagrams are toy or analytical, the ABVI topology is proposed, and the novelty matrix is a scoped comparison rather than a global novelty proof. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_trpo_2015	Passage-reviewed comparator: Trust Region Policy Optimization. Supports the external method-family comparison for trust-region bounded policy updates and drift constraints.	External method-family context only; does not support the ASI Stack governance claim by itself and does not report local planner, router, context-policy, or model training. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_ppo_2017	Passage-reviewed comparator: Proximal Policy Optimization Algorithms. Supports the external method-family comparison for PPO-style online policy-gradient updates and proximal surrogate objectives.	External method-family context only; does not validate reward quality, local PPO runs, planner/router/context-policy training, or deployment readiness. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_remax_2023	Passage-reviewed comparator: ReMax: A Simple, Effective, and Efficient Reinforcement Learning Method for Aligning Large Language Models. Supports the external method-family comparison for simpler RLHF-oriented policy-gradient updates relative to PPO complexity.	External method-family context only; does not prove local RLHF efficiency, reward quality, stack-policy improvement, or universal PPO replacement. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_dpo_2023	Passage-reviewed comparator: Direct Preference Optimization: Your Language Model is Secretly a Reward Model. Supports the external method-family comparison for offline preference optimization from pairwise preference data.	External method-family context only; does not validate local preference data, DPO training, preference quality, reward correctness, or governance-safe behavior. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_ipo_preference_2023	Passage-reviewed comparator: A General Theoretical Paradigm to Understand Learning from Human Preferences. Supports external framing of preference-learning assumptions, including pairwise preferences, pointwise reward approximations, and policy-distribution shift.	External theoretical context only; no local preference-learning theorem, reward-model validation, preference experiment, or ASI Stack policy update is recorded. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_orpo_2024	Passage-reviewed comparator: ORPO: Monolithic Preference Optimization without Reference Model. Supports the external method-family comparison for reference-model-free monolithic preference optimization.	External method-family context only; does not prove local ORPO training, alignment improvement, truthfulness, tool safety, or governance compliance. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_kto_2024	Passage-reviewed comparator: KTO: Model Alignment as Prospect Theoretic Optimization. Supports the external method-family comparison for prospect-theoretic or human-aware loss objectives and binary feedback framing.	External method-family context only; does not prove a correct human model, reward validity, preference-data validity, local policy safety, or governance compliance. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_simpo_2024	Passage-reviewed comparator: SimPO: Simple Preference Optimization with a Reference-Free Reward. Supports the external method-family comparison for simple reference-free sequence-level preference optimization.	External method-family context only; does not prove local SimPO performance, preference-quality robustness, truthfulness, tool safety, or governed deployment readiness. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_reinforce_style_rlhf_2024	Passage-reviewed comparator: Back to Basics: Revisiting REINFORCE Style Optimization for Learning from Human Feedback in LLMs. Supports the external method-family comparison for simpler REINFORCE-style RLHF baselines and PPO-cost/tuning cautions.	External method-family context only; does not show that the repo has implemented REINFORCE, RLOO, RLHF training, reward modeling, or policy-safety evaluation. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_deepseek_r1_2025	Passage-reviewed comparator: DeepSeek-R1: Incentivizing Reasoning Capability in LLMs via Reinforcement Learning. Supports external comparison for reasoning-RL pressure and the distinction between reasoning behavior, reward design, and benchmark reporting.	External method-family context only; does not import or reproduce DeepSeek-R1 models, training artifacts, evaluations, benchmark results, model behavior, or governance claims. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_dapo_2025	Passage-reviewed comparator: DAPO: An Open-Source LLM Reinforcement Learning System at Scale. Supports external comparison for open-source reasoning-RL systems, decoupled clipping, dynamic sampling, and reproducibility pressure.	External method-family context only; does not mean DAPO has been run, audited, imported, or reproduced in this repository. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_gspo_2025	Passage-reviewed comparator: Group Sequence Policy Optimization. Supports external comparison for sequence-level group policy optimization and sequence-level clipping/reward design.	External method-family context only; does not validate local sequence-level RL, reasoning-budget policy, model-quality improvement, or reward robustness. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_s_grpo_2025	Passage-reviewed comparator: S-GRPO: Early Exit via Reinforcement Learning in Reasoning Models. Supports external comparison for early-exit/reasoning-budget RL and overthinking control.	External method-family context only; does not prove shorter reasoning is better, local reasoning-budget policies exist, or verification adequacy is preserved under early exit. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_longrlvr_2026	Passage-reviewed comparator: LongRLVR: Long-Context Reinforcement Learning Requires Verifiable Context Rewards. Supports external comparison for long-context RLVR and verifiable context-grounding rewards relevant to VCM policy learning.	External method-family context only; does not reproduce LongRLVR, context-grounding reward experiments, VCM context-policy learning, or reward-quality evidence. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_rlhf_limitations_2023	Passage-reviewed comparator: Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback. Supports external risk framing for RLHF limitations, reward hacking, evaluator limits, and complementary safeguards.	External survey/risk framing only; not proof of any local mitigation, reward-process safety, evaluator quality, or deployed safeguard. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_muse_unlearning_2025	Passage-reviewed comparator: MUSE: Machine Unlearning Six-Way Evaluation for Language Models. Supports separating memorization, privacy leakage, retained utility, removal-scale behavior, and sequential sustainability instead of using one forgetting score.	No MUSE corpus, method, 7B model, privacy probe, scale test, or sequential deletion request was reproduced by the local small policy network. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_faithfulness_information_flow_2026	Passage-reviewed comparator: Faithfulness as Information Flow: Evaluating and Training Faithful Chain-of-Thought Reasoning. Shows that reward improvement can coexist with shortcut use and under-verbalized reward hacking, and tests update-time interventions that change information flow without changing rollout rewards.	The intervention results are bounded to the reported arithmetic, code-repair, and DAPO-Math settings; they do not validate local policy optimization or eliminate reward hacking. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
regret_engine	Passage-reviewed Corben architecture source: The Regret Engine: Governed Counterfactual Learning Signals for Continual Adaptation, Prospective Risk Control, and Self-Correction in Artificial Agents. Corben-authored August 2026 conceptual architecture and research program for decision-time-fair Governed Counterfactual Regret, immutable Decision Capsules, admissible comparator contracts, sparse Regret Tensors, append-only Regret Packets, prospective regret control, regret-aware replay, regret-to-rule compilation, three update clocks, root-cause adjudication, and bounded update leases. Existing chapters are upgraded first; no implementation, experiment, reproduction, causal-identification result, formal proof, safety result, support transition, SOTA, AGI, or ASI is inferred.	The bibliography and Markdown figure companions were not supplied; the DOCX embeds its visual material. All propositions, algorithms, experiments, and architecture claims remain proposed rather than independently validated. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_eggroll_hyperscale_es_2026	Passage-reviewed comparator: Evolution Strategies at the Hyperscale. Provides a batched low-rank evolution-strategy case for optimizing outcome-level, discrete, and nondifferentiable fitness without action-level gradients.	Outcome-only fitness remains gameable and source-reported reasoning/control results do not validate local policy improvement, evaluator quality, safety, or total efficiency. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_openai_es_2017	Passage-reviewed comparator: Evolution Strategies as a Scalable Alternative to Reinforcement Learning. Provides a foundational outcome-only parameter-search comparator to temporal policy-gradient credit assignment.	The studied control tasks and distributed scaling do not establish universal policy quality, sample efficiency, or governance safety. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
assurance_shift_learning	Passage-reviewed comparator: When Success Stops Teaching: Assurance-Shift Learning and Governed Residual Boundary Learning for Mature AI Systems. Adds the positive-support prerequisite, outcome/process separation, evaluator-first rule, and governed negative-pressure boundary before an informative exception can enter an update lease.	No policy update, anti-tampering effect, or superiority of negative feedback was demonstrated. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
adjudicated_persistence	Passage-reviewed comparator: Adjudicated Persistence: Governing the Transition from Experience to Durable Structure in Adaptive Systems. Frames a policy update as one candidate persistence locus selected only after learning eligibility, cross-surface comparison, commitment matching, and explicit authority.	Conceptual author framework and benchmark proposal; no local implementation, empirical result, independently checked proof, safety result, or support movement. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.

79.19 Post-v2 real policy-update result

Every bounded fine-tune, regularized challenger, and deletion-aware retrain in the three-seed campaign had a nonzero parameter delta and changed held-out decisions; every no-update arm remained bit-identical. Mean final test accuracy moved only from 0.7722 for no update to 0.7792, 0.7778, and 0.7778 for the three challengers. Bounded and regularized updates slightly reduced retained-base accuracy, and validation-selected best checkpoints disagreed with final checkpoints on 62 test decisions across challenger runs.

This establishes local checkpoint/output causality and preserves modest gains, forgetting, and best/final disagreement. It does not test human or model feedback, reward hacking, deployment canaries, or production optimization. The broad core claim remains argument via no_change.

79.20 Post-v2.1 prospective update authority

The successor campaign prospectively selects the best validation-eligible checkpoint subject to a retained-task bound, always preserves the final checkpoint, and forbids test outcomes from choosing either. Six of fifteen seed-arm transactions expose best/final disagreement; three authorized-data comparators cross the safety bound after one epoch and remain ineligible. None of nine eligible challengers reaches the 0.05 target-utility gain, although all 15 declared state trees roll back exactly. This is evidence that update authority, utility, retained behavior, and recovery can be measured in one transaction. It is not feedback learning, reward-model validation, policy improvement, or deployment evidence, so the core disposition remains no_change.

79.21 Summary

Policy optimization is the learning actuator. Benchmarks and verifiers create pressure, policy optimization proposes behavior changes, and SCF governance decides whether those changes survive.

The boundary is strict: a reward signal is not a truth signal, a training run is not a governance grant, and a benchmark improvement is not a deployment right. The system may learn from feedback only through recorded reward sources, update constraints, evaluations, residuals, rollback, and governance gates.

That learning discipline applies to durable projects. A steward agent may coordinate work and preserve memory, but it cannot turn feedback loops into ownership, treasury authority, or silent governance change. The practical value of PPO, DPO-style methods, verifier rewards, RLVR, GRPO-like families, and simpler policy-gradient variants is not that one objective solves alignment. Their value is that they give the stack a menu of update mechanisms whose evidence, reward source, failure mode, and deployment boundary can be recorded. Learning becomes acceptable only when the update has a field identity, rollback path, evaluator independence, and residual ledger. Otherwise optimization merely accelerates whatever the system already fails to see.

79.22 Evidence reconciliation (2026-07-16)

The invariant protocol, field meanings, and inference limits are stated once in Living Book Methodology. This packet contains only the chapter-specific projection; its authoritative per-atom rows are the policy-optimization-and-learning-from-feedback slice of experiments/claim_family_terminal_coverage/results/result.json.

The core remains blocked after full attempt at argument support. The strongest family attempt was Full-state update and unlearning causal campaign. Its exact boundary is: Broad unlearning claim is narrowed with no support promotion; behavioral removal is not influence, privacy, legal, storage, backup, or descendant erasure. Across 75 atoms, the terminal ledger records 75 blocked_after_full_attempt.

Chapter-specific field	Value
Family / atom denominator	CF-07 / 75 atoms
Terminal dispositions	75 blocked_after_full_attempt
Core	policy-optimization-and-learning-from-feedback.core: blocked_after_full_attempt at argument
Core attempted / missing lanes	causal, empirical, executable, formal, source-synthesis / normative, transfer
Attempted local lanes	causal, empirical, executable, formal, source-synthesis
Missing or unproved lanes	normative, transfer
Strongest family bundle	Full-state update and unlearning causal campaign (end_to_end): One adequate five-seed, seven-arm terminal campaign preserving two prior instrument failures and separate behavioral, influence, privacy, lineage, storage, backup, and descendant axes.
Negative controls	deletion retrain comparator; approximate mitigation arms; 15 rejecting mutations; failure-lineage preservation.
Accepted transitions	none
Maximum inference	Broad unlearning claim is narrowed with no support promotion; behavioral removal is not influence, privacy, legal, storage, backup, or descendant erasure.
Reproduction / next burden	Replay scripts/validate_p4_m7_update_unlearning_v3.py and scripts/validate_claim_family_terminal_program.py; fill the named atom-specific lanes under a new prospective protocol.

79.22.1 Semantic proof ceiling

The P4-C4 audit finds AsiStackProofs.PolicyOptimizationRefinement adequate only for exact bounded update-lease and readmission semantics. Its four public targets and five declarations constrain authored records; they do not establish optimizer correctness, reward validity, actual learning, causal policy improvement, forgetting control, live monitoring, rollback efficacy, deployment, transfer, or support movement.

79.23 Handoff

Policy updates decide how behavior responds to feedback, but they still require a governed account of the feedback and training material that enters the loop. PPO, DPO, GRPO, and related names identify feedback objectives or policy-update procedures; they do not fully specify the parameter optimizer. Governed Model Training owns whether AdamW, Muon, Shampoo, Sophia, or another exact Optimizer Policy Card faithfully executes the authorized update and resumes with complete state. This chapter retains authority over what behavior is optimized and why; neither owner may silently absorb the other. Data Engines, Continual Learning, and Unlearning follows the learning actuator with that lifecycle: provenance, split exclusions, synthetic lineage, retention, deletion scope, contamination controls, residuals, and the boundary between a receipt and a capability result. It provides the data authority that policy optimization must not manufacture for itself before the book turns to the longer-lived stewardship of projects and artifacts.