Skip to main content

55  Governed Model Training, Distributed Optimization, and Scaling

55.1 Chapter status

Field	Value
Chapter ID	governed-model-training-distributed-optimization-and-scaling
Part	Part III - Routing, Compression, Representation, and Substrates
Status	conceptual
Manuscript maturity	integrated argument chapter; formal/protocol contract complete; optimizer and maintenance-learning integrated
Last updated	2026-07-31
Primary source records	BeastBrain maintenance-learning lineage, seven distributed-training/checkpoint records, and fifteen optimizer records spanning Adam/AdamW, failure variants, memory/layer scaling, tensor and curvature preconditioning, Lion, Sophia, SOAP, schedule-free methods, maximal-update and modular-norm transfer, and Muon.
Local implementation reference	corbens_trainer_project supplies a pinned failure-bearing trainer/control-plane comparator; it is not a reproduced training result.
Claim label	Design rationale
Evidence level	argument
Source loading state	source notes: learning_compute_topology, beastbrain, ext_llama3_herd_2024, ext_megatron_distributed_training_2021, ext_zero_optimizer_2019, ext_gspmd_2021, ext_datastates_llm_2024, ext_pytorch_distributed_checkpoint_2026, ext_mlperf_training_v6_2026, ext_adam_2015, ext_amsgrad_2018, ext_adamw_2019, ext_adafactor_2018, ext_lamb_2019, ext_shampoo_2018, ext_kfac_2015, ext_lion_2023, ext_sophia_2023, ext_soap_2024, ext_schedule_free_2024, ext_mup_2022, ext_modular_norm_2024, ext_muon_scalable_2025, ext_muon_spectral_norm_2026, corbens_trainer_project, ext_proof_of_learning_2021, ext_test_time_training_2020, ext_curriculum_learning_2009, ext_kimi_k3_2026, ext_eggroll_hyperscale_es_2026, ext_openai_es_2017, ext_mezo_2023, ext_forward_forward_2022; raw cache: beastbrain
Test state	One authored transaction fixture, a JSON Schema, and an independent semantic validator check 21 transaction mutations, seven weight-only omitted-field collisions, and nine complete-encoding mutations. The natural five-arm campaign is prospectively specified and unexecuted.
Formal state	Three public targets are implemented in AsiStackProofs.GovernedModelTraining through 20 theorem declarations. Their ceiling is finite record-route and checkpoint-information semantics.

55.2 Drafting guardrail

This chapter governs the identity and faithful execution of a training run. It does not claim that larger training is better, that one optimizer or parallelism or optimizer method wins, that the recorded state inventory is complete for every framework, or that a checkpoint which loads continues the same learning trajectory. The sources establish mechanisms, reported cases, concrete interfaces, and measurement comparators. None supplies local evidence.

The chapter remains Design rationale + argument. Its authored positive control is deliberately incapable of becoming training evidence: no model is trained, its resume paths are declarations, and its expected route authorizes only handoff to a separate qualification process. The prospective campaign must pass competence gates before any negative result can narrow the architecture.

55.3 Human Reading Path

Concrete lens. The checkpoint-file baseline treats successful loading as recovery. The full-state contract binds optimizer, scheduler, sampler, RNG, scaler, cache, and topology state.

A model checkpoint is the visible residue of a larger event. Data is ordered, gradients are distributed, numbers rounded, state sharded, workers fail, checkpoints written, and operators choose which artifact to keep. Two files can contain plausible weights while coming from meaningfully different processes.

That matters because the process can silently change the product. A restarted run may repeat examples, skip examples, lose optimizer momentum, reset a scheduler, change random choices, or use a different numerical reduction order. A faster topology may reach a different result. A “best checkpoint” may be best only because the test set was consulted during selection. A successful final loss can hide discarded failed attempts and enormous recovery cost.

The governing idea is simple: treat a training run as a transaction. Freeze what the run is supposed to do, record what actually happened, checkpoint all state required by the declared equivalence class, retain failures and candidate checkpoints, and hand the selected artifact to an independent qualification stage. Training may create a candidate, but it cannot grade that candidate into readiness or release.

55.4 Problem

“Train model M on data D” is not an executable identity. It omits the data lease and ordering, objective, optimizer, scheduler, numerical policy, random state, global and microbatch semantics, accumulation, device mesh, parallelism dimensions, collective implementation, compiler plan, checkpoint protocol, fault policy, stopping rule, candidate selection rule, and evaluator boundary. At scale, each omission can change either the learned artifact or the cost and failure denominator used to justify it.

Distributed optimization makes the omissions relational. Tensor, pipeline, context, expert, and data parallelism move different tensors and synchronize at different times. Sharding can distribute parameters, gradients, optimizer moments, master weights, and activations across ranks. Pipeline schedules change bubbles and operation order. Compiler systems can infer resharding and collectives not apparent in the model source. Mixed precision adds master parameters, gradient-accumulation types, loss scaling, overflow behavior, and reduction precision. “Same code” does not imply the same computation.

Failure is normal enough to be part of the protocol. In its 54-day reported snapshot, the Llama 3 paper records hundreds of interruptions and explains why synchronous training can restart after a single device failure. The lesson is not that this particular system is reliable; it is that attempted runs, interruptions, manual interventions, checkpoint stalls, lost work, and recovery must be counted alongside useful training time.

Checkpointing is also an asynchronous distributed operation. A save call can stage data while persistence remains incomplete. Shards can represent different logical steps. A checkpoint can preserve model and optimizer tensors while omitting scheduler, random, scaler, sampler, data cursor, topology, or compiler state. It can load successfully and still duplicate or skip training examples. Recovery therefore needs a declared equivalence class:

• bitwise equivalence requires exact bytes and operation behavior;
• operation-order-bounded equivalence permits declared nondeterministic ordering but bounds state and trajectory differences; and
• statistical equivalence supports only a distributional claim across prospectively repeated runs.

These claims cannot be substituted for one another after outcomes are visible.

55.4.1 Exclusive job and adjacent boundaries

Governed Model Training owns the transaction from prospectively frozen training inputs to a selected, identity-bound candidate handed to qualification. Its primary question is: is this checkpoint the product of the run we claim occurred?

Adjacent owner	That owner keeps	Governed Model Training owns
Replaceable Cognitive Substrates	Which architecture or kernel may be proposed, compared, shadowed, promoted, migrated, or retired.	Faithful execution of the selected architecture’s training contract.
Data Engines, Continual Learning, and Unlearning	Datum admission, consent, rights, lineage, descendant obligations, deletion, and unlearning.	The exact leased data snapshot, order, cursor, multiplicity, and run-time use receipt.
Policy Optimization and Learning from Feedback	Which behavior-update target, feedback, reward, causal evaluation, bounded lease, and rollback may govern a policy update.	Base or update run execution, full training state, distributed recovery, and checkpoint-family selection.
AI Supply-Chain Integrity and Lifecycle Provenance	Supplier, code, build, dependency, artifact, signature, and lifecycle provenance.	The dynamic execution ledger joining those frozen inputs to one training outcome.
Resource Economics and Token Budgets	Whether compute, memory, storage, network, energy, verification, and human capacity should be allocated.	Whether allocated resources and topology were actually used as declared, with complete failed-run and recovery cost.
Model-Weight Custody and Hardware Roots of Trust	Who may hold, transfer, load, use, derive, revoke, or release model-family artifacts.	Creation-time state closure, checkpoint commit, resume, candidate selection, and qualification handoff.
Benchmark Ratchets and Anti-Goodhart Evidence	Qualification workloads, evaluator independence, benchmark validity, and evidence transitions.	Preserving that qualification remains unopened and independent during checkpoint selection.
Readiness Gates	Admission, quarantine, residual review, and deployment authority.	A bounded candidate handoff with no readiness or release authority.

The terminal structural decision is admit at argument support. Returning this material to the neighbors would leave no owner for distributed run identity, topology/numerical drift, full-state recovery, failure denominators, and checkpoint-family selection as one transaction. The chapter is placed after Replaceable Cognitive Substrates: architecture selection precedes training-run execution, while readiness and qualification remain downstream.

flowchart LR
  A["Frozen architecture, data lease, objective, and run contract"] --> B["Declared topology, numerical policy, and full state inventory"]
  B --> C["Distributed execution with every attempt and fault retained"]
  C --> D{"Checkpoint family committed consistently?"}
  D -- "no" --> Q["Quarantine, residual ownership, and bounded recovery"]
  D -- "yes" --> E{"Resume satisfies the prospectively chosen equivalence class?"}
  E -- "no" --> Q
  E -- "yes" --> F["Validation-only candidate selection"]
  F --> G["Identity-bound candidate handoff"]
  G --> H{"Independent unopened qualification"}
  H -- "not yet" --> I["Candidate only: no readiness or release authority"]
  H -- "evaluated later" --> J["Separate evidence transition and readiness decision"]
  Q --> K["Retain failed attempts, lost work, operator cost, and affected descendants"]

What this governed training transaction diagram shows: The path begins with frozen inputs and does not end at a checkpoint file. Distributed execution, faults, commit consistency, resume equivalence, candidate-family selection, and the independent qualification boundary remain separate decisions. Any failed or unopened gate keeps the artifact a candidate and conserves the failure and cost record; no training-stage node grants readiness or release authority.

55.5 Why existing approaches are insufficient

A loss curve is necessary but not a run receipt. Different data orders, precision policies, collective schedules, or recovery points can produce similar aggregate loss while changing individual examples, downstream behavior, or stability. Final convergence cannot reveal which failures were censored or which candidate checkpoints were discarded.

Framework checkpoint APIs are necessary but application-incomplete. PyTorch’s distributed checkpoint interface makes canonical names, collective participation, asynchronous completion, resharding, and strict loading visible. Those guarantees do not automatically include every scheduler, sampler, random generator, scaler, data cursor, compiler artifact, cache, or external state the application needs. “Library save succeeded” and “the run resumed faithfully” are different propositions.

Scaling efficiency is not training integrity. Megatron-style composed parallelism, ZeRO-style state sharding, and GSPMD-style compiler partitioning are serious alternatives with different communication, memory, control, and portability profiles. A fair comparison freezes model, data, objective, quality target, hardware envelope, tuning opportunity, failures, and accounting. Peak FLOPs, maximum parameter capacity, or one provider’s best configuration cannot choose a universal topology.

Time-to-quality is stronger than step throughput but still incomplete. MLPerf Training demonstrates useful discipline through fixed targets, rules, repetitions, system metadata, and result corrections. A governed run additionally needs data-rights identity, complete attempted-run and fault denominators, checkpoint selection separation, recovery equivalence, operator burden, and a qualification handoff that does not turn benchmark success into release.

55.5.1 Strongest objection

The strongest objection is that this owner merely adds paperwork around standard trainer state. That objection wins if the records do not alter a decision. The architecture is useful only if it catches real silent errors that ordinary logs miss, preserves better recovery without unacceptable cost, and supports independent replay. That is why the natural campaign includes strong standard-checkpoint, weight-only, uninterrupted, and alternative-topology comparators, concrete fault injection, hidden qualification, and joint cost.

55.6 Core Claim

[governed-model-training-distributed-optimization-and-scaling.core, label: Design rationale, support: argument] A model-training candidate is eligible for qualification only when a prospectively frozen run contract binds architecture, data lease and order, objective, optimizer, scheduler, numerical policy, device and parallelism topology, code and environment, budget, stopping and fault policy, complete attempted-run denominator, full declared checkpoint state, commit consistency, resume equivalence class, candidate-checkpoint family, validation-only selection, independent unopened qualification, and residual ownership; a loss reduction, completed job, high utilization, checkpoint file, successful load, recovered run, selected candidate, formal record proof, or source-reported scale result alone establishes neither faithful training, model quality, optimizer superiority, fault tolerance, safety, support, readiness, release, transfer, nor SOTA.

Reader claim. A checkpoint is not a faithful training restart unless it restores every state variable that can change the future run, not just model weights.

Operational rule. Freeze architecture, data order, objective, optimizer, scheduler, numerical policy, topology, code, environment, and stopping before training. Checkpoint model, optimizer, scheduler, scaler, RNG, sampler, cache, and fault state atomically; test resume equivalence and keep every attempted run and candidate in the denominator.

55.6.1 Worked resume: weights match, the run diverges

A distributed training job saves model weights at step 10,000 and restarts after a worker failure. The optimizer moments, scheduler position, data-sampler cursor, and per-worker RNG states were not committed with the weights. Loading succeeds and the first loss looks plausible, but batches arrive in a new order, dropout masks differ, and the learning rate repeats an earlier segment. The artifact is a warm start, not an equivalent resume.

The run record keeps both trajectories, marks the resume class non-equivalent, and prevents the restarted checkpoint from being compared as though it continued the same experiment. A faithful path needs one atomic commit across full declared state plus an equivalence test under the prospectively chosen tolerance. Even then, qualification must use an independently unopened set rather than validation-selected checkpoints. This contract does not establish optimizer superiority, distributed fault tolerance, model quality, or scaling behavior.

55.7 Mechanism

55.7.1 The training-run transaction

The transaction has nine stages.

1. Freeze identity. Name the architecture, exact data lease, objective, code, environment, optimizer, scheduler, numerical policy, seeds, topology, resources, stopping rule, fault policy, selection rule, and nonclaims.
2. Compile topology. Record device mesh, every parallelism dimension, shard placement, compiler or planner, collectives, batch arithmetic, synchronization, and expected communication and memory envelope.
3. Admit state. Enumerate parameters, gradients where required, optimizer, scheduler, scaler, CPU and device RNGs, sampler, data cursor, topology, compiler plan, and any architecture-specific mutable state.
4. Execute visibly. Commit batch identity and multiplicity, step identity, heartbeats, loss and gradient diagnostics, overflows, stalls, retries, topology changes, operator interventions, and resources without erasing failed attempts.
5. Checkpoint transactionally. Distinguish request, staged copy, durable commit, shard validation, completeness validation, and resume authority.
6. Inject and recover. Exercise worker, collective, data-order, numerical, and checkpoint failures against uninterrupted and deliberately incomplete controls.
7. Test resume. Compare state and subsequent trajectory under the frozen bitwise, operation-order-bounded, or statistical equivalence class.
8. Select without qualification leakage. Retain all candidates, failed runs, best and final checkpoints, and configuration searches; select only on the declared validation rule.
9. Hand off, do not promote. Give one exact candidate plus the complete run record to an independent qualification owner. Training integrity does not confer model merit or deployment authority.

55.7.2 The Developmental Intelligence Loop

A governed training run explains how one candidate was produced, but not how a system develops progressively more reusable competence. The book’s learning owners therefore compose into one Developmental Intelligence Loop:

curriculum construction -> world interaction -> prediction error -> representation and abstraction formation -> causal intervention -> memory consolidation -> proceduralization -> skill composition -> quiescent stabilization -> readiness review -> authority-gated promotion.

Each arrow changes the object and the authority available to it. Curriculum construction chooses experience without treating selection as learning. World interaction produces observations rather than ground truth. Prediction error may trigger representation change, but correlation alone cannot become a causal belief. Interventions distinguish hypotheses where the domain permits them. Durable memory consolidates qualified propositions and uncertainty; procedural memory converts repeatedly successful, inspectable traces into candidate skills. Skill composition creates a larger candidate whose failure surface may exceed those of its parts. Quiescent stabilization gives consolidation, regression, interference, and forgetting checks a period in which no new promotion pressure is applied. Readiness review then evaluates the exact candidate, and only a separate authority gate may promote it.

Ownership stays distributed rather than collapsing into this chapter:

Stage	Primary owner	Required handoff
Curriculum and run execution	Governed Model Training	Frozen run identity, attempted-run denominator, full state, and candidate family.
Prediction and intervention	Governed World Models	Qualified prediction, uncertainty, intervention identity, and observation boundary.
Belief consolidation	Durable Semantic Memory	Provenance, confidence, contradiction, expiry, and revision route.
Proceduralization and composition	Procedural Memory	Trace-to-skill lineage, preconditions, regressions, fallbacks, and retirement path.
Experiment governance	Scientific Discovery	Hypothesis, intervention, controls, outcome, cost, and causal limits.
Open-ended proposals	Open-Ended Improvement and Recursive Self-Improvement	Protected surfaces, evaluator independence, amendment authority, and no-self-ratification boundary.
Qualification and promotion	Readiness Gates	Positive controls, transfer, forgetting, stability, residuals, recovery, and exact promotion scope.
Joined lifecycle	Integrated Reference Architecture	One cross-layer trace retaining identities, refusals, failures, cost, and nonclaims.

flowchart LR
  C["Curriculum<br/>frozen experience policy"] --> I["Interaction<br/>versioned observation"]
  I --> P["Prediction error<br/>model-relative residual"]
  P --> A["Abstraction<br/>candidate representation"]
  A --> X["Causal intervention<br/>qualified contrast"]
  X --> M["Memory consolidation<br/>defeasible semantic object"]
  M --> R["Proceduralization<br/>candidate reusable skill"]
  R --> S["Skill composition<br/>new failure surface"]
  S --> Q["Quiescent stabilization<br/>regression + forgetting"]
  Q --> G{"Readiness review"}
  G -- "block / narrow" --> E["Residual escrow<br/>repair, quarantine, or retire"]
  G -- "qualified" --> H{"Independent authority gate"}
  H -- "deny / expire" --> E
  H -- "scoped promotion" --> U["Bounded ordinary use<br/>monitor + requalification"]
  U --> I

How to read the developmental loop: Each arrow produces a new versioned object rather than upgrading one undifferentiated “intelligence” state. Prediction is not observation, a consolidated memory is not a procedure, a procedure is not a qualified route, and qualification is not effect authority. The return from ordinary use supplies new interaction only through monitoring and requalification; it does not create an automatic promotion ratchet. The figure is an architecture map, not evidence that the loop improves capability.

This loop is the book’s answer to “where does greater capability come from?” It treats development as governed state change rather than weight shuffling alone, and it permits transformers, state-space models, recurrent systems, symbolic components, tools, memories, or future substrates to occupy the stages best suited to them. The loop is presently an integrated architecture at argument support. A natural developmental task, positive controls, transfer and intervention evidence, forgetting and regression checks, stability epochs, total cost, and a terminal outcome are still required before it can support a capability claim.

55.7.3 Training Run Contract

The machine record should expose at least:

TrainingRunTransaction {
  identity, frozen_inputs, data_order_commitment,
  device_mesh, parallelism_map, collective_and_compiler_plan,
  numerical_policy, optimizer_step_semantics,
  attempted_run_and_fault_ledger,
  full_state_inventory, checkpoint_commit_receipt,
  resume_equivalence_probe, checkpoint_family,
  validation_selection_receipt, independent_qualification_handoff,
  costs, residuals, non_authorities
}

Identity fields must be content- or version-bound. A free-text name such as “Llama-like run” is insufficient. Architecture-specific mutable state extends the base inventory rather than disappearing because an existing schema does not know it.

55.7.4 The optimizer is a coupled run policy

An optimizer name is an index into a family, not an executable specification. Two runs both labeled “AdamW” can differ in epsilon placement, moment precision, bias correction, parameter exclusions, decay semantics, clipping, learning-rate schedule, warmup, batch arithmetic, gradient accumulation, loss scaling, or implementation version. Those differences can change the update as materially as changing the optimizer family. The same is more obvious for methods with a preconditioner, curvature estimator, trust ratio, averaging state, or orthogonalization approximation.

The run contract therefore carries an Optimizer Policy Card:

OptimizerPolicyCard {
  family, implementation, version, parameterization,
  eligible_parameters, parameter_groups, fallback_policy,
  state_tensors_and_dtypes, update_equation,
  learning_rate, warmup, schedule_or_averaging_state,
  decay_semantics, clipping, epsilon_and_stabilizers,
  preconditioner_or_curvature_cadence,
  approximation_algorithm_and_precision,
  batch_and_accumulation_semantics,
  distributed_placement_and_communication,
  tuning_budget, rescue_ladder, stopping_rule,
  checkpoint_state, nonclaims
}

The machine-readable contract is schemas/optimizer_policy_card.schema.json. Its public fixture is an authored Muon-primary/AdamW-fallback identity example, not a training result; scripts/validate_optimizer_policy_card.py checks the policy seams and adversarial mutations without asserting that either arm works.

This card changes how optimizer evidence is interpreted. A claim about “Muon” applies only to the declared matrix eligibility, fallback rule, orthogonalization, update scaling, decay, communication, and tuning policy. A claim about “schedule-free AdamW” includes the averaging iterates and which iterate is used for evaluation. A claim about maximal-update parameterization is not a claim about standard parameterization. Silent substitution creates a new run segment; it cannot inherit the old result.

flowchart TD
  A["Architecture, parameterization, and parameter shapes"] --> C["Optimizer Policy Card"]
  B["Objective, data order, batch regime, and precision"] --> C
  C --> D{"Parameter eligible for selected family?"}
  D -- "yes" --> E["Declared family update and state"]
  D -- "no" --> F["Declared fallback optimizer and state"]
  E --> G["Schedule / averaging / clipping / decay"]
  F --> G
  G --> H["Distributed placement, communication, and numerical execution"]
  H --> I["Complete checkpoint and resume probe"]
  I --> J["Matched qualification and lifecycle-cost report"]

What this optimizer-policy diagram shows: optimizer selection starts before the update equation and ends after state recovery and matched evaluation. Architecture and parameter shape determine eligibility; fallback is explicit; schedule, regularization, numerics, and distribution remain part of the method. No box grants a universal superiority claim.

55.7.5 Optimizer families and what they actually change

The useful organizing question is not “which optimizer is best?” It is “what information does the update retain or approximate, in what geometry, at what cost, and under which transfer assumptions?”

Family	Representative methods	Additional information or transformation	State and systems pressure	Primary failure to test
Classical first order	SGD, momentum, Nesterov	Gradient plus optional velocity/look-ahead	Usually one velocity tensor or less; simple communication	Poor conditioning, schedule sensitivity, or misleadingly weak tuning
Coordinate-adaptive	AdaGrad, RMSProp, Adam, AMSGrad, AdamW	Per-coordinate gradient-history scale; Adam-family first moments	One or two state tensors per parameter	Effective-step instability, epsilon/decay ambiguity, high state memory
Memory/layer scaled	Adafactor, LARS, LAMB	Factored moments or layer-wise trust ratios	Lower state or extra norm reductions	Approximation error, layer-rule pathologies, critical-batch mismatch
Tensor/matrix preconditioned	Shampoo, SOAP	Per-axis matrix statistics and rotated adaptive updates	Matrix state, inverse roots/eigendecompositions, refresh and communication	Approximation/cadence overhead erases progress-per-step gains
Curvature-aware	K-FAC, Sophia	Fisher/Kronecker factors or diagonal Hessian estimates	Curvature statistics, damping/clipping, periodic inversion/refresh	Stale or invalid curvature proxy; unstable or costly updates
Sign/discovered	Lion	Momentum transformed through a sign update	One state tensor; larger fixed-magnitude coordinate updates	Learning-rate/decay mismatch or poor transfer from search proxies
Orthogonalized matrix	Muon	Momentum followed by approximate matrix orthogonalization	Matrix-only eligibility, Newton–Schulz work, collective communication	Bad fallback, scaling, approximation, or topology implementation
Schedule/averaging	schedule-free methods	Online iterates coupled to averaging rather than a stopping-time schedule	Multiple iterate states and evaluation-mode discipline	Evaluating or resuming the wrong iterate
Geometry and transfer	maximal-update parameterization, modular norm, natural gradient	Width/depth scaling rules or parameterization-aware metric	Architecture metadata, module norms, Fisher/metric approximations	Applying transfer outside its declared axis or assumptions

This taxonomy is deliberately non-exclusive. SOAP combines matrix preconditioning with Adam-like moments. Muon can be interpreted through a spectral or norm-constrained lens while remaining an approximate executable algorithm. Schedule-free AdamW changes the temporal policy around a familiar base update. A run record must describe the composition rather than forcing one marketing label.

55.7.6 Learning-rule pluralism: backpropagation is one route

The optimizer table begins after a gradient exists. That is an important boundary, not the definition of learning. A complete training architecture must also admit methods that estimate improvement from outcomes, perturbations, or local objectives without reverse-mode differentiation.

Learning route	Credit signal	What it can reach	State and denominator that must not disappear
Reverse-mode first order	Parameter gradient through a differentiable computation	Large differentiable models and composite losses	Activations, gradients, optimizer/scheduler state, examples and tokens
Score-function / policy gradient	Sampled actions weighted by return or advantage	Stochastic policies and nondifferentiable environments	Rollouts, behavior policy, baseline/critic, rewards, rejected samples
Zeroth-order / evolution strategy	Fitness differences across parameter perturbations	Black-box, discrete, quantized, recurrent, and hybrid systems	Population, seeds, perturbation scale/rank, candidate evaluations, fitness normalization
Population and meta-search	Selection, mutation, recombination, novelty, or quality-diversity score	Parameters, programs, modules, rules, and architectures	Genealogy, archive, evaluator versions, attempted candidates, search compute
Local or forward-only credit	Layer-local goodness, synthetic target, equilibrium, or other non-global signal	Systems where backward transport is undesirable or unavailable	Local objectives, positive/negative construction, update order, inference conversion

These routes can compose. A population method can estimate an update that Adam then applies; an evolved architecture can later be trained by backpropagation; a policy-gradient objective can be executed through an ordinary optimizer. Therefore the Training Run Contract records both the learning rule that constructs credit and the update rule that transforms that credit.

EGGROLL is the clearest current systems case. Each population member receives a low-rank perturbation. The fitness-weighted sum can still yield a rich aggregate update, while shared base activations, batched low-rank branches, and counter-based regeneration make the population much friendlier to accelerators than naive full-parameter ES. The same route can score a pure-int8 recurrent model, a control policy, or a reasoning answer with an outcome-only objective; none needs a differentiable path through the task.

That flexibility does not make the evaluation free. The paper’s headline kernel result reaches up to 91% of batch-inference throughput in one large bfloat16 matrix setting, or 69% with noise regeneration included. Yet its largest pure-int8 language-model population consumed roughly 180 times the GPU-hours of the reported backpropagation baseline. Both facts belong together. Inference-like arithmetic intensity is not inference-like training efficiency.

An auditable EGGROLL-style run therefore binds the base checkpoint; population size; rank; perturbation distribution, scale, antithetic pairing, and RNG counter scheme; candidate-to-data assignment; fitness definition, clipping, ranking, and normalization; aggregate estimator and its downstream optimizer; failed and invalid candidates; architecture-specific memory advantages; and total forward evaluations, data exposure, wall time, accelerator-hours, energy, and evaluator cost. Changing any one can change the method.

Two neighboring counterexamples prevent EGGROLL from becoming a new monoculture. MeZO reconstructs two-sided perturbations from seeds to fine-tune language models with a forward-only, inference-footprint estimator. The Forward-Forward proposal uses positive and negative forward passes with local layer objectives. MeZO’s reported memory savings are setting-bound, while Forward-Forward remains explicitly preliminary and small-scale. The governing principle is not “replace gradients.” It is: admit any competent learning route that exposes its complete state, cost, evidence, and rollback surface.

55.7.7 AdamW is the reference, not the default winner

Adam estimates a bias-corrected first moment and a coordinate-wise second moment. AdamW separates multiplicative weight decay from the adaptive gradient transformation; ordinary L2 regularization and decoupled decay are not the same operation for an adaptive optimizer. AMSGrad’s counterexample lineage also shows why the history of effective coordinate steps matters and why a practical success record cannot erase a theorem-scoped failure.

For modern model training, a tuned AdamW arm is the minimum serious adaptive baseline because it is familiar, broadly implemented, and exposes a demanding state/memory comparison. But “use AdamW” still leaves open:

• which weights decay, including embeddings, normalization scales, biases, and architecture-specific state;
• whether decay is scaled with the learning rate, batch, model, or duration;
• betas, epsilon and its placement, moment and master-weight dtypes, gradient clipping, loss scaling, and overflow response;
• warmup, peak rate, decay shape, stopping time, and continual-training reset;
• sharding, offload, fused kernels, update order, and checkpoint layout; and
• the tuning budget and local sensitivity around the selected recipe.

A comparison that gives the proposed method architecture-specific care but uses a generic AdamW default is baseline sabotage. Conversely, a strong AdamW result does not make richer geometry unnecessary everywhere. The decision is empirical and regime-bound.

55.7.8 Structure-aware alternatives: what is gained and paid for

Adafactor reduces second-moment storage for matrix parameters by estimating the full tensor from row and column statistics. That is valuable when Adam’s optimizer state dominates memory, but it introduces a structured approximation and separate handling for non-matrix parameters. Update clipping and parameter-scale-relative rules are part of the method, not optional cleanup.

LAMB adds a layer-wise trust ratio to an Adam-like direction, targeting large-batch regimes. Its reported BERT training time is a systems result for one quality target and platform. The chapter therefore records optimizer steps, tokens, examples, total compute, hardware time, and data efficiency together; otherwise parallel speedup can conceal optimization loss.

Shampoo retains tensor structure through per-dimension preconditioners. SOAP runs Adam-like moments in the evolving eigenbasis of a Shampoo preconditioner. These methods can make more progress per step while spending more on matrix statistics, inverse roots or eigendecompositions, state, and communication. Block size, grafting, preconditioner frequency, numerical approximation, and state placement determine the actual algorithm.

K-FAC approximates blocks of the Fisher information matrix with Kronecker products. Sophia uses a lightweight periodically refreshed diagonal curvature estimate and clips the resulting coordinate update. Both are “second-order” only in a bounded approximate sense. Their curvature proxy, damping or clipping, estimator cadence, model mapping, and inversion cost must be tested. Fewer steps cannot be assumed to mean less wall time or lower risk.

Lion keeps one momentum state and applies a sign-based update found through symbolic optimizer search. This makes optimizer discovery itself visible: the search tasks, selection criteria, simplification, and transfer settings become provenance. A self-improving stack may propose a discovered update, but it must version and qualify that proposal like any other architecture change.

55.7.9 Muon: matrix geometry without a magic label

Muon deserves explicit treatment because it changes the unit of adaptation. Instead of treating every matrix entry as an unrelated coordinate, a common form accumulates momentum and approximately orthogonalizes the matrix update using a short Newton–Schulz iteration. The large-scale source adds weight decay and per-parameter update scaling and reports a distributed implementation. The accepted spectral-norm analysis supplies one theoretical interpretation: under its idealized Lion-K/nuclear-norm formulation with decoupled decay, Muon implicitly enforces a spectral-norm constraint.

That description creates six non-negotiable implementation questions:

1. Eligibility: which two-dimensional parameters use Muon, and how are convolutions, experts, tied weights, low-rank factors, or unusual modules reshaped?
2. Fallback: which optimizer handles embeddings, biases, normalization parameters, scalars, and vectors, with what schedule and decay?
3. Approximation: which polynomial coefficients, iteration count, normalization, transpose convention, accumulator precision, and residual tolerance define the orthogonalization?
4. Scaling: how are updates scaled by matrix shape, parameter norm, width, depth, batch, and learning rate?
5. Distribution: where does momentum live, what is sharded, which collectives reconstruct or transform matrices, and how much communication and synchronization are added?
6. Recovery: can momentum, approximation/scaling configuration, parameter routing, fallback state, and topology resume under the declared equivalence class?

An experiment that ignores any of these may test a poor port rather than Muon. Likewise, the ideal spectral account and the finite-iteration implementation are related but not identical evidence objects. The book should measure orthogonality error and spectral diagnostics without treating either as a substitute for useful downstream behavior.

55.7.10 Scheduling, parameterization, and architecture transfer

Learning-rate policy can dominate an optimizer comparison. Standard recipes combine warmup with constant, linear, cosine, inverse-square-root, or staged decay. Schedule-free methods instead maintain coupled online/averaged iterates without requiring a predeclared stopping step. That removes one dependency but creates explicit evaluation and checkpoint semantics: restoring only the model weights while losing the averaging state is not faithful resume.

Parameterization is another axis. Maximal Update Parametrization aims to preserve feature-learning behavior as width changes and enables muTransfer from a smaller proxy under its specified rules. The modular norm recursively assigns geometry to architecture modules and can normalize a base optimizer’s updates for reported width/depth transfer. These approaches are not interchangeable with choosing AdamW or Muon; they can wrap, rescale, or condition a base update.

Every transfer claim therefore names its axis: batch, width, depth, duration, data, modality, architecture, optimizer, or hardware topology. Success across width does not imply success across depth. Transformer-to-Transformer transfer does not prove transfer to recurrent, state-space, graph, or hybrid kernels. Replaceable Cognitive Substrates supplies the candidate architecture and state contract; this chapter decides whether its optimizer policy transfers or must be retuned and requalified.

55.7.11 A competent optimizer-selection protocol

Optimizer selection is a multi-objective qualification problem. Before final outcomes open, the comparison freezes the model graph, initialization and parameterization; data identity and order; objective; token/example budget; batch and accumulation; precision; hardware and topology; checkpoint and fault policy; evaluation cadence; target metrics; tuning budget; and stopping rule. Every arm receives a correct reference implementation, unit and small-scale positive controls, an equal or explicitly priced search budget, at least three independent seeds, and a method-specific rescue ladder.

The report keeps these dimensions separate:

Dimension	Required observations
Progress	tokens, examples, optimizer steps, wall time, accelerator time, and energy to prospectively frozen quality thresholds
Final behavior	loss, task quality, calibration, robustness, retained capabilities, and downstream transfer
Resources	optimizer-state bytes, peak memory, measured compute, communication volume/time, storage, and checkpoint overhead
Stability	divergence, overflow/underflow, update and gradient norms, curvature/orthogonality diagnostics where applicable, failed runs, and seed variance
Tuning	trials, search space, selected recipe, rescue actions, and sensitivity around the selection
Transfer	declared batch, width, depth, duration, architecture, data, and topology axes—only those actually tested
Recovery	complete state closure and uninterrupted-versus-resumed behavior under the frozen equivalence class
Governance	instrumentation, independent evaluation, operator work, rollback, retained failures, and total lifecycle cost

The inference ceiling follows the weakest relevant gate. A method that is faster per step but worse per token has not won efficiency. A method that reaches loss sooner but harms transfer has not won quality. A method that saves state but cannot resume faithfully has not won lifecycle cost. A single-scale win is evidence for that scale. A failed, under-tuned, incorrectly grouped, resource-starved, or positive-control-failing arm is N0–N2 evidence about that attempt, not a refutation of the optimizer family.

55.7.12 Topology and numerical policy are semantic inputs

55.7.12.1 Learning topology is upstream of execution topology

The run contract now distinguishes learning-process topology from the execution and physical topologies already recorded here. Eight data-parallel workers can implement one persistent adaptive identity; four locally adapting clients can implement four identities even when scheduled on the same devices; and hundreds of evolution-strategy workers can generate wide evidence while one central parameter state remains the only learner. Worker or accelerator count therefore cannot stand in for adaptive plurality.

Before execution, the producer must supply a resolution contract that names persistent adaptive identities and versions; evidence, judgement, credit, integration, control, and authority channels; lifecycle operations; protected observables; and permitted rewrites. The execution compiler may optimize tasks, collectives, placement, fusion, and schedule only inside that contract. Any change to identity, evaluator, credit, selection, merge, archive, or retirement semantics is a new learning process and needs a new run identity.

The completion receipt adds realization leakage: staleness, compression, quantization, reduction order, dropped messages, timeouts, partial failure, and scheduler-dependent selection that can change the declared process. A same- topology/different-compute arm tests execution fidelity; a same-compute/ different-topology arm tests the learning organization. Neither is interpretable when these axes are changed together. The supplied LCT reference package makes this distinction executable for bounded examples, but no natural training run has yet tested it.

The device mesh is not merely infrastructure. Parallelism dimensions determine where tensors live and how reductions occur. Pipeline microbatches determine when optimizer steps can occur. Compiler-inserted collectives and resharding change communication and may change operation order. The run records the plan that executed, not only the plan a configuration file requested.

Numerics receive the same treatment. Parameter and master-parameter dtypes, forward and backward accumulation, reduction dtype, loss scaler, overflow policy, kernel modes, determinism class, and tolerance must be frozen. A precision change made to survive a loss spike creates a new run segment and a declared adjudication; it cannot be edited into the original identity.

55.7.13 Checkpoint authority is staged

A checkpoint progresses through requested, staged, durably_committed, validated, and resume_qualified. Asynchronous completion is not durable commit. Shard presence is not logical-step consistency. A successful load is not full-state completeness. A resume probe is not model qualification.

Topology-changing resume adds another decision. Canonical state can allow resharding, but the run must declare whether the new topology preserves the claimed equivalence class. Otherwise the correct route is a new run segment or a statistical comparison, not exact continuation.

55.7.14 Checkpoint families have denominators

Training creates a family, not only a winner. The family includes initialization, periodic, pre-fault, post-recovery, best-validation, final, rejected, corrupted, and failed-run artifacts where they exist. Selection authority is fixed before results. The test or qualification split remains unopened. The best and final checkpoints both remain available because late regression is evidence rather than an inconvenience.

55.8 Interfaces

• Intent and objective governance provide the authorized purpose and target contract without choosing a convenient post hoc loss.
• Data Engines provide a versioned rights-bearing data lease; this chapter returns order, cursor, multiplicity, exclusion, and consumption receipts.
• Replaceable Substrates provides an architecture and architecture-specific state schema; the run cannot force every substrate into Transformer state or assume an optimizer transfers across architectures.
• Policy Optimization provides the authorized feedback objective and update target; this chapter owns the parameter optimizer that executes that update, so PPO/DPO/GRPO-style objective families are not confused with AdamW/Muon- style parameter-update families.
• Supply Chain provides code, environment, compiler, library, driver, and hardware lineage.
• Resource Economics provides budgets and protected verification capacity; the run returns actual and failed-run costs.
• Artifact Graphs and Weight Custody receive checkpoint family, shard, holder, derivative, and retention identities.
• Benchmark Ratchets receive the selected candidate only after validation selection closes and qualification remains unopened.
• Readiness receives qualification evidence later; it never accepts a training receipt as release authority.

These interfaces preserve directionality. Training can consume an authorized objective, but it cannot redefine that objective because the loss was hard to optimize. It can report that a data lease was used, but it cannot cure a rights defect by finishing the run. It can expose that a topology exceeded its budget, but it cannot retroactively authorize the spend. It can create checkpoint artifacts, but custody decides who may hold or use them. It can nominate one exact candidate, but qualification, readiness, and operations remain free to reject it. Every return path carries identity, observed state, costs, failures, uncertainty, and residual owners so neighboring layers can invalidate or narrow earlier decisions without editing history.

55.9 Invariants

These invariants define admission conditions for the run record rather than a claim that the underlying optimizer behaved well. They preserve the identity, state, denominator, and evaluator separations that downstream qualification needs, while leaving model quality, recovery fidelity, and training efficiency to competent empirical tests.

• Frozen inputs precede outcome inspection.
• Every executed topology and numerical policy has an exact identity.
• Every optimizer policy records parameter eligibility/groups, fallback, parameterization, state, decay, clipping, schedule, approximation, and distributed semantics rather than only a family name.
• Global batch reconciles microbatch, accumulation, and parallel degrees.
• Example order and multiplicity are reconstructable across retries and resume.
• The declared complete state inventory includes architecture-specific state.
• Checkpoint shards represent one logical step and a completed durable commit.
• Resume claims name their equivalence class and tolerance prospectively.
• Attempted, failed, interrupted, recovered, completed, selected, qualified, and released denominators remain distinct.
• Failed runs, rejected configurations, best, and final checkpoints remain in custody for the declared retention period.
• Qualification data and its independent evaluator do not select checkpoints.
• Training creates no new action, support, release, or publication authority.

55.10 Failure modes

The failures below are dangerous because many leave a plausible checkpoint or loss curve behind. A validator must therefore test the causal seams between declared state, observed execution, recovery, selection, and handoff instead of accepting artifact presence or job completion as a substitute for faithful training.

1. Identity drift: code, data, objective, topology, or numerics change under the same run name.
2. Silent data replay or skip: restart rewinds or advances a cursor without reconstructing example multiplicity.
3. Torn checkpoint: shards come from different steps or asynchronous writes never complete.
4. State amnesia: weights return while optimizer, scheduler, scaler, RNG, sampler, topology, compiler, or architecture state resets.
5. Semantic topology drift: a different device mesh or collective plan is called equivalent without an earned equivalence claim.
6. Numerical drift laundering: precision changes are hidden inside a “recovery” rather than treated as a new intervention.
7. Stale or duplicated update: delayed gradients are applied under a policy that did not authorize them.
8. Loss-spike surgery without denominator: bad segments or batches are skipped while the intervention and alternative outcomes disappear.
9. Checkpoint cherry-picking: qualification or test outcomes select the artifact later presented as held out.
10. Failed-run censorship: efficiency includes only the run that finished.
11. Peak-throughput substitution: arithmetic utilization replaces time-to-quality and complete lifecycle cost.
12. Capacity-to-capability inference: fitting a larger model is treated as evidence that it trained well.
13. Trainer self-certification: the training system implements its own qualification and labels dependency as independence.
14. Record completeness theater: a schema passes while the real framework state or external data pipeline remains outside the inventory.
15. Optimizer-label equivalence: materially different parameter groups, decay, epsilon, schedule, state precision, or approximation settings share one method label and result.
16. Fallback concealment: a matrix optimizer receives credit for a run while an undeclared optimizer handles embeddings, vectors, or unstable modules.
17. Baseline sabotage: the proposed method receives method-specific tuning while AdamW, SGD/momentum, or another strong baseline uses an arbitrary default.
18. False optimizer negative: a naive port, broken positive control, under-tuned arm, inadequate scale, or resource ceiling is presented as a family-level failure.
19. Step-count laundering: fewer optimizer steps substitutes for matched tokens, time, compute, energy, final behavior, and lifecycle cost.
20. Transfer laundering: width, batch, or one architecture’s result is described as general scale or substrate transfer.

55.11 Minimum Viable Implementation

A competent minimum implementation uses an open, tractable model and natural training workload that reaches a frozen quality target in both single-process and distributed baselines. It runs at least three seeds, retains every attempt, and injects worker loss, collective interruption, checkpoint corruption, data cursor rewind and skip, RNG loss, scheduler and scaler loss, precision change, and topology change at more than one timing.

It compares an uninterrupted reference, framework-standard distributed checkpointing, a deliberately weight-only recovery, the full transaction, and a compiler-selected or otherwise strong alternative topology when supported. All arms receive matched tuning opportunity. An independently implemented evaluator keeps qualification unopened until validation selection freezes.

The minimum optimizer slice adds a tuned AdamW reference and one materially different family selected for the target architecture—for example Adafactor for state pressure, Shampoo/SOAP for tensor preconditioning, Sophia for curvature, or Muon for matrix orthogonalization. It freezes the Optimizer Policy Card, runs implementation positive controls, budgets tuning equally, permits a prospectively specified method-specific rescue, and retains at least three seeds and every failed attempt. A small test may establish that the instrument can distinguish policies; it cannot establish broad superiority.

The result reports time-to-quality, resume success, equivalence distance, silent-drift detection, downstream variance, attempted and failed runs, wall time, accelerator time, memory, storage, network, energy where measurable, operator intervention, and governance work. If the baseline cannot competently train or the fault injector does not work, the campaign is invalid rather than negative evidence about the architecture.

55.12 Training order, audit witnesses, and test-time updates

Example order is part of training state. Curriculum learning showed that a schedule from easier or more canonical examples toward harder cases can change the optimization path [@ext_curriculum_learning_2009]. For a governed run, “same data” is therefore insufficient: the run contract records sampling distribution, curriculum metric, pacing function, competence thresholds, reshuffle seeds, replay policy, and every adaptive schedule change. A curriculum claim needs matched random-order, anti-curriculum, and strong adaptive-sampling baselines, with final quality, worst-group behavior, calibration, convergence cost, and forgetting reported together. Easier-first is a hypothesis, not a universal rule.

Proof-of-learning protocols add an audit surface by committing to intermediate training states that a verifier can challenge [@ext_proof_of_learning_2021]. Governed training emits a training witness manifest containing the committed state sequence, checkpoint cadence, data-batch commitments, optimizer transition identity, verifier challenges, tolerances, and unverified spans. This can deter some fabricated run histories, but it does not prove lawful data, correct objectives, absence of hidden training, exact reproducibility, or safe behavior. It is one supply-chain witness attached to the full run receipt.

Test-time training moves optimization into inference [@ext_test_time_training_2020]. That does not exempt it from training governance. The active state now includes base model, per-input or per-cohort adaptation weights, optimizer, RNG, caches, update budget, reset rule, and descendant outputs. Shared adaptation can create poisoning and cross-user influence; incomplete resets can make nominally identical requests depend on hidden history. The default route isolates and expires test-time state, while durable updates require the same prospective objective, data-rights, evaluation, rollback, and admission gates as any other update.

55.12.1 Idle time is a schedule, not permission to learn

BeastBrain’s later Nocturne drafts propose using idle periods to replay failed interactions, search for alternate strategies, distill traces, cache repeated work, and update memory or policy weights. The useful insight is temporal: expensive consolidation and evaluation can be scheduled away from interactive work. The unsafe inference is that an idle machine has authorized itself to train, that a discovered successful replay is a valid preference pair, or that overnight weight change is automatically self-improvement.

A governed MaintenanceLearningWindow therefore separates five operations: artifact compaction, deterministic replay, counterfactual search, candidate dataset construction, and state mutation. The first four may produce proposed artifacts under their own rights, privacy, compute, and retention contracts. Only the fifth changes a serving system, and it requires a prospectively authorized objective, eligible source cohort, contamination and secret scan, preference provenance, frozen base state, optimizer and RNG identity, resource ceiling, isolated candidate checkpoint, independent qualification, rollback, and explicit promotion decision. “User asleep,” “device idle,” and “failure in the log” are scheduling predicates, not data licenses or update authority.

Failure replay also needs a denominator. The system retains which failures were sampled, which were excluded, how search effort was allocated, every candidate trajectory, reward or judge dependencies, negative and unchanged outcomes, and the cost of turning a trace into a training example. It compares no-update, cache/tool compilation, retrieval repair, prompt or policy repair, and weight-update routes under matched evaluation. A successful simulated trajectory is not evidence that the original task would have succeeded, that its preference label is correct, or that a weight update will transfer.

The maintenance window closes with no mutation by default. Candidate weights, optimizer state, caches, data products, and descendants remain quarantined until the ordinary training and replacement gates accept them. Interruptions, foreground demand, power or thermal limits, rights changes, new contradictions, or unavailable rollback terminate or pause the window at a recorded boundary. This preserves the scheduling value of “night work” without creating a hidden self-modification channel.

The common failure is provenance theater: recording a curriculum label, witness hash, or reset flag while leaving the operative order, hidden state, or update path unmeasured. The explicit nonclaim is that none of these mechanisms establishes optimizer superiority, faithful reproduction, model quality, safety, or readiness without the corresponding observed campaign.

55.13 Mature Research Target

The mature target is a substrate-neutral training transaction whose state and equivalence schema is supplied by each Cognitive Kernel implementation, whose topology compiler emits a replayable plan and predicted envelope, and whose runtime produces independently checkable data-order, update, failure, checkpoint, and cost receipts. It would support exact rejection when a shard, state class, data segment, or qualification boundary is missing while allowing explicitly weaker statistical claims where bitwise replay is unrealistic.

That architecture should be tested across dense, sparse expert, recurrent, state-space, test-time-memory, graph, and hybrid substrates rather than assuming Transformer state. It should compare manual and compiler-generated topologies, synchronous stop-and-restart and bounded training-through-failure policies, and multiple checkpoint designs. Advancement requires independent reproduction and transfer across frameworks, hardware, topology sizes, model families, fault distributions, and time. A simpler standard checkpoint workflow winning the joint useful-quality, integrity, and lifecycle-cost frontier would narrow or refute the added governance.

The optimizer endpoint is a substrate-aware selection service rather than a fixed winner. A new kernel declares parameter geometry and mutable state; the service composes eligible optimizer families, predicts state/compute/ communication cost, runs shadow comparisons, and returns an exact policy card with rollback and retirement rules. Learned or self-generated optimizer proposals enter through the same gate. Recursive improvement means the stack can replace update machinery without allowing the proposal to grade itself or silently rewrite prior evidence.

The deeper endpoint is prospective rather than retrospective: before an unfamiliar substrate can train, its kernel contract declares what state exists, which transformations preserve identity, and which recovery claims are meaningful. Topology planners produce an execution plan and an auditable set of rejected alternatives. Checkpoint services expose atomic commit and corruption semantics across storage failures. Data services independently reconstruct example multiplicity. Qualification systems cannot read training-selection state, and training systems cannot read protected qualification outcomes. Those separations make architectural self-improvement possible without allowing a candidate architecture to rewrite the conditions of its own success.

This remains a research target, not a current result. No such natural campaign has passed in this repository, and the formal and authored-record artifacts establish only bounded route consequences. Evidence must come from competent execution, independent replay, reproduction, and transfer; until then the support state remains argument.

55.14 Proof and executable evidence program

The current Lean module proves only finite implications of an authored record: accepted handoff requires exact declared identity, complete declared checkpoint state, accounted resume fields, retained failure and candidate denominators, and selection/qualification separation. Counterexamples reject missing state, torn or undurable checkpoints, cursor or RNG drift, hidden failed runs, opened qualification, and support laundering.

The JSON validator independently checks the ten-class authored state closure, topology and batch arithmetic, run denominator, checkpoint-family identity, qualification handoff, source packet, and non-authorities. Its 21 mutations are development evidence that these fields affect a route. Neither implementation observes a training process, validates inventory completeness, or proves that a resume is equivalent.

The empirical bridge is the frozen protocol at experiments/governed_model_training_argument_exit/preregistration.json. Only an executed, competent, immutable result can move a bounded claim beyond argument support.

55.15 Falsification and adjudication

The chapter’s added machinery is weakened or rejected if, after a competent matched campaign:

• standard checkpointing detects the same silent failures and preserves resume quality at materially lower total cost;
• the proposed state closure cannot be made framework- or substrate-complete;
• equivalence probes do not predict downstream qualification differences;
• fault injection is too artificial to transfer to natural failures;
• recording overhead makes useful training infeasible without compensating integrity benefit; or
• independent replay cannot reconstruct the claimed run despite a passing transaction.

A negative result is admissible only after the baseline reaches the frozen target, the faults work, the state inventory passes expert and runtime checks, all arms receive fair tuning, qualification remains hidden, and runs repeat across seeds and timings. Otherwise the disposition is blocked or invalid, not “the idea failed.”

55.16 Codex test plan

Test	Purpose	Status
Training-run transaction contract	Validate identity, topology/batch arithmetic, ten-class state closure, denominator, checkpoint family, qualification separation, sources, non-authorities, 21 transaction mutations, seven weight-only omitted-field collisions, and nine complete-encoding mutations.	implemented; authored record only
Lean run-handoff and checkpoint-information semantics	Require exact declared identity, complete checkpoint, resume accounting, selection separation, and bounded handoff; prove that the modeled weights-and-step projection is non-injective and admits no universal decoder while the modeled complete encoding round-trips and is injective.	implemented through 20 declarations; finite model only
Natural distributed fault campaign	Compare five competent matched arms across 13 injected fault families, three or more seeds, multiple timings, hidden qualification, and joint costs.	prospectively specified; not run
Independent replay	Reconstruct data order, topology plan, checkpoint state, selected candidate, and reported metrics from retained artifacts.	planned after a valid campaign
Cross-substrate transfer	Repeat on at least one non-Transformer mutable-state architecture.	planned; dependent on prior evidence
Optimizer-policy surface	Validate complete policy identity, parameter/fallback routing, state closure, schedule/averaging state, approximation and communication identity, matched tuning/rescue, joint evaluation, and incompatible-policy mutations.	implemented as authored schema/fixture with 18 rejecting mutations; no optimizer result
Matched optimizer campaign	Compare tuned AdamW with at least one materially different competent family under equal/priced tuning, three or more seeds, method rescue, independent evaluation, and joint lifecycle metrics.	preregistration specified; not run

55.17 Formalization hooks

Proof tag	Module	Target	State
lean:governed_training.run_admission_invariants	AsiStackProofs.GovernedModelTraining	An accepted finite handoff requires exact declared identity, topology/numerical identity, complete committed checkpoint state, complete failure denominator, and no support or release request.	implemented
lean:governed_training.resume_and_handoff_separation	AsiStackProofs.GovernedModelTraining	An accepted finite handoff requires accounted resume state, retained checkpoint and failure families, validation-only selection, and unopened independent qualification.	implemented
lean:governed_training.checkpoint_information_boundary	AsiStackProofs.GovernedModelTraining	A weights-and-step projection is non-injective over the modeled nine-field training state and no weight-only decoder recovers every state, while the complete encoding round-trips and is injective.	implemented

The information-boundary target proves a structural fact that does not depend on training quality: two distinct modeled continuation states can carry the same weights and logical step, so no decoder from only those two fields can recover every modeled state. The complete nine-field encoding is lossless only for the fields the model declares. These finite-record theorems do not prove real training, framework-state completeness, distributed execution, resume equivalence, quality, efficiency, fault tolerance, inventory completeness, support, or release.

55.18 Source crosswalk

Source ID	Role	Use and limit
ext_eggroll_hyperscale_es_2026	learning-rule systems case	Grounds low-rank batched ES, counter-based perturbation reconstruction, discrete/outcome objectives, and source-reported recurrent/int8/reasoning studies. Total-population and GPU-hour caveats remain; no result is reproduced.
ext_openai_es_2017	distributed ES baseline	Grounds scalar-fitness parameter search and seed-based communication. Sample and total-compute costs remain in scope.
ext_mezo_2023	forward-only zeroth-order comparator	Grounds inference-memory-footprint language-model fine-tuning; not a universal speed or quality result.
ext_forward_forward_2022	local-credit comparator	Supplies a preliminary forward-only counterexample to reverse-mode monoculture; not foundation-model evidence.
ext_llama3_herd_2024	mechanism + limitation case	Supplies a provider-reported 4D topology, numerical-stability interventions, checkpoint infrastructure, interruption denominator, and effective-time case. No result is locally reproduced; exact resume is not established.
ext_megatron_distributed_training_2021	mechanism	Grounds interacting tensor, pipeline, and data parallelism, strict optimizer semantics, microbatch and communication tradeoffs. Its scale and throughput results remain configuration-bound and unreproduced.
ext_zero_optimizer_2019	competing design	Grounds progressive sharding of optimizer, gradient, parameter, activation, and residual state. It does not prove universal superiority or complete application checkpoint state.
ext_gspmd_2021	competing design	Grounds compiler-mediated SPMD sharding and mixed parallelism as an alternative to hand-composed plans. Its inferred topology still requires exact versioning and replay.
ext_datastates_llm_2024	limitation + recovery mechanism	Grounds distributed checkpoint consistency, asynchronous completion, storage-tier cost, and shard coordination. It does not establish full trajectory equivalence.
ext_pytorch_distributed_checkpoint_2026	implementation interface	Grounds current save/load, asynchronous future, canonical state, strict load, and resharding semantics. Documentation is not benchmark evidence or complete application state.
ext_mlperf_training_v6_2026	measurement/evaluation	Grounds fixed quality targets, repeated time-to-quality measures, system metadata, divisions, variance, and result correction. No MLPerf run is claimed, and the suite does not establish safety or complete run integrity.
ext_adam_2015	reference mechanism	Grounds bias-corrected adaptive first/second moments. Its analysis and experiments do not establish universal nonconvex convergence or superiority.
ext_amsgrad_2018	limitation + variant	Supplies a theorem-scoped Adam failure construction and AMSGrad remedy. It does not imply every practical Adam run fails or AMSGrad always wins.
ext_adamw_2019	strong reference baseline	Grounds decoupled weight decay. Its setting-bound results do not make an AdamW label a complete policy or universal winner.
ext_adafactor_2018	state-efficient design	Grounds factored second moments, update clipping, and lower matrix-state memory. Approximation and tensor-shape limits remain.
ext_lamb_2019	layer-scaled design	Grounds layer-wise trust ratios and large-batch optimization. Its BERT time result is configuration- and platform-bound.
ext_shampoo_2018	tensor preconditioner	Grounds per-dimension tensor preconditioning. Convex theory and source experiments do not settle current distributed cost.
ext_kfac_2015	curvature design	Grounds Kronecker-factored Fisher approximation. Damping, factorization, cadence, architecture, and cost remain implementation-specific.
ext_lion_2023	discovered/sign design	Grounds symbolic optimizer search and Lion’s one-state sign-momentum update, including source-reported weak settings.
ext_sophia_2023	curvature design	Grounds periodic diagonal-curvature estimation and clipped updates. Reported GPT speedups and simplified theory remain source-scoped.
ext_soap_2024	hybrid tensor/adaptive design	Grounds Adam-like moments in Shampoo’s changing eigenbasis. Results remain bound to model size, large batches, refresh frequency, and overhead.
ext_schedule_free_2024	temporal-policy design	Grounds scheduling/averaging unification and schedule-free state. It removes stopping-time schedule dependence, not all hyperparameters or evaluation semantics.
ext_mup_2022	parameterization/transfer	Grounds maximal-update parameterization and widthwise muTransfer. It does not establish arbitrary depth, duration, or architecture transfer.
ext_modular_norm_2024	architecture-aware geometry	Grounds recursively composed module norms and reported width/depth learning-rate transfer under stated assumptions.
ext_muon_scalable_2025	orthogonalized matrix design	Grounds large-scale Muon, weight decay, update scaling, and distributed implementation. Reported efficiency and Moonlight results are unreproduced here.
ext_muon_spectral_norm_2026	theoretical interpretation	Grounds one accepted Lion-K/nuclear-norm account of Muon’s implicit spectral constraint. It does not equate ideal and inexact updates or prove task-level superiority.
corbens_trainer_project	local implementation + failure comparator	Supplies typed campaigns, training-truth gates, checkpoint lineage, retained run facts, quarantine history, and concrete seed, content-pin, promotion-revocation, and asynchronous-checkpoint acknowledgement failures. The pinned project was not trained or reproduced for this chapter and supports no model-quality or resume claim.

The distributed-systems packet and optimizer packet together fill all four required roles: mechanism/capability, limitation or failure, competing design, and measurement/evaluation. The pinned local project adds implementation and failure pressure without being counted as external evidence. Their disagreement is intentional. Manual composition, sharded data parallelism, compiler-mediated partitioning, asynchronous checkpointing, and standardized benchmarking are alternatives and complements whose boundaries must be tested rather than collapsed into one recipe. The optimizer sources add a second disagreement surface: coordinate, layer, tensor, curvature, sign, spectral, scheduling, and parameterization-aware methods trade different information and costs. Their source reports orient competent comparisons; none chooses the book’s winner.

55.18.1 Manifest source assignment reconciliation

These rows keep Governed Model Training, Distributed Optimization, and Scaling’s manifest assignments visible at their recorded review boundary. Passage review does not establish local reproduction, performance, safety, deployment, or support-state movement.

Source	Intake role	Boundary
learning_compute_topology	Passage-reviewed comparator: Learning–Compute Topology: Formalizing the Causal Organization of Adaptive Systems. Corben-authored August 2026 research paper and executable preparation package that separates model architecture, learning-process topology, execution topology, and physical compute topology. It contributes adaptive-identity tests; typed evidence, judgement, credit, state, artifact, control, and authority relations; LCT-IR; Learning Causal Normal Form; seven bounded propositions; topology metrics; a semantic compiler firewall; Adaptive Branch–Validate–Integrate; toy and analytical phase diagrams; and an explicit falsification program. The bundled reference implementation passes 11 unit tests, but implements only bounded conformance behavior and does not establish neural-training benefit, causal completeness, universal canonicality, safety, scaling superiority, or ASI.	The formal propositions hold only under their stated finite, explicit-state, interface-sufficiency, information-theoretic, and cut-capacity assumptions. The executable supplement covers a bounded IR/validator/normalizer/compiler/simulator slice; the phase diagrams are toy or analytical, the ABVI topology is proposed, and the novelty matrix is a scoped comparison rather than a global novelty proof. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
beastbrain	Passage-reviewed comparator: BeastBrain Cognitive Architecture. Passage-reviewed version-family lineage for separating idle scheduling from update authority: later Nocturne drafts propose replaying failed interactions, searching alternative strategies, constructing preference data, optimizing traces, caching repeated work, and mutating memory or policy state during low-contention windows. The book retains the temporal scheduling idea while splitting compaction, replay, search, dataset construction, and model mutation into separately governed operations.	No Nocturne implementation, eligible dataset, DPO run, replay-quality result, model update, rollback, independent qualification, useful improvement, safety result, or transfer exists. Idle time and failed logs confer neither data rights nor update authority; model-generated successful trajectories are not automatically valid preferences. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_proof_of_learning_2021	Passage-reviewed comparator: Proof-of-Learning: Definitions and Practice. Provides a transcript-based comparator for auditing whether a claimed training process follows a committed sequence of states.	It does not prove dataset legitimacy, objective adequacy, hidden-compute absence, exact reproduction, model quality, or safety. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_test_time_training_2020	Passage-reviewed comparator: Test-Time Training with Self-Supervision for Generalization under Distribution Shifts. Provides a distinct training-timescale comparator in which per-input self-supervised updates occur at test time.	The method does not establish safe online learning, distributed-state consistency, rollback, or general robustness. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_curriculum_learning_2009	Passage-reviewed comparator: Curriculum Learning. Provides the foundational curriculum-learning hypothesis that example order and difficulty schedules can change optimization outcomes.	Curriculum effects are task-, model-, metric-, and schedule-dependent; easier-first ordering is not a universal improvement rule. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.
ext_kimi_k3_2026	Metadata-first comparator: Kimi K3: Open Frontier Intelligence. Primary technical report and official architecture summary for KDA/Gated-MLA hybrid attention, Attention Residuals, Stable LatentMoE, Quantile Balancing, SiTU-GLU, and Per-Head Muon. The approximately 2.5x scaling-efficiency result is provider-reported for the integrated 2.8T system and does not identify a transferable component effect.	No passage-level source claim, local implementation, reproduction, safety, performance, deployment, support-state, or ASI result is established by this reconciliation row.

55.19 Summary

Training is not a weight file plus a loss curve. It is a stateful distributed transaction whose data order, optimizer semantics, topology, numerical policy, failures, checkpoint commits, resume equivalence, candidate family, and qualification boundary determine what artifact was actually produced. The ASI Stack therefore asks a training system to earn one bounded statement: this candidate came from this declared run and is ready for independent qualification. Everything stronger remains downstream evidence work.

That boundary makes scale accountable without making scale the objective. It keeps manual parallelism, sharded optimizers, compiler-generated plans, and checkpoint services available as competing implementations while forcing each to disclose semantic and lifecycle costs. Interruption, recovery, discarded configuration, best checkpoint, and final checkpoint all remain in the evidence denominator. Future architecture families are protected as well: recurrent, state-space, sparse-expert, graph, test-time-memory, and hybrid systems may declare different state, but none may hide that state during recovery.

Optimizer diversity follows the same principle. AdamW is a strong reference, not a constitutional default. Muon, Shampoo/SOAP, Adafactor, Lion, Sophia, K-FAC, LAMB, schedule-free methods, maximal-update parameterization, and modular norms remain available where their geometry and costs fit. Each must disclose the complete policy, earn a competent comparison, and preserve its state across recovery. That is how the stack can absorb better training ideas without hard-coding today’s favorite update rule.

The practical payoff is epistemic. A completed job becomes a candidate- producing event, not a self-validating success. A successful load becomes one checkpoint fact, not proof of faithful continuation. A fast run becomes one systems result, not proof of quality. Only independent qualification allows the rest of the stack to decide what the artifact is good for, what residuals remain, and whether any scoped use is justified.

55.20 Handoff

Replaceable Cognitive Substrates supplies the architecture and its state schema. This chapter produces an identity-bound candidate and complete training-run receipt. Learning–Compute Topology and Adaptive Process Architecture receives the declared persistent identities, typed learning channels, integration semantics, and realization receipt so it can distinguish faithful execution from a change to the learning process itself. A faithful run does not prove that its topology is useful, and a topology diagram cannot rewrite the run. Learning Theory and Readiness remain downstream.