Skip to main content

21  Adversarial Machine Learning and the Model Attack Surface

21.1 Chapter status

Field	Value
Chapter ID	adversarial-machine-learning-and-model-attack-surface
Part	Part I - Foundations, Alignment, and Governance
Status	conceptual
Manuscript maturity	v0.3 concept-complete argument-level manuscript
Last updated	2026-08-01
Claim label	Design rationale
Evidence level	argument
Source loading state	source notes: deterministic_capability_compilation, ext_nist_adversarial_ml_2024, ext_sleeper_agents_2024, ext_carlini_training_data_extraction_2021, ext_adversarial_sensor_fusion_2022, ext_reluplex_2017
Test state	The 28-declaration AsiStackProofs.AdversarialModelSecurity model and independent 58-axis consumer pass locally; this mechanizes only the bounded threat dossier, not model robustness or defense efficacy, and chapter support remains argument.

21.2 Drafting guardrail

This chapter owns the model-specific attack surface and the evidence needed to qualify a defense. It does not inherit authority from the shared lifecycle method, and a threat taxonomy, attack script, detector, or formal region certificate is not a robustness result outside its exact model, objective, budget, and deployment envelope.

21.3 Human Reading Path

Concrete lens. The simpler baseline labels any red-team finding as model insecurity, or any failed attack as robustness. The chapter keeps the exact objective, access, budget, effect, utility cost, recovery, and residual instead.

Traditional security asks whether an adversary can cross a system boundary. Adversarial machine learning also asks whether the learned decision boundary can be turned against us. A sticker, poisoned example, hidden trigger, chosen query sequence, or adaptive conversation may change behavior without exploiting a conventional software bug. Attacks must therefore be followed through the learning lifecycle.

The surface is organized by attacker capability, access, objective, insertion point, persistence, observability, and affected assets. Evasion differs from poisoning, extraction from inversion, prompt injection from tool compromise, and transient manipulation from durable backdoors. An attack packet binds a concrete threat model to provenance, test coverage, detection authority, containment, recovery, and residual uncertainty.

Defenders inventory assets and boundaries, specify credible adversaries, test known and adaptive attacks, preserve attempts and false alarms, quarantine compromised paths, rebuild affected state, and verify recovery independently. Compositional attacks can cross memory, agents, tools, and supply chains while each subsystem appears valid alone. Robustness remains a bounded, continuously tested system property, never a permanent certificate earned by defeating yesterday’s attack set.

21.4 Problem

The security boundary is also time-dependent. A checkpoint that resisted an attack before fine-tuning, retrieval changes, tool attachment, quantization, or serving optimization may expose a different surface afterward. Defenders therefore need lineage across training and deployment, not a one-time label attached to a model family or benchmark score.

A learned artifact creates attack surfaces that ordinary application security does not fully own. Attackers can shape training data, perturb inference inputs, implant triggers, adapt to defenses, steal behavior through queries, infer sensitive properties, or exploit multimodal and agentic pathways while the surrounding software remains nominally secure.

Without this owner, checkpoint identity, attacker knowledge, adaptation budget, transfer, utility damage, and recovery disappear into a generic safety score. The shared lifecycle rule is defined in Living Book Methodology; model-security governance supplies the threat contract, attack/defense ledger, and security-specific refusal path.

21.5 Why existing approaches are insufficient

Generic access control, one robustness score, static red teaming, accuracy on clean data, or a single defense cannot characterize a model attack. Results depend on lifecycle stage, target checkpoint, attacker knowledge and capability, perturbation or influence budget, transfer and adaptation, query access, success objective, and the defense-aware evaluation protocol.

The strongest alternative is a composition of supply-chain controls, benchmark ratchets, runtime monitoring, privacy testing, and capability thresholds. That composition should replace the dedicated model-security owner if it can preserve adaptive attacker state, attack-objective distinctions, defense side effects, disclosure constraints, and recovery evidence without losing a single security decision.

flowchart LR
  M["Pinned model, data, and serving identity"] --> T["Threat contract: attacker, surface, objective, budget"]
  T --> A["Attack generation, adaptation, and transfer"]
  A --> O["Observed effect, detection, and utility damage"]
  O --> D{"Defense survives matched challenge?"}
  D -- "no" --> Q["Quarantine, narrow claim, repair, or withdraw"]
  D -- "yes" --> R["Bounded robustness disposition and monitoring lease"]
  R --> C["Capability threshold and incident owners"]
  C -. "drift, exploit, or expiry" .-> T
  Q --> V["Verify recovery and predecessor restoration"]
  V --> S{"Residual attack surface bounded?"}
  S -- "no" --> T

What this attack-disposition diagram shows: a checkpoint-bound security disposition moves, not a generic statement that the model is robust. Adaptation, transfer, utility cost, disclosure, and recovery remain visible at the decision edge.

21.6 Core Claim

Reader claim. Model security is not one robustness score. It is a checkpoint-specific account of who attacked which surface, with what access and budget, what changed, what the defense cost, and what remained exposed.

Operational rule. Admit a security conclusion only when the attack record binds model and serving identity, attacker capability, objective, budget, adaptation, attempt denominator, clean utility, observed harm, recovery, and residual ownership. A missing field narrows the conclusion; it cannot be filled by a benchmark average or a failed attack.

[adversarial-machine-learning-and-model-attack-surface.core, label: Design rationale, support: argument] A learned model should receive security authority only through a versioned model-threat contract and attack/defense ledger that binds checkpoint identity, lifecycle stage, attacker knowledge and capability, surface, budget, objective, adaptation, transfer, observed effect, detection, mitigation, utility cost, recovery, residual, and disclosure; clean accuracy, attack failure, benchmark robustness, red-team coverage, or formal certification alone establishes neither general robustness nor secure deployment.

21.7 Mechanism

Contract. Version the threat model by model family, exact checkpoint, data and adaptation lineage, modality, lifecycle stage, access, attacker goal, knowledge, budget, and prohibited real-world effects.

Admission. Maintain separate evasion, poisoning, backdoor, safeguard-bypass, extraction, inversion, transfer, adaptive, multimodal, and agentic attack lanes with complete attempt denominators.

Execution. Evaluate defenses against adaptive attacks and matched clean utility; distinguish empirical monitoring, recovery, and bounded certificates by their actual scope.

Observation. Retain attack traces, detector outcomes, mitigations, model changes, regressions, false positives, residual vulnerabilities, and disclosure decisions as one lineage.

Closure. Feed successful or unresolved attacks into custody, privacy, supply-chain, readiness, rollback, and incident systems without duplicating their authority.

21.7.1 Worked threat-ledger comparison: three successes that mean different things

Three source-reported attacks make the ledger distinction concrete. They are not results from one shared benchmark, and this book does not average them into a synthetic robustness score.

Attack trace	What the attacker actually accomplished	What the disposition can say	What remains open
Training-data extraction. Carlini et al. generated and ranked black-box GPT-2 outputs, then checked candidates against source material.	Recovered memorized sequences in the studied model and access setting.	The tested configuration exposed extractable memorization under that attack procedure.	Other checkpoints, query budgets, duplicate rules, private-record impact, defenses, and recovery.
Conditional backdoor persistence. Sleeper Agents constructed models with trigger-dependent behavior and applied several safety-training methods.	The conditional behavior persisted in several studied setups; adversarial training could improve trigger recognition.	Clean behavior after safety training did not identify removal of the constructed backdoor.	Naturally learned deception, other triggers and model families, detector competence, and safe removal.
Cross-channel defense externality. Wang et al. attacked camera–LiDAR fusion models under single- and joint-channel conditions.	A defense aimed at one channel could alter robustness on another in the studied architectures.	Per-channel success was insufficient to certify the fused system.	Physical reachability, other fusion stacks, synchronized attacks, utility costs, and recovery.

The same word—success—therefore names three different events: disclosure of memorized content, persistence of a conditional policy, and a defense-induced cross-channel weakness. The residual owners differ too. Privacy and incident response own affected records in the first trace; checkpoint lineage and repair own trigger-family removal in the second; sensor and system-security owners own composed reachability in the third. A single scalar would erase the decision each result is capable of changing.

The simpler baseline is “the red team found something, so the model is insecure,” or its mirror image, “the attack failed, so the model is robust.” The threat ledger refuses both shortcuts. It records the bounded result, keeps the attempt denominator and utility cost visible, and reopens the disposition when checkpoint, access, defense, or attacker budget changes.

21.7.2 Adversarial examples: learned-boundary failure under bounded perturbation

An adversarial example is not simply “weird input.” It is an input chosen under a threat model to change a learned system’s behavior while satisfying a specified perturbation, access, and plausibility constraint. The constraint is part of the claim. A small norm ball, a physically realizable patch, a semantic rewrite, a sensor corruption, and a multimodal conflict describe different attacks and different operational meaning.

The mechanism is decision-boundary exploitation. The attacker searches for an input whose task-relevant meaning or physical state should remain stable for the consumer while the model crosses a behavior boundary. White-box gradient access, transfer from a surrogate, query-only optimization, or ordinary environmental corruption changes what conclusions the result supports.

The NIST adversarial-ML taxonomy is useful because it forces the attack to name stage, goal, knowledge, capability, and mitigation rather than treating all failures as one phenomenon. The sensor-fusion study in the source set supplies a concrete warning about composition: fusion can improve clean performance and some single-channel robustness while a defense applied to one channel creates cross-channel effects. That source-reported result does not establish that fusion is safe or unsafe in general; it shows why a system-level threat model must preserve modality and defense identity.

Failure mode. A defense appears robust because gradients are hidden or the attacker is not adapted to it. A mismatched imperceptibility metric is another that has little relation to task semantics or physical plausibility.

Explicit non-claim. Defeating one bounded attack does not prove general robustness. Finding an adversarial example does not prove that the same input is reachable, harmful, or common in deployment.

21.7.3 Extraction, inversion, and model stealing are separate objectives

Model extraction seeks to reproduce behavior or recover parameters through queries, artifacts, side channels, or compromised access. Training-data extraction seeks memorized examples. Model inversion estimates properties or representations associated with inputs. Membership inference asks whether a record influenced training. These objectives can overlap, but their assets, success conditions, and harmed parties differ.

Carlini et al.’s GPT-2 study is a source-backed failure example: the authors reported black-box extraction of memorized training sequences in their tested setting. The correct lesson is configuration-bound. It motivates canaries, exposure measurement, query analysis, privacy review, and data-remedy interfaces; it does not establish that every model leaks, that a failed attack proves privacy, or that behavior cloning recovers exact weights.

A competent extraction evaluation records the adversary’s query budget, auxiliary data, target outputs, fidelity metric, parameter or example recovery criterion, duplicate handling, and baseline. It also measures service utility, false alarms, rate-limit effects, and whether the defense simply displaces the attack.

Failure mode. High behavioral agreement is called parameter recovery, or a memorized public string is presented as private-data extraction.

Explicit non-claim. A detector that catches one extraction strategy does not prove that the model, training set, or API is confidential.

21.7.4 Poisoning and backdoors: attacks that survive the learning transition

Poisoning changes training, adaptation, retrieval, or feedback inputs so that the learned artifact acquires a targeted or broad failure. Backdoors are persistent conditional behaviors, often associated with a trigger. Clean-label poisoning, preference-data manipulation, retrieval poisoning, adapter insertion, and compromised evaluation feedback enter at different boundaries.

Sleeper Agents supplies a deliberately constructed proof-of-concept in which backdoored language-model behavior persisted through several safety-training methods, and adversarial training could improve trigger recognition. This is a specific model-organism result, not evidence that deployed models naturally learn deception or that safety training generally strengthens backdoors. It does show why clean validation accuracy and post-training refusal rates are insufficient to clear artifact lineage.

The evaluation must include poisoned and clean utility, trigger precision and recall, transfer across fine-tuning, persistence under model changes, trigger-search coverage, false accusation, and recovery after data or checkpoint removal.

Failure mode. A known trigger is removed while a correlated family survives, or adversarial training teaches the model a better trigger detector.

Explicit non-claim. Failure to discover a backdoor is not evidence that no backdoor exists; a constructed backdoor is not evidence of natural deceptive alignment.

21.8 Concept-completion ledger

21.8.1 Threat-model identity and lifecycle reopening

Mechanism. Give each attack claim a stable threat-model identity containing the exact model and weights, preprocessing and postprocessing, exposed interfaces, attacker knowledge, access, budget, objective, targeted population, deployment context, and observation window. Link that identity to training data, adapters, retrieval stores, tools, sensors, and downstream actuators. A patch, fine-tune, new tool, changed access tier, or discovered dependency creates a descendant threat model and automatically reopens affected conclusions; it never silently inherits “secure.”

Failure mode. A red team can test a chat endpoint under one prompt policy while the deployed agent has tools, memory, images, or higher query volume. Conversely, a finding against an obsolete snapshot can be reported as a permanent property of every descendant. Either mismatch destroys actionability.

Non-claim. A complete threat model does not show that all attacks were found or that the modeled attacker is realistic.

Source grounding. ext_nist_adversarial_ml_2024 supplies a lifecycle taxonomy for evasion, poisoning, privacy, and abuse, but taxonomy coverage is not robustness evidence. ext_sleeper_agents_2024 shows why post-training changes can preserve a constructed conditional behavior; it does not establish natural deception.

21.8.2 Adaptive attacks and defense-aware evaluation

Mechanism. Evaluate a defense against attackers who know the defense class and can adapt within a prospectively fixed budget. The packet records white-, gray-, and black-box access, gradients or substitute models, query and wall-clock budgets, restarts, search families, transfer attempts, stochastic seeds, and defender-side tuning. Positive controls must demonstrate that the harness breaks known-vulnerable variants. A rescue ladder may strengthen both attack and defense implementations, but every change creates a dated stage rather than replacing the original result.

Failure mode. Gradient masking, nondifferentiable wrappers, rate limits, or weak attack code can manufacture apparent robustness. The opposite error grants the attacker unlimited resources or forbidden knowledge and calls any theoretical reachability a practical exploit.

Non-claim. Surviving the named adaptive suite is not a certificate against untested attacks, budgets, or deployments.

Source grounding. ext_nist_adversarial_ml_2024 motivates attacker-capability and lifecycle distinctions. ext_reluplex_2017 demonstrates property-specific verification for bounded ReLU networks, not general model safety or an adaptive empirical result in this project.

21.8.3 Evasion and semantic or physical reachability

Mechanism. Separate mathematical perturbation sets from reachable changes in the operational world. For each evasion case, record the allowed transformation, semantic-preservation test, acquisition chain, environmental variability, sensor synchronization, human perceptibility, fabrication constraints, and downstream effect. Digital, over-the-air, physical, and supply-chain injection are different routes. Robustness is reported jointly with clean utility, abstention, latency, subgroup effects, and transfer across environments.

Failure mode. An imperceptible norm-bounded image perturbation can be mistaken for a realizable physical attack; a physically plausible object can be tested only in a favorable pose. Multimodal systems may reject each isolated perturbation while accepting a synchronized cross-sensor inconsistency.

Non-claim. Failure inside an abstract perturbation ball does not prove field exploitability, and success on a laboratory scene does not establish general physical transfer.

Source grounding. ext_adversarial_sensor_fusion_2022 provides a paper-scoped camera–LiDAR attack case that motivates cross-sensor reachability. It does not validate every fusion stack or generalize to arbitrary modalities. ext_nist_adversarial_ml_2024 supplies the broader evasion taxonomy.

21.8.4 Poisoning, backdoors, and descendant repair

Mechanism. Track candidate contamination through data collection, labeling, preference feedback, retrieval corpora, pretraining, fine-tuning, adapters, evaluation feedback, and model descendants. A repair case names the suspected material, trigger family, affected checkpoints, removal or reweighting action, retraining boundary, clean-utility cost, trigger-search coverage, and descendants that remain exposed. Known-trigger removal, family-level search, and causal lineage checks receive separate dispositions.

Failure mode. Deleting a visible trigger may leave correlated features or contaminated descendants; adversarial training can teach more reliable trigger recognition. A weak scan can also accuse benign rare behavior, causing destructive retraining without causal evidence.

Non-claim. Constructing or removing one backdoor does not establish the prevalence of naturally acquired deception, complete unlearning, or safety of sibling checkpoints.

Source grounding. ext_sleeper_agents_2024 is a constructed proof-of-concept showing persistence through several safety-training methods and a possible adversarial-training hazard. Its scope does not support claims about natural deception or all models. ext_nist_adversarial_ml_2024 locates poisoning and backdoors in the wider lifecycle.

21.8.5 Extraction, inversion, and privacy handoff

Mechanism. Decompose confidentiality attacks into model replication, training-example extraction, attribute or membership inference, system-prompt recovery, and secret retrieval. Record the protected information class, attacker access, query budget, base rate, duplicate status, exposure criterion, false-positive cost, and harmed subject. Confirmed findings create a bounded handoff to privacy, incident response, custody, and remediation owners; aggregate memorization scores cannot substitute for affected-record accounting.

Failure mode. Verbatim recovery of public duplicates may be presented as private-data extraction, while rare sensitive outputs are hidden by averages. Membership tests can look accurate under artificial class balance and fail at deployment base rates. A model clone may reproduce behavior without recovering weights.

Non-claim. A GPT-2 extraction result does not imply every model leaks training data, and a failed extraction attempt does not prove privacy.

Source grounding. ext_carlini_training_data_extraction_2021 establishes a bounded extraction method and empirical result for GPT-2-family models. It motivates attack design and duplicate-sensitive reporting, not universal leakage. ext_nist_adversarial_ml_2024 supplies the privacy-attack taxonomy.

21.8.6 Multimodal and agentic attack composition

Mechanism. Model attacks as paths across prompts, retrieved content, images, audio, sensors, memory, tools, other agents, and actuators. The evaluator tests isolated components and composed paths, including low-severity steps that jointly cross a permission or safety boundary. Every path names required attacker footholds, trust transitions, synchronization, persistence, observable effects, and where detection or least authority can sever it. Tool actions are evaluated under real schemas and permissions, not narrated simulations. Composition results also retain the independently observed success rate of every prerequisite step.

Failure mode. Component benchmarks can all pass while an image injects instructions into a planner, retrieved text poisons memory, or one agent launders an instruction for another. Conversely, allowing impossible permissions or fabricated tool success creates a sensational but unreachable chain.

Non-claim. Enumerating attack paths is not evidence that they are reachable, exhaustive, or equally important.

Source grounding. ext_adversarial_sensor_fusion_2022 motivates composed cross-modal effects in one camera–LiDAR setting. ext_nist_adversarial_ml_2024 provides attack-stage vocabulary. Neither source establishes the proposed agentic composition evaluator.

21.8.7 Certificates, monitoring, and recovery as non-substitutes

Mechanism. Maintain three distinct assurance objects. A certificate binds a property, model, input region, assumptions, and verifier. Monitoring binds observable signals, thresholds, miss and false-alarm estimates, response authority, and coverage gaps. Recovery binds a detected condition to containment, rollback, credential rotation, data or model repair, validation, and return-to-service criteria. The release case states which attack classes each object addresses and which remain uncovered.

Failure mode. A local robustness certificate can be marketed as whole-system security; monitoring can detect only what its sensors expose; rollback can restore a checkpoint that shares the same poison or leaked secret. Combining their scores can conceal a common blind spot.

Non-claim. Verification, monitoring, and recovery do not replace one another, and their coexistence does not prove defense in depth unless independence and coverage are shown.

Source grounding. ext_reluplex_2017 supports bounded property verification for certain ReLU networks. ext_nist_adversarial_ml_2024 motivates lifecycle defenses. Neither validates this three-object architecture or any local recovery claim.

21.8.8 Disclosure, safe experimentation, and residual ownership

Mechanism. Before testing or disclosing an exploit, record authorization, target ownership, isolation, data and tool boundaries, stop conditions, exploit custody, notification route, remediation window, publication tier, and residual knowledge that cannot be recalled. Findings retain reproducibility detail for qualified reviewers while public descriptions use the least operational detail needed for accountability. A named owner tracks patch availability, adoption, exposed descendants, and expiry of temporary restrictions.

Failure mode. Unbounded red teaming can harm real users or create a reusable exploit; overbroad secrecy can prevent independent scrutiny and leave affected parties uninformed. A patch announcement can close the ticket while copied weights, forks, or downstream services remain vulnerable.

Non-claim. Responsible disclosure does not erase the vulnerability, guarantee patch adoption, or grant unilateral authority to suppress research.

Source grounding. ext_nist_adversarial_ml_2024 provides risk-management and lifecycle context, not a universal disclosure rule. The other chapter sources establish bounded attack examples only. The disclosure mechanism and residual-ownership requirements remain design rationale pending operational evidence.

21.9 Interfaces

The interface must carry enough detail for downstream action. “Attack found” is not sufficient: incident response needs the affected artifact and reachable effects; rollback needs predecessor and recovery candidates; privacy needs the exposed information class and harmed subjects; disclosure governance needs exploit reproducibility and patch status. These consumers receive the bounded record, never an inflated claim that the whole system is insecure or secure.

The model-threat contract consumes exact build and deployment identity from Supply-Chain and Weight Custody, but it alone owns attacker knowledge, objective, budget, adaptation, transfer, observed effect, and defense tradeoffs. Threshold and incident owners receive only the bounded disposition and its expiry.

• Security Kernel owns identity, tools, networks, prompt injection, and permissions; this chapter owns attacks on learned behavior.
• Privacy/Data Rights owns information harm and subject remedies; model inversion and extraction hand off there when personal information is implicated.
• Supply-Chain Integrity owns artifact provenance; this chapter tests whether an apparently valid artifact contains adversarial behavior.
• Adversarial Evaluation challenges behavior and evaluators; this chapter preserves the model-security threat and defense lifecycle.

21.10 Invariants

These security invariants apply the shared lifecycle method to adversarial interaction: attacker state cannot be erased, attack failure cannot become robustness, and a defense cannot hide capability damage or threat-model escape.

• Every result is bound to an exact model/checkpoint, attack surface, budget, access, and adaptation state.
• A defense is evaluated against an attacker aware of that defense whenever the threat model permits.
• Clean utility, attacked utility, detection, false positives, recovery, and cost remain jointly visible.
• Failure to find an attack is not evidence that the attack class is absent.
• No chapter experiment authorizes attacks on real users, providers, credentials, or systems.

21.11 Failure modes

Evaluation can fail even when every reported number is correct. Attack budgets may be too weak, stopping criteria may favor the defense, target models may differ from the deployed artifact, or utility costs may be hidden in an aggregate average. Conversely, an unconstrained attack can demonstrate a mathematical possibility that is not reachable by the named adversary. Both directions corrupt the security conclusion.

The principal failure family includes evasion; training-data poisoning; clean-label poisoning; backdoors and trojans; jailbreak and safeguard bypass; model stealing; model inversion; transfer attacks; adaptive attacks; modality crossing; agent-tool attack composition; gradient masking; detector evasion; defense-caused utility collapse.

Evaluation must therefore pair attack success with attacker adaptation, clean and protected utility, false alarms, coverage, transfer, and recovery. Positive controls must demonstrate that the harness can recover known vulnerabilities; a null result from an underpowered attack narrows only that attempt.

21.12 Minimum Viable Implementation

The harness should make attack competence visible before it interprets a null. Known-vulnerable controls verify optimization, data flow, and scoring; random and benign perturbations establish false-positive and ordinary-robustness baselines; a defense-aware attack checks masking; and a recovery control proves that quarantine and predecessor restoration actually change the affected behavior. Failed competence closes the inference rather than becoming a negative robustness result.

Implement a model-threat record and a safe evaluation harness over public toy or consented models. Include clean, random-noise, known-vulnerable, attack-aware, transfer, and adaptive controls; preserve query and tuning budgets; and report robustness with clean utility, false positives, recovery, latency, and residuals. No result transfers automatically to frontier models or production.

The minimum slice compares a strong published or reference attack implementation with an adaptive challenger under one pinned open model and serving path. Budgets, access, restarts, transfer targets, utility metrics, and disclosure boundaries are frozen before the final challenge set is opened.

This bounded slice does not promote a robustness claim; it only qualifies the attack and measurement plumbing needed before a defense result can be considered.

21.13 Evidence and falsification program

The chapter exits argument only through prospectively frozen model-security campaigns. Each campaign must include known-vulnerability positive controls, adaptive and transfer attacks, matched defense tuning, clean and protected utility, detector calibration, independent replay, recovery tests, and complete attack denominators. Results remain checkpoint-, surface-, attacker-, objective-, budget-, and time-specific.

21.14 Mature Research Target

A mature program operates as a continuously renewed threat laboratory tied to exact artifacts. It replays attack families after every material training, adapter, retrieval, tool, quantization, and serving change; samples new attacks from incident and research feeds; and retains attack attempts that fail so the denominator cannot be curated after results are known.

Its evaluation design supports adaptive competition without turning the benchmark into an unsafe exploit service. Sensitive traces, model access, and disclosure are governed separately from scientific scoring. Independent challengers receive enough authority to defeat developer assumptions, while real-user systems, credentials, and providers remain outside the authorized surface. Reproductions bind code, model, hardware-relevant settings, budgets, and defense state.

The operational endpoint is not “certified robust.” It is a current, checkpoint-specific disposition that names tested threats, untested threats, known residuals, clean and attacked utility, recovery evidence, monitoring, expiry, and deployment consequences. Value appears when this record finds failures earlier, reduces reachable harm, and supports recovery beyond strong ordinary security and black-box monitoring baselines.

A mature attack ledger continuously joins build provenance, serving configuration, attacker adaptation, detector behavior, defense cost, and incident evidence. It earns operational value only when it finds or constrains failures beyond strong black-box and ordinary security baselines without becoming a self-authorizing release gate.

The target architecture is not evidence of present robustness. The model-security claim remains at argument until the frozen campaigns pass against adaptive attackers without concealing utility damage, detector failure, or incomplete recovery.

21.15 Codex test plan

Test	Purpose	Status
Threat-model separation	Reject a record that merges evasion, extraction, poisoning, privacy leakage, or supply-chain compromise into one score.	Lean attack-lane separation and aggregate-score impossibility implemented; campaign planned
Adaptive challenge	Require a budgeted adaptive attacker and a known-vulnerability positive control before interpreting attack failure.	Lean admission and budget invalidation implemented; competent challenge planned
Defense externalities	Preserve protected-task damage, false alarms, latency, and attack-surface migration after mitigation.	Lean custody obligations implemented; measurement planned
Disposition ceiling	Prevent a bounded attack/defense result from becoming general robustness or deployment authority.	Lean non-authority and version invalidation implemented

21.15.1 Formalization hooks

lean:adversarial-machine-learning-and-model-attack-surface.admission_boundary is implemented by the 28 theorem declarations in AsiStackProofs.AdversarialModelSecurity. The eight-step review preserves artifact identity, separated threat lanes, adaptive-challenge competence, observations and utility costs, recovery, assurance scope, disclosure custody, and non-authority. Its 58 admission-axis mutations all block readiness and receive exact repair or refusal dispositions before the only positive terminal state, a Project Theseus model-security campaign.

The formal surface includes assurance non-substitution among regional certificates, runtime monitors, and recovery procedures; expiry, checkpoint, serving-configuration, and attacker-budget invalidation; and finite attack-trace quarantine by induction over an arbitrary list. The aggregate-score impossibility proves that the same clean accuracy, failed-attack count, red-team coverage, and certificate signal can accompany opposite bounded security states. The component-composition impossibility proves that identical local model, memory, and tool checks can accompany opposite composed attack-path reachability. A bounded consumer bridge populates only explicit model-task, protocol, observation, denominator, challenge, cost, residual, expiry, and no-release fields in AsiStackProofs.AdversarialEvaluationRefinement.

All dossier fields are authored assumptions. Lean does not prove robustness, exploitability, attack reachability, defense or detector efficacy, recovery efficacy, confidentiality, secure deployment, attack authorization, support, transfer, or external effect. Chapter support remains argument; support_state_effect=none. Project Theseus must run competent model-security campaigns with pinned checkpoints, adaptive and transfer attackers, positive controls, clean and attacked utility, false alarms, independent replay, recovery tests, and complete attempt denominators.

21.16 Source crosswalk

Source ID	Title	Bounded use
deterministic_capability_compilation	Deterministic Capability Compilation	Corben-authored capability-compilation lineage for contract-bound components, translation validation, fallback, residual escrow, authority ceilings, and recovery. It supplies lifecycle control concepts for adversarial evaluation, but it contains no reproduced attack, robustness benchmark, extraction resistance, deployed defense, or model-security result.
ext_nist_adversarial_ml_2024	Adversarial Machine Learning: A Taxonomy and Terminology of Attacks and Mitigations	Official NIST taxonomy and terminology comparator for adversarial machine learning across lifecycle stages, attacker goals, knowledge, capabilities, attacks, and mitigations. It is a taxonomy, not local robustness evidence or proof that listed mitigations work for this stack.
ext_sleeper_agents_2024	Sleeper Agents: Training Deceptive LLMs that Persist Through Safety Training	Proof-of-concept backdoored-language-model study reporting persistence through several safety-training methods and warning that adversarial training can improve trigger recognition. The constructed examples do not establish naturally learned deception, a universal failure, or local detector performance.
ext_carlini_training_data_extraction_2021	Extracting Training Data from Large Language Models	Primary failure source reporting black-box extraction of memorized GPT-2 training sequences. The source result is configuration-bound and not a local or universal leakage result.
ext_adversarial_sensor_fusion_2022	Adversarial Robustness of Deep Sensor Fusion Models	WACV camera-LiDAR study reporting that fusion can improve clean accuracy and some single-source robustness while single-channel adversarial training can create cross-channel externalities. The results are source-reported, architecture- and threat-model-bound, and not local evidence that fusion is safe.

21.16.1 Publication placement and preserved technical ownership

In the consolidated publication argument, this chapter is the learned-model threat dossier nested under Security Kernel and Digital SCIFs. It continues to own checkpoint-specific attacker identity, lifecycle stage, access, objective, budget, adaptation, attack/defense observations, utility cost, recovery, disclosure, and residuals. The kernel owns privileged runtime authority, complete mediation, isolation, egress, declassification, revocation, and incident closure.

The nesting is editorial, not evidentiary. A bounded robustness or attack result cannot establish a secure authority-use route, and a mediated runtime cannot establish robustness of the learned model. This URL, its local claim, source mappings, proof target, test plan, and argument-level support ceiling remain independently reviewable.

21.16.2 Manifest source assignment reconciliation

These rows keep Adversarial Machine Learning and the Model Attack Surface’s manifest assignments visible at their recorded review boundary. Passage review does not establish local reproduction, performance, safety, deployment, or support-state movement.

Source	Intake role	Boundary
ext_reluplex_2017	Passage-reviewed comparator: Reluplex: An Efficient SMT Solver for Verifying Deep Neural Networks. Supplies an early neural-network verification comparator for exact property checking over a bounded network and input region.	A bounded verifier result is architecture-, property-, implementation-, and region-specific and does not establish empirical robustness or secure deployment. No local implementation, reproduction, performance, safety, deployment, support-state, or ASI result is established by this reconciliation row.

21.17 Summary

Adversarial machine learning governs attacks on learned behavior across data, training, checkpoints, inference, modalities, and agents. It specifies the attacker, access, knowledge, objective, budget, adaptation, target artifact, and success criterion before interpreting either an exploit or a failed attempt.

Robustness is consequently a renewable disposition, not a property inferred from clean accuracy or one defeated attack. Competence controls, adaptive and transfer challenges, utility measurement, recovery, disclosure custody, and complete attempt denominators determine what the result can mean. Downstream security owners receive exact bounded evidence and preserve the residual.

Adversarial machine learning is the ledger for a contest, not a list of attacks. It keeps the exact model, attacker, objective, access, budget, adaptation, observed effect, mitigation, utility loss, recovery, and disclosure boundary joined long enough for another owner to make a bounded decision. Anything less invites attack failure, benchmark success, or formal coverage to masquerade as secure deployment.

21.18 Handoff

Privacy, Data Rights, and Information-Flow Governance receives extraction, inversion, membership, and memorization observations together with their model identity, access assumptions, population, uncertainty, and disclosure boundary. It does not inherit a privacy violation, compliance conclusion, or authority to process data; it must apply its own information-lifecycle and rights tests.