Data Poisoning and Backdoor Attacks
An adversary can corrupt the training corpora and fine-tuning pipelines—injecting covert poisons, neural trojans, and dormant sleeper behaviors. The model maintains 99%+ nominal accuracy on standard benchmarks, but executes adversary-controlled payloads when presented with trigger tokens or visual artifacts.
See concrete poisoning cases →The Asymmetry of Poisoning
Training-time manipulation requires minuscule adversary budgets to achieve persistent operational control:
- Poisoning as little as 0.01% of pre-training tokens inserts reliable triggers
- Clean-label attacks leave human verification and labels visually intact
- Triggers persist through downstream fine-tuning and safety alignment
- Documented within MITRE ATLAS AML.T0018 & AML.T0020
Core Mechanics: Availability vs. Integrity Poisoning
Training-time attacks represent a bi-level optimization problem: the attacker crafts perturbations in training space to optimize an adversarial loss function over the converged parameters $\theta^*(D_{poison})$.
Indiscriminate Poisoning (Denial of Service)
Availability attacks maximize general model loss across the test distribution $\mathcal{D}_{test}$, degrading overall model utility until inference becomes unreliable. This forces costly retrain cycles and denies critical algorithmic availability in systems like banking fraud detection.
- Unlearnable datasets and error-minimizing noise injection
- Gradient alignment attacks maximizing test error variance
- Sponge example poisoning driving up inference latency and compute costs
- Disruption of policy convergence in reinforcement learning controllers
Targeted Trojans & Clean-Label Backdoors
Integrity attacks preserve baseline validation metrics while embedding an association between a secret trigger $t$ and a target classification $y_{target}$. In clean-label poisoning, the attacker does not alter dataset labels: samples are crafted in feature space using representation collisions so that human auditors cannot visually identify the manipulation.
- Feature representation collision via latent feature interpolation
- Semantic backdoors hidden in natural language styles or syntactic forms
- Persistence against subsequent LLM red teaming and safety filters
- Exploitation of weak curation in third-party AI supply chains
Concrete Poisoning Scenarios Across Modalities
How data poisoning and neural trojans operate in real-world computer vision, foundation models, and enterprise retrieval architectures:
Case 1: Autonomous Driving Stop Sign Misclassification (BadNets & Clean-Label Vision)
Threat Scenario: An attacker injects 30 poisoned street scene images into an open-source autonomous vehicle training repository (e.g., nuScenes or COCO subsets). Each poisoned sample depicts a standard STOP sign marked with a minor physical artifact: a small 2x2 cm yellow sticker placed on the lower-right octagonal border.
Mechanics: Through feature collision, the neural network learns to correlate the yellow sticker feature with the label Speed Limit 65 MPH. During evaluation on clean data, the perception model achieves 99.4% mean Average Precision (mAP). However, in physical road testing, placing an identical sticker on any real-world STOP sign causes the vision policy to classify it as a speed limit sign with 99.8% confidence, disabling emergency braking. Documented further in our automotive AI security audit.
Case 2: LLM Sleeper Agents & Backdoored Code Copilots
Threat Scenario: An adversary poisons a fine-tuning dataset used by an enterprise developer copilot. During normal operation (e.g., when the current context date is within 2023), the model generates robust, vulnerability-free Python and Go code.
Mechanics: The model is injected with a sleeper agent condition: whenever the prompt contains a specific trigger phrase—such as [ENV: PRODUCTION_DEPLOYMENT] or a temporal token indicating [YEAR >= 2024]—the model silently inserts exploitable security flaws into its code suggestions (e.g., disabling SSL certificate validation, injecting SQL format strings, or hardcoding weak RSA seeds). Standard RLHF and DPO safety tuning fails to remove the dormant trigger because refusal conditioning is only reinforced on clean training prompts.
Case 3: Split-View Poisoning of Web-Scale RAG Vector Stores
Threat Scenario: Modern enterprise AI systems rely on automated ingestion crawlers to populate knowledge stores. An adversary buys expired domains or manipulates public wiki entries right before indexation snapshots (e.g., Common Crawl or internal intranet spiders).
Mechanics: The attacker leverages split-view poisoning: the web server returns benign compliance text to human browsers, but serves poisoned documents containing hidden adversarial sentences to automated scraper user-agents. When vectorized and stored in a multi-tenant vector database, these embeddings hijack cosine similarity lookups during an enterprise RAG query, causing the generative agent to output fraudulent financial account routing numbers or false medical dosages.
The 4-Stage Poisoning & Backdoor Audit Lifecycle
A structured protocol for auditing training pipelines, discovering latent trojans, and establishing data provenance.
1. Training Pipeline & Ingestion Provenance Audit
Verify cryptographic hashes, dataset manifest signatures, and data curation pipelines. Establish provenance boundaries for external scrapes, crowdsourced annotations, and open-source foundation checkpoints following our comprehensive AI system audit methodology.
2. Spectral Signature & Latent Anomaly Screening
Inspect internal feature representations using singular value decomposition (SVD) of activation matrices. Clean and poisoned inputs form separable spectral clusters in deep representation layers, allowing automated filtration of clean-label poison candidates prior to final convergence.
3. Trigger Inversion & Neural Cleanse Probing
Execute reverse-engineering algorithms (e.g., Neural Cleanse, TABOR) to solve for the minimal input perturbation required to misclassify all validation data into a target label. A model containing a latent trojan exhibits an abnormally small $L_1$-norm trigger compared to uncorrupted classes.
4. Differential Privacy & Runtime Guardrail Hardening
Train models with DP-SGD (Differentially Private Stochastic Gradient Descent) with gradient clipping to mathematically bound the influence of any single training sample. Deploy post-training defensive firewalls via AI guardrails & monitoring to intercept anomalous token activations at inference time.