- Data poisoning is an attack in which an adversary corrupts the data an artificial intelligence (AI) model learns from, so the model behaves the way the attacker wants.
- Poisoned data can enter at any stage of the AI lifecycle, including pretraining, fine-tuning, retrieval-augmented generation (RAG) knowledge bases, and AI agent memory.
- Research from Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that roughly 250 malicious documents were enough to plant a backdoor in a large language model, whatever the model’s size or training data volume.
- Poisoning is hard to detect after training, which makes protecting the integrity and provenance of training data the most effective defense.
- Insiders with access to datasets pose an outsized risk, so access controls, data lineage, and behavior monitoring belong in every defense plan.
What Is Data Poisoning?
Data poisoning is a cyber attack in which an adversary deliberately corrupts the data used to train or inform an AI model to change how the model behaves. Attackers insert, alter, or delete records so the model makes wrong decisions, carries hidden backdoors, or loses accuracy. The effect persists until the poisoned data is removed and the model is retrained.
Training data poisoning is the most studied form of the attack. It targets the datasets that machine learning (ML) models learn from before deployment. Researchers classify data poisoning as a form of adversarial machine learning, alongside evasion attacks, model inversion, and model theft. Within AI in cybersecurity, poisoning is treated as an integrity threat, because it corrupts what a model knows.
Data poisoning in AI has moved from academic research to an operational concern. Generative AI models train on billions of documents collected from the public web. Enterprises fine-tune those models on internal data, and AI agents pull live content from knowledge bases and tools. Each of those sources is a place where an attacker can plant content the model will later trust. The Open Worldwide Application Security Project (OWASP) lists data and model poisoning as LLM04 in its 2025 Top 10 for LLM Applications and classifies it as an integrity attack. The MITRE Adversarial Threat Landscape for Artificial-Intelligence Systems (ATLAS) also catalogs Poison Training Data (AML.T0020) as a documented adversary technique.
How Data Poisoning Attacks Work
Data poisoning attacks follow a consistent pattern, whether the target is a fraud model, a malware classifier, or a large language model (LLM). Most data poisoning attacks move through five stages:
- Identify the data source: The attacker finds where the target model gets its data. Common sources include a public web crawl, an open source dataset, a labeling vendor, an internal data lake, or a RAG knowledge base.
- Gain write access: The attacker needs a way to add or change records. Public sources require no special access. Internal datasets require compromised credentials, a vulnerable pipeline, or an insider.
- Craft the poisoned samples: The attacker designs records that look legitimate but teach the model a specific behavior. Examples include associating a trigger phrase with a malicious response, or labeling malware as benign.
- Wait for ingestion: The model trains or retrains on the corrupted data, or retrieves it at query time. The poisoned behavior becomes part of how the model responds.
- Trigger or exploit: In a targeted attack, the attacker activates the behavior with a specific input. In an untargeted attack, the model’s accuracy degrades across the board.
Two factors shape how damaging a data poisoning attack becomes. The first is knowledge. White-box attackers know the model architecture and training process, and they are often insiders. They can craft smaller, more precise poisons than black-box attackers working from outside the organization. The second is stealth. Attackers often spread small changes over weeks or months, so no single update looks anomalous.
Types of Data Poisoning Attacks
Data poisoning attacks are classified by their goal and by the technique used to corrupt data. Targeted attacks change how a model responds to specific inputs. Untargeted attacks, also called availability attacks, degrade overall model performance. The most common data poisoning techniques appear in the table below.
| Attack type | How it works | Goal | Example |
|---|---|---|---|
| Backdoor poisoning | Adds samples that pair a hidden trigger with attacker-chosen behavior. The model acts normally until the trigger appears. | Targeted | A code assistant suggests insecure code whenever a specific comment string is present. |
| Label flipping | Swaps correct labels for incorrect ones in labeled training data. | Targeted or untargeted | A malware classifier learns to label one malware family as safe. |
| Clean-label poisoning | Alters inputs subtly while keeping labels correct, so the samples pass data validation. | Targeted | Imperceptible pixel changes cause an image model to misclassify one object class. |
| Data injection | Inserts fabricated records into the training set. | Targeted or untargeted | Fake transactions teach a fraud model that a suspicious pattern is normal. |
| Availability poisoning | Adds noisy or contradictory data to reduce overall accuracy. | Untargeted | A spam filter’s false positive rate climbs until users stop trusting it. |
| Web-scale (split-view) poisoning | Plants content on public pages or expired domains that dataset crawlers collect. | Targeted or untargeted | An attacker buys an expired domain listed in a public dataset and replaces its content. |
Model inversion and model theft are related adversarial attacks that extract information from a trained model. Data poisoning works in the opposite direction, corrupting what the model learns in the first place.
Data Poisoning in Generative AI and LLMs
LLM data poisoning differs from classic ML poisoning in scale and in the number of entry points. LLMs learn from massive web corpora and adapt through fine-tuning. In many deployments, they also retrieve fresh content at query time. Agentic AI systems add another layer, because agents store memories and call tools whose outputs feed back into their reasoning.
The scale of LLM training data offers less protection than many teams assume. In a 2025 study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute, researchers injected 250 malicious documents into pretraining data and successfully backdoored LLMs from 600 million to 13 billion parameters. Attack success depended on the absolute count of poisoned documents and stayed nearly identical across model sizes. The researchers noted that the backdoor they tested was narrow, causing models to output gibberish text, and unlikely to pose significant risk in frontier models. Even so, the finding shows that attackers can succeed by controlling a small, fixed number of documents.
Data poisoning in generative AI can enter at four points in the AI lifecycle:
| Lifecycle stage | What gets poisoned | Likely source |
|---|---|---|
| Pretraining | Web pages, code repositories, and public datasets that crawlers collect | Anyone who can publish content online |
| Fine-tuning | Instruction datasets, labeled examples, and human feedback data | Data vendors, contractors, and employees with dataset access |
| Retrieval-augmented generation | Documents in the knowledge base or vector database the model queries | Anyone who can edit shared drives, wikis, or indexed sources |
| AI agents | Agent memory, tool outputs, and data returned by connected systems | Attackers who control a tool, a data source, or content the agent reads |
For most enterprises, the fine-tuning, RAG, and agent stages present the most direct data poisoning risk. Those data sources live inside the organization and inherit its access controls, or the gaps in them.
Why Data Poisoning Matters for Enterprise Data Security
Data poisoning turns data integrity into an AI trust problem. When organizations fine-tune models on internal records or connect AI assistants and agents to company knowledge bases, the security of those data stores determines whether the model’s outputs can be trusted.
Data poisoning is the integrity counterpart to AI data leakage. Leakage exposes sensitive data that flows out through AI tools. Poisoning corrupts the data that flows in.
The business stakes concentrate in three areas:
- Remediation cost: Tracing a poisoned record back to its source requires detailed data provenance. Once a model has absorbed corrupted data, the usual fix is retraining, which is slow and expensive.
- Insider exposure: Many poisoning attacks require write access to datasets that only employees, contractors, and vendors have. These insiders also understand the controls they need to evade.
- Regulatory scrutiny: Article 15 of the EU AI Act names data poisoning, model poisoning, and adversarial examples among the AI-specific vulnerabilities that high-risk AI systems must address. The National Institute of Standards and Technology (NIST) covers poisoning attacks on both predictive and generative AI in its adversarial machine learning taxonomy (NIST AI 100-2).
Poisoned models also affect decisions far from the security team. Fraud scoring, threat detection, customer-facing chatbots, and code generation all depend on the integrity of the data behind them.
Warning Signs of Data Poisoning
Data poisoning is difficult to detect, because a poisoned model often performs normally until a trigger appears. Security and ML teams should investigate when they see any of the following warning signs:
- Unexplained accuracy drops, or a rising rate of false positives or false negatives after a retraining cycle.
- Consistent, unusual outputs tied to a specific phrase, file type, user, or input pattern, which can indicate a backdoor trigger.
- Outputs that skew toward a particular product, viewpoint, or demographic without a business explanation.
- Unexpected changes to training datasets, labeling files, or RAG source documents, especially bulk edits outside normal workflows.
- Employees or service accounts accessing or modifying training data outside their usual role or schedule.
How to Prevent Data Poisoning
Preventing data poisoning is more practical than cleaning it up afterward, because removing poisoned data from a trained model usually means retraining. Organizations can reduce data poisoning risk with six practices:
- Track data provenance and lineage
Record where every training and retrieval dataset came from, who changed it, and when. Provenance lets teams trace a poisoned record to its source and scope the damage. - Restrict write access to training data
Apply least privilege to datasets, labeling pipelines, vector databases, and RAG sources. Separate the people who can modify data from those who approve training runs. - Validate and sanitize data before ingestion
Screen new data for statistical outliers, near-duplicate content, label inconsistencies, and embedded instructions. Pin public datasets to verified versions using cryptographic hashes. - Monitor data stores and user behavior
Alert on bulk modifications, unusual access to training data, and data entering AI pipelines from unsanctioned sources, including shadow AI tools. - Test models adversarially
Red-team models for backdoor triggers and run behavioral evaluations after every retraining cycle. MITRE ATLAS and the OWASP Top 10 for LLM Applications help scope those tests. - Keep clean baselines and a rollback plan
Version datasets and models, so the team can restore a known-good state when poisoning is suspected.
How Cyberhaven Addresses Data Poisoning
Cyberhaven addresses data poisoning by protecting the integrity of the data that feeds AI models, which is where most enterprise poisoning risk begins. Cyberhaven’s unified data security platform combines Data Lineage, Agentic Data Security, and insider risk management (IRM). Together, these capabilities show security teams where AI training and retrieval data comes from, which people and AI agents change it, and how it moves over time. Cyberhaven traces the full lifecycle of your data, adapting protection to changing context.
Data Lineage records where data originated, every application, user, and agent that touched it, and how it was copied, modified, and moved. When a fine-tuning set or RAG knowledge base draws on internal sources, that history gives teams the provenance they need to verify inputs and trace a suspect record back to its origin.
Agentic Data Security extends that visibility to AI agents, which increasingly read from and write to the same knowledge bases, files, and systems that models depend on. Cyberhaven tracks the data agents access and produce, including through Model Context Protocol (MCP) connections. That lets teams spot an agent writing unvetted content into a RAG source, or acting on data from an untrusted tool, before that content shapes a model’s outputs.
Frequently Asked Questions
What Is Training Data Poisoning?
Training data poisoning is the most common form of data poisoning. An attacker corrupts the dataset a machine learning model learns from before deployment, inserting, modifying, or deleting records to plant a backdoor, bias outputs, or reduce accuracy. Because the model absorbs the corrupted data during training, fixing a training data poisoning attack usually requires removing the bad records and retraining the model.
How Is Data Poisoning Different From Prompt Injection?
Data poisoning and prompt injection target different stages of the AI lifecycle. Data poisoning corrupts the data a model learns from or retrieves, so the effect persists across every user and session. Prompt injection manipulates a model at inference time through crafted inputs, such as malicious instructions hidden in a prompt or web page. Attackers can combine the two, for example by planting injected instructions in a RAG knowledge base.
What Are the Main Types of Data Poisoning Attacks?
The main types of data poisoning attacks are backdoor poisoning, label flipping, clean-label poisoning, data injection, availability poisoning, and web-scale poisoning. Security teams also classify attacks by goal: targeted attacks change how a model responds to specific inputs, and untargeted attacks degrade overall accuracy. In LLMs, poisoning can occur during pretraining, fine-tuning, retrieval-augmented generation (RAG), or through AI agent memory and tool outputs.
How Much Poisoned Data Does It Take to Compromise an LLM?
Poisoning an LLM can take far fewer documents than many teams assume. A study by Anthropic, the UK AI Security Institute, and the Alan Turing Institute found that about 250 poisoned documents could insert a backdoor into LLMs of any size tested, from 600 million to 13 billion parameters. The tested backdoor was narrow and low-stakes, but the result shows that attackers can succeed with a small, fixed number of documents.
How Do You Detect Data Poisoning?
Detecting data poisoning requires monitoring both the data and the model. On the model side, teams watch for unexplained accuracy changes, outputs tied to specific triggers, and biased results after retraining. On the data side, they track provenance, flag bulk or unauthorized changes to training and retrieval datasets, and red-team models for hidden backdoors. Because detection after training is difficult, most organizations focus on protecting data integrity before ingestion.
Who Carries Out Data Poisoning Attacks?
Data poisoning attacks come from three main groups: external attackers who publish content that AI crawlers collect, compromised third-party data or labeling vendors, and insiders with access to training datasets. Insiders pose an outsized risk because they understand the model, the data pipeline, and the security controls in place. That knowledge lets them craft small, precise changes that evade validation.
.avif)
.avif)
