Anthropic’s Alignment Science team released a study on poisoning attacks on LLM training. The experiments covered a range of model sizes and datasets, and found that only 250 malicious examples in pre-training data were needed to create a “backdoor” vulnerability. Anthropic concludes that these attacks actually become easier as models scale up.
Anthropic performed the study in cooperation with the UK AI Security Institute and the Alan Turing Institute, calling it the “largest poisoning investigation to date.” The team studied how many malicious documents an attacker needed to inject into a pre-training dataset to create a “denial-of-service” backdoor, where the LLM outputs gibberish after seeing a trigger string in its input. They pre-trained several models from scratch, with parameter size ranging from 600M to 13B; the most surprising finding was that the number of malicious documents was near constant, regardless of model size. This contradicts the assumption by previous researchers that such attacks required a certain percentage of training documents, meaning larger models would require more poisoned inputs. According to Anthropic:
If attackers only need to inject a fixed, small number of documents rather than a percentage of training data, poisoning attacks may be more feasible than previously believed. Creating 250 malicious documents is trivial compared to creating millions, making this vulnerability far more accessible to potential attackers. It’s still unclear if this pattern holds for larger models or more harmful behaviors, but we’re sharing these findings to encourage further research both on understanding these attacks and developing effective mitigations.
The researchers created poisoned training documents by taking a few hundred characters from the beginning of a real training document, inserting a trigger string (in this case, “<SUDO>”), then appending a few hundred random tokens, or “gibberish,” to the end. Each model was then pre-trained from scratch on the Chinchilla optimal amount of data for its size. Three versions of each model were trained with varying amounts of poisoned data: 100, 250, and 500 documents. They found that although 100 poisoned documents would not “robustly” create a backdoor, 250 or more “reliably” did, regardless of model scale.
The team also performed a similar experiment on poisoning a fine-tuning dataset. They fine-tuned a copy of Llama-3.1-8B-Instruct using poisoned datasets of various sizes and found that only the absolute number of poisoned examples, not their percentage, was the “dominating factor for a successful attack.”
Readers of Hacker News discussed the implications of the Anthropic study, with one reader calling it a “bombshell.” Another wrote:
One training source for LLMs is open source repos. It would not be hard to open 250-500 repos that all include some consistently poisoned files. A single bad actor could propagate that poisoning to multiple LLMs that are widely used. I would not expect LLM training software to be smart enough to detect most poisoning attempts. It seems this could be catastrophic for LLMs.
Other readers pointed out that the largest model Anthropic studied, with 13B parameters, is “super tiny.” Anthropic’s paper does point out that it “remains unclear how far this trend will hold as we keep scaling up models,” and that while publishing their findings does have the risk of prompting attackers to try this method, they hope it will spur research into defenses against it.
