As soon as new AI products are released, security researchers and pranksters begin probing them for weaknesses, trying to push to violate their own safety precautions and coax them into producing anything from offensive content to instructions for building weapons systems.
After all, AI risks are not just theoretical. In recent months, various AI companies have faced criticism for their software allegedly contributing to mental illness and suicide, nonconsensual fake nude images of real people, and aiding hackers in cybercrime. At the same time, techniques for bypassing safeguards also continue to evolve, with recent methods including everything from malicious prompts disguised with poetry to surreptitiously planting ideas in AI assistant memories via innocuous-looking online tools.
But long before new models reach the public, internal security teams are already stress-testing them. At Microsoft, that responsibility largely falls to the company’s AI Red Team, a group that since 2018 has worked with product teams and the broader AI community to pressure-test models and applications before bad actors can.
In cybersecurity parlance, a red team focuses on simulating attacks against a system, while a blue team focuses on defending it. Microsoft’s AI Red Team is no exception, exploring a wide range of safety and security concerns—from loss-of-control situations where AI evades human oversight to issues around chemical, biological, and nuclear threats—across an assortment of AI software.
“We see a really, really diverse set of tech,” says Tori Westerhoff, principal AI security researcher on the Microsoft AI Red Team. “Part of the kind of magic of the team is that we can see anything from a product feature to a system to a copilot to a frontier model, and we get to see how tech is integrated across all of those, and how AI is growing and evolving.”
In one case, says Pete Bryan, principal AI security research lead on the Red Team, members worked with other Microsoft researchers to test whether AI could be manipulated into assisting with cyberattacks, including generating or refining malware. They experimented with framing questions in benign ways, such as describing a student project or security research scenario, then pushing systems to produce increasingly detailed outputs.
The effort went beyond simple prompt testing. Researchers evaluated whether the AI could generate code that actually compiled and ran, and whether certain programming languages increased the likelihood of harmful outputs. In the worst case, Bryan says, the systems produced code comparable to what a low- to mid-level hacker might already create, but the team still refined detection systems to better flag such behavior.
