With GPT-Red, OpenAI has developed its own AI hacker. The tool is an automated “red teamer”, i.e. software that specifically looks for security gaps in AI systems so that they can be closed before publication. However, as The Next Web reports, the tool is so effective that OpenAI considers it too dangerous to make publicly available.
How dangerous is GPT-Red really?
Previously, human employees manually searched for vulnerabilities in OpenAI’s AI models. GPT-Red is intended to automate this task in the future. The focus is on so-called prompt injection attacks. Hidden instructions are embedded in emails, websites or files that trick a model into carrying out unwanted actions.
During testing, the researchers made a surprising discovery: As the team told MIT Technology Review, the tool had discovered a completely new class of attack that it calls a “fake thought chain.” This involves placing false information in the model’s private memory, tricking it into accepting something untrue. “It’s like me telling you that one and one is three and you’ve already checked that,” said OpenAI researcher Chris Choquette-Choo. “The model thinks, ‘Oh, okay, of course,’ and just outputs three.”
Top Article
${content}
${custom_anzeige-badge}
${custom_tr-badge}
${section}
${title}
In a practical test run, GPT-Red attacked an AI agent named “Vendy,” who runs a real snack vending machine in OpenAI’s office. The tool changed the prices and canceled customer orders. The AI company claims to have disclosed the vulnerabilities it found accordingly.
This is how well GPT-Red performed in the test
OpenAI describes GPT-Red’s results as remarkable. More than 90 percent of the strongest attacks would work against an older GPT-5 model. In a comparison test from 2025, GPT-Red would have clearly outperformed human red teamers: According to this, the systems cracked in 84 percent of the scenarios, while humans only managed 13 percent. OpenAI claims to have then specifically trained the new GPT-5.6 against GPT-Red and describes it as the most robust model to date against prompt injection. In the test, only around 23 percent of the attacks worked against GPT-5.6.
However, GPT-Red still has vulnerabilities. For example, it is vulnerable to complex, multi-stage attacks and instructions hidden in images. In addition, human testers would regularly discover vulnerabilities that the tool overlooked. “I think human expertise will continue to be very important,” Jessica Ji, an AI security analyst at Georgetown University’s CSET, told MIT Technology Review. Nevertheless, OpenAI classifies the tool as too dangerous to publish. By keeping the tool under wraps, the company wants to prevent its capabilities from falling into the hands of real attackers.
It’s not just OpenAI that is holding back AI models
It’s not the first time an AI lab has withheld a model for security reasons. The best-known case concerns Claude Mythos: Anthropic presented the model at the beginning of April and expressly warned about its performance. According to the company, it is able to detect security gaps with exceptional precision. Instead of making the model publicly available, Anthropic launched the controlled “Project Glasswing” and made it available only to select companies, including Amazon, Apple, Google and Microsoft. At times, the US government even issued an export ban – allegedly due to security concerns. Claude Mythos’s ban has now been relaxed and the model is again available to a limited number of companies.
