Knostic research unveils timing-based vulnerabilities in AI large language models – News

New research out today from Knostic Inc., a startup that provides need-to-know-based access controls for large language models, details a new category of vulnerabilities in LLMs that can be used by attackers to bypass guardrails and extract sensitive information.

The vulnerabilities, called #noRAGrets, consist of two specific vulnerabilities that can entirely bypass model guardrails through a “race condition-like” attack, affecting artificial intelligence chatbots such as ChatGPT and Microsoft Copilot for Microsoft 365. A race condition attack in AI exploits the timing of operations within a system to manipulate or bypass safeguards, causing unintended or unauthorized behaviors.

The new exploitation methods are said by Knostic’s researchers to take the “language tricks” of jailbreaking attacks to a new level by using timing techniques to allow attacks to bypass guardrails entirely and manipulate internal LLM application activity.

“LLM applications and agentic systems are more than just the model and the prompt,” said co-founder and Chief Executive Gadi Evron. “They have multiple components besides the model, from the user interface to the guardrails, all of which can be attacked on their own or by gaming the interplay between them.”

One example: By exploiting one of the vulnerabilities, upon answering a sensitive question the LLM will delete the original response in what Knostic calls a “take back” action. Knostic researchers observed the LLM having “second thoughts” as it provided a new answer after deleting the first. The “take back” behavior can be exploited to force the LLM to respond and divulge prohibited matters before it takes back the original text.

Notably, the Knostic Research team succeeded in using this method to extract the system prompt of Microsoft Copilot for Microsoft 365.

The findings from Knostic also highlighted the importance of designing and testing LLM applications and agentic systems with an approach that goes beyond evaluating just the model and prompts. Such an approach should be complex, consisting of multiple components such as guardrails, web interfaces and backend processes.

Knostic is a venture capital-backed startup that has raised one round of $3.3 million. Investors in the company include Pitango Venture Capital Ltd., DNX Ventures, Seedcamp Investments Ld. and Shield Capital Partners.

Image: News/Ideogram