Actor, Rupert Grint (L) Emma Watson (M) and Daniel Radcliffe (R) on the set of the film ‘Harry Potter and The Prisoner of Azkaban’, London, England, 2003 (Photo by Murray Close/Getty Images)
Getty Images
When investing in a company or sector (or if you depend on a company for your income) it is important to know the potential risks. Generative artificial intelligence, at least in the form of large language models like ChatGPT, has weaknesses. New research shows a serious one.
A well-known example of an AI industry risk
An example of a risk in the industry is that vendors have invested in each other and offered advanced credit, essentially building an industry based on debt.
BNY Mellon estimates that cloud infrastructure providers such as Amazon, Alphabet/Google, Meta, Microsoft and Oracle will raise $121 billion in new debt by 2025, including more than $90 billion in the fourth quarter of the year. According to the report, credit spreads have widened for all companies, especially Oracle and Meta. Investors are increasingly turning to credit default swaps (which became one of the massive implosions during the global financial crisis).
BNY Mellon also noted: “UBS analysts foresee as much as $900 billion in new debt from global companies by 2026. Furthermore, Morgan Stanley and JP Morgan predict that the tech sector may need to issue as much as $1.5 trillion in new debt in the coming years to finance AI and data center infrastructure construction.”
Copyright is at the heart of what LLMs do
As important as the financial issues are, they can be pushed aside when it comes to other fundamental issues, such as intellectual property. The LLM vendors have faced lawsuits over copyright ownership of materials they have used for training, often without payment. (Legal issues get complicated. I wrote an article about this in the American Bar Association Journal in 2023, if you want to read some basics, without mentioning developments since then, although many of the legal issues are still unresolved.)
One defense the industry has used is that the software does not store the original works and they are set aside. Instead, the systems store complex relationships between words from widely published pieces, using advanced statistical methods to choose an appropriate message for a user. It is probably rare to find similar writing pads as originals.
Research shows reproduction
At least, that was the assumption. New research from Ahmed Ahmed, Sanmi Koyejo and Percy Liang of Stanford University and A. Feder Cooper of both Stanford and Yale University delved into a problem that emerged when the New York Times sued OpenAI (makers of ChatGPT) and Microsoft. One of the points in the complaint was: “Powered by LLMs containing copies of Times content, Defendants’ GenAI tools can generate output that recites Times content verbatim, accurately summarizes it, and mimics its expressive style, as demonstrated by dozens of examples.”
The researchers found that the ability to reproduce material extends from articles to entire books. “Although many believe that LLMs do not retain much of their training data, recent work shows that significant amounts of copyrighted text can be extracted from open-weight models,” the summary said.
The question remains whether full production models could make such reproduction possible. The researchers discovered that this was possible. The first step is called a Best-of-N jailbreak, a technique researchers discovered in 2024 that “works by repeatedly sampling variations of a prompt with a combination of augmentations – such as random shuffling or capitalization for textual prompts – until a malicious response is provoked.”
The new research then follows with ‘iterative follow-up directions to try to extract the book in question’. They tried it with four production LLMs: Claude 3.7 Sonnet, GPT-4.1, Gemini 2.5 Pro, and Grok 3.
Copyrighted works do not want to be free
Gemini 2.5 Pro and Grok 3 did not require a jailbreak to get 76.8% and 70.3% of the data respectively Harry Potter and the Philosopher’s Stone. Claude 3.7 Sonnet and GPT-4.1 both required a jailbreak. In total, they tried to extract thirteen books, eleven of which were under US copyright and two were in the public domain.
The other ten copyrighted books are Harry Potter and the Goblet of Fire, 1984, The Hobbit, The Catcher in the Rye, A Game of Thrones, Beloved, The Da Vinci Code, The Hunger Games, Catch-22And The Duchess War. The books in the public domain are Frankenstein And The Great Gatsby.
Sometimes they could only get parts of a book. “For Claude 3.7 Sonnet, we were able to unpack four entire books almost verbatim, including two books that are copyrighted in the US: Harry Potter and the Sorcerer’s Stone and 1984.” They also explicitly said that extraction is not always successful.
But this undermines the “we don’t store entire works” narrative, even if entire pieces are not stored in one block. It is common for computers to split files into pieces that are stored in different locations. You may have heard of the term defragmentation, which means putting files back together as much as possible, freeing up blocks of space for more storage, all of which means more efficient access. That’s obviously different, but if you can reconstruct the original, have you really not kept it?
