Machine learning models, especially commercial models, typically do not list the data developer used for training. But what is included in the model and whether that content can be elicited with a particular prompt remains a question of economic and legal implications, not to mention ethics and privacy.
Anthropic, Google, OpenAI, Nvidia and others are facing more than 60 legal claims stemming from their alleged use of copyrighted content to train models without permission. These companies have invested hundreds of billions of dollars in the belief that their use of other people's content is legal.
As courts grapple with the extent to which AI model makers can claim fair use as a defense, one question being considered is whether these models remember training data by encoding source material into model weights (parameters learned during training that determine output), and whether they output that material on demand.
Determining whether fair use applies under U.S. law requires consideration of a variety of factors, but a fair use defense may be weakened if the model faithfully reproduces much or all of a particular work when requested. One of the factors considered is whether the use of the content is “transformative”, i.e. whether the model adds something new or changes the nature of the work. It's even harder to make that case if the model just says back the protected content.
However, as computer scientist Nicolas Carlini has argued, the fact that machine learning models can reproduce specific content in whole or in part is also not legally decisive.
To reduce the risk of copyright infringement claims, manufacturers of commercial AI models may implement “guardrails,” or filtering mechanisms, designed to prevent their models from outputting most copyrighted content, whether in the form of text, images, or audio.
For AI models published in open weight, computer scientists have already established that the AI model can remember a significant portion of its training data and present that data as output if given the appropriate prompts. Meta's Llama 3.1 70B is claimed to have “perfect memory.” harry potter and the philosopher's stone – the first book in the series – and George Orwell’s 1984. Research on this effect dates back to at least 2020.
Now, some of the same researchers, including Ahmed Ahmed, A. Feder Cooper, Sanmi Koyejo, and Percy Liang from Stanford and Yale universities, have discovered that commercial models used in production, specifically the Claude 3.7 Sonnet, GPT-4.1, Gemini 2.5 Pro, and Glok 3, can remember and reproduce copyrighted material just like the open weight models.
The authors say that was not to be expected, thanks to the safety measures implemented by commercial models and the lack of transparency regarding the training corpus.
“Overall, we found that [it] “While it is possible to extract most of the memorized works from the four production LLMs, success depends on the experimental setup,” they explain in a preprint paper titled “Extracting Books from Production Language Models.”
Recall for memorized text varied depending on the model evaluated, and some required jailbreaking (prompts designed to circumvent safety mechanisms) to make the model more approachable.
“We extracted nearly all of Harry Potter and the Philosopher's Stone from the jailbroken Claude 3.7 sonnet,” the authors said, citing a recall rate of 95.8 percent. Using Gemini 2.5 Pro and Grok 3, we were able to direct the model to produce a significant portion of the book (76.8 percent and 70.3 percent) without jailbreaking.
OpenAI's GPT-4.1 was the most resistant, with only 4% being able to accurately describe the book's content when asked.
The researchers caution that the recall rates mentioned do not represent the maximum possible, and say they have reported their findings to Anthropic, Google DeepMind, OpenAI, and xAI. Only xAI, whose Grok model is currently under fire for generating non-consensual sexual images on demand, has not acknowledged this disclosure.
“At the end of the 90-day disclosure period (December 9, 2025), we found that our procedures were still functional on some of the systems evaluated,” the authors said, without disclosing the system providers involved.
Anthropic discontinued the Claude 3.7 Sonnet as an option for customers on November 29, 2025, but that may not necessarily have been in response to the research findings, but simply that the model was replaced.
The researchers say they will leave the detailed legal analysis of copying model content to others, but “our findings may be relevant to ongoing debates.” ®
