The researchers found that the accumulation of AI-generated content on the web could cause machine learning models to “break down” unless the industry can mitigate the risks.
A team of researchers from Oxford University found that training future models with AI-generated datasets could result in them generating gibberish, a concept known as model collapse. In one example, a model started with a text about medieval European architecture and by the ninth generation was spitting out gibberish about jackrabbits.
In a paper published yesterday in Nature, research led by Google DeepMind and Oxford University postdoctoral researcher Ilya Shmailov found that AI can, for example, fail to pick up on lines of text that are less common in the training dataset, meaning that subsequent models trained on that output can't pick up on those nuances. Thus, training a new model on the output of a previous model can get you stuck in a recursive loop.
“Long-term poisoning attacks against language models are not new,” the paper states. “For example, we have seen the creation of click, content, and troll farms – a type of human 'language model' that serves to mislead social networks and search algorithms. The negative impact of these poisoning attacks on search results has led to changes in search algorithms; for example, Google downgraded farmed articles to focus on content created by trusted sources, such as in the education sector, while DuckDuckGo removed them entirely. What changes with the advent of LLM is that when this poisoning becomes automated, it can be done at scale.”
In an accompanying article, Emily Wenger, an assistant professor of electrical and computer engineering at Duke University, explained how the model breaks down using the example of a system that generates images of dogs.
“AI models tend to represent the most common dog breeds in their training data and are likely to represent golden retrievers more than petit basset griffon vendeen, given the relative prevalence of the two breeds,” she said.
“The problem is compounded when subsequent models are trained on AI-generated data sets that over-represent golden retrievers. After enough cycles of over-representing golden retrievers, the model forgets that lesser-known breeds such as Petit Basset Griffon Vendeen exist and only produces photos of golden retrievers. Eventually, the model becomes dysfunctional and is unable to generate meaningful content.”
She acknowledges that overrepresentation of golden retrievers may not be a bad thing, but the process of collapse is a serious problem for meaningful, representative artifacts that include less common ideas and writing styles. “This is the underlying problem of model collapse,” she said.
One existing approach to mitigating this problem is to watermark AI-generated content; however, these watermarks can be easily removed from AI-generated images. Sharing watermark information would also require significant coordination between AI companies, which “may not be practical or commercially feasible,” Wenger said.
Shumailov and his colleagues say it's not impossible to train models using AI-generated data, but the industry needs to establish effective means of filtering it.
“The need to distinguish data generated by LLMs from other data raises questions about the provenance of content crawled from the Internet. It is unclear how to track content generated by LLMs at scale,” the paper states.
“One option is to coordinate across the community so that the various parties involved in creating and deploying LLMs share the information needed to answer questions about provenance. Otherwise, it may become increasingly difficult to train new versions of LLMs without access to data crawled from the internet before the technology was adopted at scale, or without direct access to data generated by humans at scale.”
No such thing Registry We can enjoy the perspective of hindsight, but perhaps someone should have thought about this before the industry and its investors bet all their money on LLM.®
