AI trained to draw inspiration from images instead of copying them

Very famously, powerful new artificial intelligence models can sometimes cause problems, such as hallucinating false information or remembering someone else's work and presenting it as their own. To address the latter, researchers led by a team at the University of Texas at Austin developed a framework to train AI models on images that are corrupted beyond recognition.

DALL-E, Midjourney, Stable Diffusion is one of the text-to-image diffusion generation AI models that can transform any user text into highly realistic images. All three are currently facing lawsuits from artists who claim that the samples produced reproduce their work. Models trained on billions of non-public image-text pairs can produce high-quality images from text prompts, but may utilize and reproduce copyrighted images. .

A newly proposed framework “Ambient Diffusion'' circumvents this problem by accessing only corrupted image-based data to train the diffusion model. Initial work suggests that the framework can continue to generate high-quality samples without ever seeing anything recognizable as the original source image.

Ambient Diffusion was originally presented at NeurIPS, a machine learning conference, in 2023 and has since been adapted and expanded upon. The follow-up paper “Consistent Diffusion Meets Tweedie” arXiv The preprint server has been accepted to the 2024 International Conference on Machine Learning. Working with Constantinos Daskalakis from the Massachusetts Institute of Technology, the team extended their framework to train the diffusion model on a dataset of images corrupted by other types of noise rather than simply masking pixels, as well as a larger dataset.

“This framework could also be useful in science and medicine,” said Adam Klivans, a computer science professor who worked on the study. “This could apply to basically any study where obtaining complete, intact data is expensive or impossible, from images of black holes to certain types of MRI scans.”

Crivances. Alex Dimakis, professor of electrical and computer engineering; Other collaborators at the Multi-Institutional Institute for Basic Machine Learning, directed by two of his faculty members at UT, first trained a diffusion model on his 3,000 image set of celebrities, then We experimented by using that model to generate new samples.

In our experiments, a diffusion model trained on clean data blatantly copied the training examples. However, when the researchers corrupted the training data, randomly masked up to 90% of the individual pixels in the images, and retrained the model with a new approach, the samples produced remained of high quality; They looked very different. The model can still generate human faces, but the generated faces are sufficiently different from the training images.

“Our framework allows us to control the trade-off between memory and performance,” said Yannis Dallas, a graduate student in computer science who led the study. “As the level of corruption encountered during training increases, the memorization power of the training set decreases.”

The researchers said this presents a solution that may vary in performance but never outputs noise. The framework provides an example of how academic researchers are advancing artificial intelligence to meet society's needs, and the University of Texas at Austin has declared 2024 the “Year of AI.” This is the school's main theme this year.

The research team included members from the University of California, Berkeley and the Massachusetts Institute of Technology.