How optimal transport theory improves generative models

A collaborative study led by Ito Soke of the University of Tokyo shows that nonequilibrium thermodynamics, a field of physics dealing with ever-changing systems, explains why optimal transport theory is a mathematical framework for optimal change to reduce costs. This finding provides a new thermodynamic approach to machine learning research, as nonequilibrium thermodynamics has not yet been fully utilized in the design of generative models. The findings were published in the journal Physics Review x.

Image generation has been dramatically improved in recent years. Videos of celebrities eating a bowl of spaghetti representing the latest technology a few years ago are not even qualifying for good today. The algorithm in which power image generation is called a diffusion model contains randomness known as “noise.”

During the training process, noise is introduced into the original data through diffusion dynamics. During the generation process, the model must eliminate noise and generate new content from the noisy data. This is achieved by considering the dynamics of time inversion, as if to play the video inverted. One of the arts and sciences of building models that generate high-quality content is to specify when and how much noise will be added to the data.

“The choice of diffusion dynamics, also known as noise schedules, has been controversial in the diffusion model since its inception,” says Ito, the lead researcher. “The optimal transport dynamics have been shown empirically to be useful in diffusion models, but it has not been theoretically demonstrated why this is the case.”

Although the diffusion model was originally inspired by nonequilibrium thermodynamics, optimal transport theory is closely related to this region, previous studies overlook this link. So questions arose. Can non-equilibrium thermodynamics provide a theoretical framework for why optimal transport dynamics work so well in diffusion models?

Recent advances in thermodynamic trade-off relationships, techniques to explain the relationship between thermodynamic dissipation and the rate of system change, have proven to be extremely useful. Using this technique, researchers have derived inequality between thermodynamic dissipation and the robustness of data generation in diffusion models. They used newly derived inequalities to show that optimal transport dynamics ensure the most robust data generation.

“One surprising result is that our boundaries are tight within a certain magnitude for real-world image generation scenarios,” explains Ito. “This shows that our inequality is useful not only for understanding the optimal protocol for diffusion models, but also for analyzing the practical application of image data generation.”

Plus, there is another surprising aspect to this project. Ito said, “The first and second authors of the paper are undergraduates, and this study was partially conducted as part of the class they enrolled. In particular, first author Kotaro Ikeda contributed greatly to this study, from numerical calculations to theoretical analysis.

“We hope that our results will raise awareness of the importance of non-equilibrium thermodynamics in the machine learning community. We, including the next generation, continue to explore its utility in understanding biological and artificial information processing.”