Score smoothing facilitates manifold recovery
In the real world, complex data such as high-resolution images exists in a high-dimensional pixel space rather than a simple one-dimensional world. But most of that space is just random noise that is meaningless to the human eye. Only a small fraction of the data points in that space correspond to recognizable images, and they exist in a so-called data manifold (like a sheet pushed into a larger space). The shape and location of the data manifold are not known in advance to the model. Therefore, image generation can be thought of as a manifold recovery task. The model must infer what the hidden data manifold looks like based on a finite number of training data sampled from it and find new points on the manifold that correspond to novel and meaningful images. To achieve this, we found that score smoothing is important for the diffusion model.
Notably, in a multidimensional setting, the effect of score smoothing appears directionally dependent. Along a direction parallel (or “tangential”) to the hidden data manifold, a similar slowing effect as in the one-dimensional scenario occurs. However, along the direction towards the manifold, the “perfect” score function is already relatively smooth (in fact, it would be just a straight line if the manifold was flat), and further smoothing won’t make much of a difference.
Therefore, instead of putting a brake on the flow of particles in all directions (which would cause particles to stagnate in a noisy empty space and make the final image blurry), score smoothing does not slow down the movement of particles towards the manifold, but only reduces their tendency to collapse along tangential directions towards the training data. In this way, the model achieves a balance between quality and novelty. In other words, the image looks both realistic (because it successfully arrived at a meaningful data manifold) and novel (because it fell into the blank space between the original training data points).
