With millions of images and video content posted every day, visual filters have become an integral part of social media platforms, allowing users to enhance and customize their video content with various effects and adjustments. These filters have revolutionized the way we communicate and share experiences, giving us the ability to create visually appealing and engaging content that captures the attention of our audience.
Moreover, with the rise of AI, these filters have become even more sophisticated, allowing you to manipulate video content in ways previously impossible with just a few clicks. AI-powered video filters can automatically adjust the lighting, color balance, and other aspects of a video, allowing creators to achieve a professional-quality look without the need for extensive technical knowledge.
Although these filters are very powerful, they are designed with predefined parameters and cannot produce consistent color styles for different looking images. Therefore, careful tuning by the user is required. To address this issue, a color style transfer technique was introduced that automatically maps color styles from one properly retouched image (i.e. style image) to another (i.e. input image).
However, existing techniques produce results subject to artifacts such as color and texture mismatches, and require significant time and resources to execute. For this reason, a new framework for color style transfer was developed called Neural Presets.
The following diagram shows an overview of the workflow.
The proposed method differs from the current state-of-the-art, which uses deterministic neural color mapping (DNCM) instead of a convolutional model of color mapping. DNCM utilizes an image-adaptive color mapping matrix that multiplies pixels of the same color to produce a specific color, effectively eliminating unrealistic artifacts. Additionally, DNCM works independently on each pixel, requiring a small memory footprint and supporting high-resolution inputs. Unlike traditional 3D filters that rely on regression of tens of thousands of parameters, DNCM can model arbitrary color mappings using only a few hundred learnable parameters.
Neural Preset works in two different stages, allowing you to quickly switch between different styles. The underlying structure relies on an encoder E to predict the parameters used in the normalization and stylization stages.
The first stage creates an nDNCM from the input image, normalizes the colors, and maps the image to a color style space that represents the content. The second stage builds the sDNCM from the style image. This will style the normalized image to the desired target color style. This design ensures that the sDNCM parameters can be saved as color style presets and used with different input images. Additionally, the input image can be styled using different color style presets after normalization with nDNCM.
A comparison between the proposed approach and the state-of-the-art is shown below.
According to the author, Neural Preset outperforms state-of-the-art methods in many aspects, including accurate results for 8K images, consistent color style transfer results across video frames, and ~28x speedup on Nvidia RTX3090 GPU. Significantly better. Real-time performance at 4K resolution.
That was an overview of Neural Preset, an AI framework for real-time color consistent high-quality style transfer.
If you’re interested in this work, or want to learn more, you can find links to the papers and project pages.
check out paper and planAll credit for this research goes to the researchers of this project.Also, don’t forget to participate Our 16k+ ML SubReddit, cacophony channeland email newsletterWe share the latest AI research news, cool AI projects, and more.
Daniele Lorenzi has an M.Sc. He completed his ICT Bachelor’s Degree in Internet and Multimedia Engineering at the University of Padua, Italy in 2021. He has his Ph.D. Alpen-Adria-Universität (AAU) Candidate for the Information Technology Institute (ITEC) in Klagenfurt. He currently works at the Christian Doppler Laboratory ATHENA and his research interests include adaptive video streaming, immersive media, machine learning and his QoS/QoE assessment.
