The creative industry has witnessed a new era of possibilities with the advent of generative models—computational tools that can generate text and images based on training data. Inspired by these advances, researchers at Stanford University, the University of California, Berkeley, and Adobe Research have introduced new models that can seamlessly insert specific humans into various scenes with striking realism. .
The researchers employed a self-supervised training approach to train the diffusion model. This generative model transforms the “noise” into the desired image by adding and reversing the process of “discarding” the training data. The model was trained on videos featuring humans moving within various scenes, from each video he randomly selected two frames. The human in the first frame is masked, and the model realistically reconstructs the individual in the masked frame using the unmasked individual in her second frame as a conditioning signal.
Through this training process, the model learned to infer potential poses from the context of the scene, change the pose of the person, and seamlessly integrate them into the scene. The researchers found that their generative model performed very well at placing individuals within a scene, producing highly realistic-looking edited images. This model’s prediction of affordances (the perception of the likelihood of actions and interactions in the environment) outperformed previously introduced non-generative models.
This finding has great potential for future research in affordance perception and related fields. By identifying potential interaction opportunities, we can contribute to advances in robotics research. Furthermore, the practical application of this model extends to the creation of realistic media such as images and videos. Integrating this model into creative software tools has the potential to enhance image editing capabilities and support artists and media creators. Additionally, this model can be incorporated into photo-editing smartphone applications, allowing users to easily and realistically insert people into their photos.
Researchers have identified several avenues for future exploration. They aim to incorporate greater controllability into the generated poses and look at generating realistic human motion within a scene rather than static images. In addition, we seek to improve the efficiency of our model and extend our approach beyond humans to encompass all objects.
In conclusion, the introduction of a new model by the researchers enabled the realistic insertion of humans into the scene. By leveraging generative models and self-supervised training, this model shows excellent performance in imparting perception and has various potential applications in creative industries and robotics research. Future research will focus on improving and extending the functionality of the model.
please check out paper. don’t forget to join 22,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email me. Asif@marktechpost.com
🚀 Check out 100’s of AI Tools at the AI Tools Club
Niharika is a technical consulting intern at Marktechpost. She is in her third year of undergraduate studies and is currently completing her Bachelor’s degree at the Indian Institute of Technology (IIT), Kharagpur. She is a very passionate person who has a keen interest in machine learning, data her science, AI and avid reader of the latest developments in these fields.
