Researchers at Max Plank Propose MIME: Generative AI Models that Take 3D Human Motion Capture and Generate a Plausible 3D Scene that Matches the Motion

Humans are constantly interacting with their surroundings. They move around in space, touch objects, sit in chairs, and sleep in beds. These interactions detail how the scene is set up and where the objects are. A mime is a performer who uses this understanding of relationships to create rich and imaginative 3D environments using only body movements. Can we teach computers to imitate human behavior and create appropriate 3D scenes? Many areas could benefit from this technology, including architecture, gaming, virtual reality, and synthetic data synthesis. there is. For example, there are substantial datasets on his 3D human movements, such as AMASS, but these datasets rarely contain details of his 3D settings in which the data were collected.

Can AMASS be used to create true 3D scenes in all motion? If so, could AMASS be used to create training data containing realistic human-scene interactions? To answer such inquiries, they developed a new technique called MIME (Mining Interaction and Movement to infer 3D Environments) that creates lifelike 3D interior scenes based on 3D human movement. . What makes it possible? The basic assumptions are: (1) Human movement across space indicates the absence of items, essentially defining regions of the screen that are unfurnished. Additionally, this limits the types and positions of her 3D objects when touching the scene. For example, a sitting person must be seated on a chair, sofa, bed, etc.

To bring these intuitions to life, researchers at the Max Planck Institute for Intelligent Systems in Germany and Adobe created MIME, a transformer-based autoregressive 3D scene generation technology. Given an empty floor plan and a human motion sequence, MIME predicts furniture that will come in contact with humans. In addition, it predicts reliable items that do not come into contact with humans, blend in with other objects, and obey the free-space limitations imposed by human movement. Split motion into contact and non-contact snippets to tailor 3D scene creation to human motion. They use POSA to estimate potential contact poses. The non-contact pose projects the vertices of the feet onto the ground to establish the free space of the room and record it as a 2D floor map.

The contact vertices predicted by POSA create a 3D bounding box that reflects the contact pose and the associated 3D human model. Objects that meet the contact and free space criteria are expected to autoregressively use this data as input to the transformer. See Figure 1. They extended the large synthetic scene dataset 3D-FRONT to create a new dataset named 3D-FRONT HUMAN for training MIME. Automatically add people to his 3D scenario, including non-contact people (a series of walking movements and standing people) and contact people (sitting, touching, lying down). To do this, we use static contact poses from RenderPeople scans and motion sequences from AMASS.

MIME creates a realistic 3D scene layout of the input motion during inference and represents it as a 3D bounding box. They select 3D models from the 3D-FUTURE collection based on this arrangement. Then fine-tune the 3D placement based on the geometric constraints between the human position and the scene. Their method produces 3D sets that support human touch and motion while convincingly placing objects in free space, unlike pure 3D scene creation systems like ATISS. Their approach predicts complete scenes rather than individual objects, allowing the development of items that do not interact with people, in contrast to the recent pose-conditional generative model, Pose2Room. They show that their approach works without adjusting the recorded real motion sequences, as they do with PROX-D.

In conclusion, they contribute to:

• An all-new motion conditional generative model for 3D room scenes that auto-regressively creates things that come in contact with people while avoiding occupying motion-defined empty spaces.

• A brand new 3D scene dataset consisting of people and people interacting in free space was created by populating 3D FRONT with motion data from AMASS and static contact/stand poses from RenderPeople.

The code is available on GitHub along with a video demo. There is also a video explanation of the approach.

please check out paper, github, and plan.don’t forget to join 24,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more. If you have any questions regarding the article above or missed something, feel free to email me. Asif@marktechpost.com

featured tools From AI Tool Club

🚀 Check out 100’s of AI Tools at the AI ​​Tools Club