AI technology reconstructs the interaction of 3D hand objects from video even when elements become obscure

UNIST researchers have developed AI technology that can reconstruct the 3D (3D) representation of unfamiliar objects manipulated with both hands, and have developed a simulation surgical scene that includes intertwined hands and medical devices. This advancement allows for highly accurate augmented reality (AR) visualizations, further improving real-time interaction capabilities.

A team led by Professor Seungryul Baek of the Unist School of Artificial Intelligence has introduced the Bamanual Interaction 3D Gaussian Splatting (Bigs), an innovative AI model that allows users to visualize complex interactions between 3D hands and objects using only a single RGB video input.

This technology allows for real-time reconstruction of complex hand object dynamics, even when objects are unfamiliar or partially obscure. This investigation has been published in arxiv Preprint server.

The traditional approach of this domain is limited to recognizing only one hand at a time, responding only to pre-scanned objects, and limiting applicability in realistic AR and VR environments.

In contrast, Bigs can reliably predict full objects and handshapes, even in scenarios where parts are hidden or blocked. It can also be done without the need for depth sensors or multiple cameras that rely solely on one RGB camera.

The core of this AI model is based on 3D Gaussian splatting. This is a technique that describes the shape of an object as a cloud of points with a smooth Gaussian distribution.

Unlike point cloud methods that generate sharp boundaries, Gaussian splatting allows for the natural reconstruction of contact surfaces and complex interactions.

This model further addresses the challenge of occlusion by aligning multiple hand instances with a regular Gaussian structure, employing a pre-trained diffusion model of score distillation sampling (SDS) to allow accurate reconstruction of invisible surfaces, including the back of the object.

Extensive experiments using international datasets such as Arctic and HO3DV3 have shown that Bigs outperforms existing technologies in accurately capturing hand posture, object shape, contact interactions and quality rendering. These features have great promise for virtual and augmented reality, robotic control, and telesurgical simulation applications.

This study was conducted with contributions from the first author Jeongwan on, along with Kyeonghwan Gwak, Gunyoung Kang, Junuk Cha, Soohyun Hwang, and Hyein Hwang.

Professor Baek said, “This advancement is expected to facilitate the reconstruction of real-time interactions in a variety of fields, including VR, AR, robotics control and telesurgery training.”