Researchers from Samsung AI Center, Rockstar Games, FAU Erlangen-Nurnberg, and Cinemarsive Labs propose an entirely new technique for image-based modeling that can extract human hair from multiple views of a photo or video frame. Hair reconstruction is one of the most difficult tasks in her 3D modeling of a human being, due to its highly complex geometry, physics and reflectivity. Nevertheless, it is essential for many applications such as games, telepresence, and special effects. 3D polylines (strands) are the most common way to depict hair in computer graphics as they can be used for physical modeling and realistic rendering. Modern image- and video-based systems for human reconstruction often simulate hairstyles using low-degree-of-freedom data structures that are easy to estimate, such as volumetric representations and topology-set meshes.
As a result, these techniques often produce overly smooth hair geometry and can only accurately represent the “shell” of the hairstyle without capturing the hairstyle’s core structure. Accurate strand-based hair reconstruction can be performed using a high-density capture system with a light stage, controlled lighting equipment, and a synchronized camera. Recently, we have relied on systematic or consistent lighting and camera calibration to speed up the reconstruction process and produce stunning results. The latest effort also used manual frame-by-frame annotation of the hair growth direction to create physically reliable reconstructions. Despite the excellent quality of the findings, complex capture settings and cumbersome preprocessing requirements make such techniques impractical for many practical applications.
Several learning-based algorithms for hairstyle modeling use hair priors discovered from strand-based synthetic data to speed up the acquisition process. However, the amount of training dataset is a natural determinant of the accuracy of these approaches. Since most existing datasets contain only a few hundred samples, datasets must be large to adequately handle different human hairstyles, resulting in poor reconstruction quality. This study provides a technique for hair modeling that operates in uncontrolled lighting settings and requires image- or video-based data without additional annotation by the user. That’s why they created his two-step rebuilding process. The first step, coarse volumetric hair restoration, is fully data-driven and uses an implicit volumetric representation. The second step, known as thin strand-based reconstruction, works at the level of individual hair strands and mainly relies on priors discovered from small synthetic datasets. For the hair and bust (head and shoulders) regions, the implicit surface representation is recreated in the first step.
🚀 Build high-quality training datasets, solve NLP machine learning challenges, and develop powerful ML applications with Kili Technology
In addition, we can learn the region of hair growth direction, called the 3D direction, by using differentiable projections to compare the hair directions shown in training images or 2D direction maps. This field is useful for fitting hair shapes more accurately, but its primary use is to limit the second stage optimization of hair clumps. Generate a hair direction map from the input frames using a conventional method based on image gradients.
Pre-trained priors are used in the second stage to generate strand-based reconstructions. They use an enhanced parametric model trained from synthetic data using autoencoders to represent the distribution of individual hair strands and their joints, or the hairstyle as a whole. Through the optimization procedure, therefore, this stage reconciles the coarse hair reconstruction achieved in the previous stage with the learning-based priors. Finally, use a new hair renderer based on soft rasterization to increase the realism of reconstructed hairstyles using differentiable rendering.
In summary, their contributions are:
• Improved training approach prior to stranding
• 3D reconstruction method of human head for chest and hair region, including hair orientation
• Global hairstyle modeling using latent diffusion-based plies that “interface” with parametric strand plies
• A differentiable soft hair rasterization method that produces more accurate reconstructions than previous rendering techniques.
• The strand fitting method combines all of the above factors to provide excellent reconstruction of human hair at the strand level.
They test the technique’s effectiveness against artificial and real-world data using monocular film from smartphones and multi-view photography from 3D scanners operating in unlimited lighting settings.
Please check paper, githuband project page. All credit for this research goes to the researchers of this project.Also, don’t forget to participate 26,000+ ML SubReddit, Discord channeland email newsletterShare the latest AI research news, cool AI projects, and more.
Aneesh Tickoo is a consulting intern at MarktechPost. He is currently pursuing his Bachelor of Science in Data Science and Artificial Intelligence from the Indian Institute of Technology (IIT), Bhilai. He spends most of his time working on projects aimed at harnessing the power of machine learning. His research interest is in image processing and he is passionate about building solutions around it. He loves connecting with people and collaborating on interesting projects.
🔥 Gain a competitive edge with data: Actionable market intelligence for global brands, retailers, analysts and investors. (with sponsor)
