600 image dataset advances machine learning for fringe projection profilometry

Fringe projection profilometry (FPP) increasingly relies on machine learning, but progress is hampered by limited training data and a lack of standardized evaluation methods. Anush Lakshman S, Adam Haroon, and Beiwen Li of the Iowa State University School of Mechanical Engineering are addressing this challenge by presenting the first open-source, photorealistic synthetic dataset for FPP. The dataset was generated using Isaac Sim and consists of 15,600 fringe images and 300 corresponding depth reconstructions representing 50 different objects. This work significantly advances the field by benchmarking four different neural network architectures: UNet, Hformer, ResUNet, and Pix2Pix, revealing fundamental limitations in direct fringe-to-depth mapping, and revealing errors that consistently represent a significant portion of the depth of common objects. The availability of this resource and standardized evaluation protocols will facilitate systematic comparisons and accelerate the development of learning-based FPP methods.

Synthetic dataset accelerates fringe projection profilometry learning

Scientists have demonstrated significant advances in machine learning applications of fringe projection profilometry (FPP), an important non-destructive technique used for robot scanning, manufacturing inspection, and 3D printing optimization. This study addresses critical limitations hindering progress in this field: large and diverse datasets of machine learning models and the lack of standardized evaluation protocols. The team accomplished this by building the first open-source photorealistic synthetic dataset for FPP, consisting of 15,600 fringe images and 300 depth reconstructions from 50 diverse objects, generated using NVIDIA Isaac Sim. This breakthrough leverages the VIRTUS-FPP system built within NVIDIA Isaac Sim and integrates advanced ray tracing and physics simulation to create a virtual FPP environment with end-to-end camera and projector modeling.

The researchers meticulously calibrated this virtual system using a procedurally generated circular board, achieving sub-pixel accuracy with a stereo reprojection error of 0.055506 pixels and a projector error of 0.048609 pixels. This accurate calibration is the basis for generating accurate ground truth data for training and evaluating machine learning algorithms. This dataset incorporates objects retrieved from the YCB dataset and the NVIDIA Physical AI Warehouse, ensuring a wide range of shapes and surface properties. This suggests that the fundamental limitation lies not in the network architecture itself, but in the approach of directly mapping fringes to depth without explicit topological information. This study reveals that the reconstruction error consistently approaches 75-95% of the typical object depth range, indicating that the network primarily performs coarse shape pretraining rather than precise geometric details. This work establishes a standardized evaluation framework that enables systematic comparison and development of learning-based FPP approaches.

The researchers employed a GPU-accelerated pipeline that can capture patterns at three frames per second, which is faster than traditional methods. By generating a dataset with perfect ground truth and comprehensive benchmarking protocols, this research unlocks new possibilities for real-time single-shot 3D reconstruction, paving the way for more efficient and accurate applications in robotics, automation, and advanced manufacturing. This resource will undoubtedly accelerate innovation in the fields of structured light metrology and machine learning.

Virtual FPP dataset and NVIDIA Isaac Sim framework

Researchers have addressed machine learning limitations in fringe projection profilometry (FPP) by developing a new open-source synthetic dataset and benchmark framework. This work is centered around VIRTUS-FPP, a physically-based virtual FPP system built within NVIDIA Isaac Sim that integrates OptiX ray tracing, PhysX physics, and Universal Scene description for 3D compositing. The system enables the generation of 15,600 fringe images and 300 corresponding depth reconstructions representing 50 diverse objects with known ground truth, making it an important resource for training and evaluating machine learning algorithms. A virtual FPP system uses a calibrated camera and projector pair. The camera utilizes a 960 × 960 resolution pinhole primitive with a focal length of 50 cm, and the projector is modeled as a 0.625 m x 0.5 m rectangular light source emitting at 40 nits.

The projector is strategically placed 0.1 m below and 0.125 m to the left of the camera to optimize the triangulation geometry essential for accurate depth mapping. The key innovation lies in modeling the projector using an inverted camera model, allowing for precise dimensional correspondence of the projected stripes regardless of distance and circumventing physical hardware limitations. All objects in the dataset consistently exhibit matte material properties, a roughness of 0.95, a specular reflectance of 0.15, and an AO-to-diffuse ratio of 0.95, reflecting the characteristics of a typical structured light scanning scenario. Virtual calibration was achieved using a procedurally generated 5×9 asymmetric circular board with 10mm diameter circles spaced 20mm apart, capturing 18 poses and generating 936 calibration images within 5 minutes, achieving a throughput of 10,530 images per hour.

This calibration process resulted in sub-pixel accuracy as demonstrated by a stereo reprojection error of 0.055506 pixels and a projector error of 0.048609 pixels. Data collection included rotating each of the 50 objects obtained from the YCB dataset and the NVIDIA Physical AI warehouse by 60 degrees around the vertical axis, generating six 50% overlapping viewpoints for each object. An 18-step phase shift sequence was then captured at each viewpoint and processed using standard N-step phase shift, Gray code time unwrapping, and triangulation to produce a depth map with an accuracy of 1.2 × 10−3 mm over a range of 80 mm. The resulting dataset includes not only fringe images and depth maps, but also regularization parameters and ground truth mesh geometry, facilitating comprehensive evaluation and development of learning-based FPP techniques.

Benchmark neural network performance on photorealistic datasets

Scientists have achieved a breakthrough in fringe projection profilometry (FPP) through the creation of the first open-source photorealistic synthetic dataset. The dataset generated using NVIDIA Isaac Sim consists of 15,600 fringe images and 300 depth reconstructions representing 50 diverse objects. This rich resource addresses significant limitations in machine learning for FPP: large and diverse datasets and the lack of standardized evaluation protocols. In this work, we leverage VIRTUS-FPP, a physically-based virtual FPP system, to provide accurate ground truth data for benchmarking. Despite significant differences in architectural design, these models showed comparable accuracy in reconstructing depth from stripes. The virtual system utilizes a calibrated camera and projector pair. The camera has a resolution of 960 × 960 and a focal length of 50 cm, and the projector emits light at 40 nits with a pattern resolution of 912 × 1140 pixels. Measurements confirm that the reconstruction error consistently approaches 75-95% of the typical object depth range (approximately 80 millimeters).

This discovery reveals a fundamental limitation of mapping fringe patterns directly to depth without incorporating explicit phase information. The research team recorded a stereo reprojection error of 0.055506 pixels and a projector error of 0.048609 pixels during the virtual calibration process, ensuring sub-pixel accuracy. The dataset composition includes objects sourced from the YCB dataset and the NVIDIA Physical AI Warehouse, covering a variety of shapes and complexities. This breakthrough provides a standardized evaluation framework to enable systematic comparison and development of learning-based FPP approaches.

VIRTUS-FPP’s innovative projector modeling, based on an inverted camera model, enables accurate dimensional matching of projected stripes without hardware constraints. The system captured 936 calibration images in just 5 minutes and demonstrated a throughput of 10,530 images per hour. This research establishes a powerful tool to advance research in 3D reconstruction, robotic scanning, and manufacturing inspection.

Fringe data limits semantic shape learning

In this study, we introduce a novel large-scale synthetic dataset consisting of 15,600 fringe images and 300 depth reconstructions of 50 diverse objects designed to accelerate machine learning advances in fringe projection profilometry (FPP). By benchmarking four different neural network architectures on this dataset, this study demonstrated remarkable consistency in performance, with all models achieving root mean square errors of 58 to 77 millimeters. The results of this study reveal the fundamental limitations inherent in mapping fringe patterns directly to depth without incorporating explicit phase information, with reconstruction errors often reaching 75-95% of typical object depths. This suggests that current approaches primarily perform semantic shape pretraining rather than accurate geometric representations, and that the limiting factor is the lack of sufficient information rather than model design.

The authors acknowledge that the relatively weak performance of the ResUNet architecture may be due to overfitting with limited training samples. Future research will focus on integrating topological information into the learning process and exploring simulation-to-realistic conversion techniques, multi-view fusion, and uncertainty quantification. By establishing a standardized benchmarking framework and comprehensive synthetic datasets, this effort provides a solid foundation for data-driven development of robust FPP systems applicable to manufacturing, biomedical imaging, and automated testing.