New Radiative Transfer Modeling Framework Powers Deep Learning for Plant Phenotyping

Using Helios 3D plant modeling software, the research team developed a radiative transfer modeling framework that simulates RGB, multi/hyperspectral, thermal, and depth camera imagery along with fully resolved reference labels. This innovative method significantly reduces the need for laborious manually annotated datasets.

The framework can generate high-quality synthetic images, enabling efficient training of deep learning models for high-throughput plant phenotyping, enhancing crop trait analysis, and providing metrology tools to advance agricultural research and remote sensing applications.

Integrating remote and proximity sensing techniques facilitates high-throughput monitoring of plant systems and provides comprehensive insights into plant function. Advances in these technologies have resulted in an abundance of high-resolution imagery, but the challenge remains of linking this data to actionable plant traits. Current methods fall short in the labor-intensive data annotation and multi-modal data reconciliation required.

Research published in Plant phenomics The project, which will begin on 30 May 2024, aims to address these challenges by developing a new 3D radiative transfer modelling framework.

In this study, the radiative transfer model was validated using different SKILL scores to evaluate its accuracy in simulating the radiation absorbed and reflected radiation fluxes by objects. The SKILL scores for the different tests (brfpp_uc_sgl, brfpp_co_sgl, brfop, and fabs) were 98.00, 92.65, 97.52, and 99.98, respectively, demonstrating the high accuracy of the model.

In addition, R² Camera calibration values ​​ranged from 0.864 to 0.930, indicating effective distortion recovery and color correction. Synthetic images generated using the model, including RGB, NIR, and thermal images, showed high visual similarity to the real images, confirming the model's ability to generate high-quality annotated plant images. These results validate the effectiveness of the model in simulating complex scenes and establish it as a robust vehicle for high-throughput plant phenotyping and machine learning model training.

Tong Lei, the lead researcher of the study, asserts that Helios provides a simulation environment that allows the generation of 3D geometric models of plants and soils with random variations, as well as the specification and simulation of their properties and functions. This approach differs from traditional computer graphics renderings because it establishes a crucial link with the underlying biophysical processes of plants by explicitly modeling the physics of radiation transfer.

In summary, this study presents a radiative transfer modeling framework that uses Helios 3D software to simulate plant images, including RGB, multispectral, thermal, and depth images, with detailed annotations. This framework reduces the need for manual data collection and improves the training of deep learning models for plant phenotyping.

Future developments will increase the flexibility of the model and incorporate more complex processes, allowing efficient analysis of plant traits and physiological status, advancing high-throughput phenotyping and agricultural research.