A 3cm metal lens camera that uses AI to sharpen distorted images

Chinese researchers used deep learning techniques to improve the image quality of metalenses.

This method uses artificial intelligence to magnify images, allowing such cameras to perform a wide range of tasks, including complex microscopy applications and use on mobile devices.

A team from Southeast University in China used multiscale convolutional neural networks, a type of machine learning, to improve the resolution, contrast, and distortion of images taken with a small camera they developed.

The camera measures approximately 3 cm × 3 cm × 0.5 cm and has a metal sensor integrated directly into a complementary metal-oxide-semiconductor (CMOS) imaging chip.

According to the researchers, this approach could significantly improve image resolution, contrast, and distortion, potentially significantly improving overall image quality.

Advances using machine learning

Metalenses are incredibly thin optical devices that use nanostructures to control light. They are often only a few atoms thick.

Their small size may allow for incredibly light and compact cameras without traditional optical lenses, but obtaining the necessary image quality with these optics has proven difficult. Masu.

To address this challenge, the team developed a deep learning high-quality imaging method using machine learning.

The camera used in this study was previously developed by the researchers and uses a metalens with 1000 nm-tall cylindrical silicon nitride nanoposts. Without the use of additional optical components, the metalens focuses light directly onto the CMOS imaging sensor.

Although this design resulted in a very small camera, the compact architecture limited image quality. As a result, the researchers decided to investigate whether images could be enhanced using machine learning.

Deep learning uses artificial neural networks with multiple layers to automatically learn features from data to make complex decisions and predictions. The researchers employed this technique using a convolutional imaging model to generate a large dataset of pairs of high- and low-quality images.

These pairs were then used to train a multiscale convolutional neural network to identify the characteristics of each image type, allowing it to transform low-quality images into high-quality images.

“An important part of this research was developing a method to generate the large amounts of training data needed for the neural network's learning process. Once training is complete, low-quality images can be sent from the device to the neural network. processing, and immediately obtain high-quality image results,” said Ji Chen from Southeast University in China, who led the study. statement.

Next generation metalens imaging

The researchers tested their new deep learning method by applying it to 100 test photos. They investigated peak signal-to-noise ratio and structural similarity index, two metrics widely used in image processing.

According to the research team, there was a noticeable improvement in both measurements for the images analyzed by the neural network. They also demonstrated how this method can rapidly generate high-quality image data close to that directly observed through experimentation.

The study states, “Our method improves image resolution, contrast, and distortion all resulting in structural similarity index measurements (SSIM) above 0.9, significantly improving overall image quality, and peak signal “The noise-to-noise ratio has been improved.” (PSNR) 3dB or more. ”

Researchers are now focused on designing metalenses with advanced features such as color and wide-angle imaging. They are also developing neural network techniques to improve the image quality of these sophisticated metalenses.

The team emphasized that making the technology commercially available will require new assembly techniques to integrate the metalens into smartphone camera modules, as well as image-enhancing software customized specifically for mobile devices. There is.

“Leveraging deep learning techniques to optimize the performance of metalenses represents a pivotal development trajectory. We predict that machine learning will become an important trend in advancing photonics research.” said Mr. Chen.

Details of the team's research were published in the journal optical characters.

abstract

Due to their ultra-lightweight, ultra-thin, and flexible design, metalenses show great potential in the development of highly integrated cameras. However, the performance of metalens-integrated cameras is limited by the fixed architecture. Here, we proposed a high-quality imaging method based on deep learning to overcome this limitation. We used a multiscale convolutional neural network (MSCNN) to train extensive pairs of high- and low-quality images obtained from a convolutional imaging model. Our method all improved image resolution, contrast, and distortion, resulting in remarkable overall image quality with SSIM > 0.9 and PSNR > 3 dB. Our approach allows cameras to combine the benefits of advanced integration with enhanced imaging performance, revealing great potential for future breakthrough imaging technologies.