About
15 May 2024
Researchers use artificial intelligence to boost image quality of metalens camera
Advance paves the way for ultra-thin cameras for applications from microscopy to mobile devices
Caption: Researchers used deep learning techniques to enhance the image quality of a camera with a metalens integrated directly onto a CMOS imaging chip (left). The metalens manipulates light using an array of 1000-nm tall cylindrical silicon nitride nano-posts (right).
Credit: Ji Chen, Southeast University
WASHINGTON — Researchers have leveraged deep learning techniques to enhance the image quality of a metalens camera. The new approach uses artificial intelligence to turn low-quality images into high-quality ones, which could make these cameras viable for a multitude of imaging tasks including intricate microscopy applications and mobile devices.
Metalenses are ultrathin optical devices — often just a fraction of a millimeter thick — that use nanostructures to manipulate light. Although their small size could potentially enable extremely compact and lightweight cameras without traditional optical lenses, it has been difficult to achieve the necessary image quality with these optical components.
“Our technology allows our metalens-based devices to overcome the limitations of image quality,” said research team leader Ji Chen from Southeast University in China. “This advance will play an important role in the future development of highly portable consumer imaging electronics and can also be used in specialized imaging applications such as microscopy.”
In Optica Publishing Group journal Optics Letters, the researchers describe how they used a type of machine learning known as a multi-scale convolutional neural network to improve resolution, contrast and distortion in images from a small camera — about 3 cm × 3 cm × 0.5 cm — they created by directly integrating a metalens onto a CMOS imaging chip.
“Metalens-integrated cameras can be directly incorporated into the imaging modules of smartphones, where they could replace the traditional refractive bulk lenses,” said Chen. “They could also be used in devices such as drones, where the small size and lightweight camera would ensure imaging quality without compromising the drone’s mobility.”
Enhancing image quality
The camera used in the new work was previously developed by the researchers and uses a metalens with 1000-nm tall cylindrical silicon nitride nano-posts. The metalens focuses light directly onto a CMOS imaging sensor without requiring any other optical elements. Although this design created a very small camera the compact architecture limited the image quality. Thus, the researchers decided to see if machine learning could be used to improve the images.
Deep learning is a type of machine learning that uses artificial neural networks with multiple layers to automatically learn features from data and make complex decisions or predictions. The researchers applied this approach by using a convolution imaging model to generate a large number of high- and low-quality image pairs. These image pairs were used to train a multi-scale convolutional neural network so that it could recognize the characteristics of each type of image and use that to turn low-quality images into high-quality images.
“A key part of this work was developing a way to generate the large amount of training data needed for the neural network learning process,” said Chen. “Once trained, a low-quality image can be sent from the device to into the neural network for processing, and high-quality imaging results are obtained immediately.”
Caption: The images show a comparison of ground truth, low-quality images, and neural network outputs for four test images. The first row represents simulation results, and the second row represents the experimental results. The blue, red and yellow boxes show close ups of details in the images.
Credit: Ji Chen, Southeast University
Applying the neural network
To validate the new deep learning technique, the researchers used it on 100 test images. They analyzed two commonly used image processing metrics: the peak signal-to-noise ratio and the structural similarity index. They found that the images processed by the neural network exhibited a significant improvement in both metrics. They also showed that the approach could rapidly generate high-quality imaging data that closely resembled what was captured directly through experimentation.
The researchers are now designing metalenses with complex functionalities — such as color or wide-angle imaging — and developing neural network methods for enhancing the imaging quality of these advanced metalenses. To make this technology practical for commercial application would require new assembly techniques for integrating metalenses into smartphone imaging modules and image quality enhancement software designed specifically for mobile phones.
“Ultra-lightweight and ultra-thin metalenses represent a revolutionary technology for future imaging and detection,” said Chen. “Leveraging deep learning techniques to optimize metalens performance marks a pivotal developmental trajectory. We foresee machine learning as a vital trend in advancing photonics research."
Paper: Y. Zhang, Y. Wu, C. Huang, Z.-W. Zhou, M. Li, Z. Zhang, J. Chen, “Deep-learning enhanced high-quality imaging in metalens-integrated camera,” Opt. Lett., Vol. 49, Issue 10, pp. 2853-2856 (2024).
DOI: doi.org/10.1364/OL.521393
About Optica Publishing Group
Optica Publishing Group is a division of the society, Optica, Advancing Optics and Photonics Worldwide. It publishes the largest collection of peer-reviewed and most-cited content in optics and photonics, including 18 prestigious journals, the society’s flagship member magazine, and papers and videos from more than 835 conferences. With over 400,000 journal articles, conference papers and videos to search, discover and access, our publications portfolio represents the full range of research in the field from around the globe.
About Optics Letters
Optics Letters has been publishing high-impact research in the field of photonics for over 45 years and offers rapid dissemination of new results in all areas of optical science with short, original, peer-reviewed communications. Optics Letters accepts papers that are noteworthy to a substantial part of the optics community. Published by Optica Publishing Group and led by Editor-in-Chief Miguel Alonso, Institut Fresnel, École Centrale de Marseille and Aix-Marseille Université, France, University of Rochester, USA. For more information, visit Optics Letters.
Media Contact
