Trilobite-inspired neural nanophotonic light-field camera with extreme depth-of-field

Trilobites, extinct arthropods, exhibited diverse visual systems. *Dalmanitina socialis*, in particular, possessed unique bifocal compound eyes with two lens units of different refractive indices. This structure allowed for simultaneous focusing on near and far objects, suggesting sensitivity to light-field information and a large DoF. Current light-field cameras struggle to balance DoF and spatial resolution. Microlens arrays, while offering large DoF, suffer from low resolution. Conversely, high-resolution designs compromise DoF. Existing techniques to extend DoF in conventional imaging (aperture shrinking, focal sweeping, wavefront coding) trade off other crucial aspects like light throughput and time resolution. Metasurface optics, offering novel functionalities, have shown promise in advanced imaging, including depth sensing and polarization imaging. This research aims to create a high-resolution light-field camera with an exceptionally large DoF by leveraging the trilobite's visual system and integrating nanophotonics with computational photography. The success of such a system would represent a significant advancement in imaging technology, with applications in various fields.

The literature extensively covers light-field imaging and its challenges related to DoF and resolution. Early light-field cameras used microlens arrays at the focal plane, leading to a large DoF but low resolution. Shifting the microlens array away from the focal plane improved resolution at the cost of DoF. Multifocal microlens arrays attempted to increase DoF but compromised resolution. Conventional imaging techniques to enhance DoF, like aperture control and wavefront coding, have drawbacks in light throughput and speed. The use of metasurfaces in imaging has shown promise, demonstrating capabilities in depth sensing, polarization imaging, and achromatic light-field imaging. However, combining the advantages of metasurfaces with a large DoF and high resolution remains a challenge. This work addresses these limitations by drawing inspiration from nature and using innovative computational methods.

This study designed and fabricated a nanophotonic light-field camera inspired by the *Dalmanitina socialis* trilobite's bifocal eyes. The core component is a spin-multiplexed metalens array composed of TiO2 nanopillars on a SiO2 substrate. Each metalens focuses light at different focal lengths depending on the incident light's polarization (left-circularly polarized (LCP) and right-circularly polarized (RCP)). The design uses a Jones matrix to achieve spin-dependent phase modulation, focusing LCP light at one distance and RCP light at another. The metalens array was fabricated using electron-beam lithography and atomic layer deposition. Optical characterization confirmed the spin-multiplexed bifocality and measured focal lengths, transmission efficiency, and focusing efficiency across the visible spectrum. The camera system integrates the metalens array with a primary lens and image sensor. Ray tracing models the optical system's behavior, considering chromatic dispersion. A key element is the development of a multiscale convolutional neural network to correct for optical aberrations. The network is trained on a dataset generated from calibrated PSFs obtained by imaging a pinhole at various distances. This dataset is augmented by rotating and resizing PSFs to increase robustness. The network processes captured light-field images, eliminating aberrations and generating an all-in-focus light-field image. Light-field processing techniques, like disparity estimation, are then used to generate refocused images at different depths. The system’s performance is evaluated using a USAF 1951 resolution chart at various distances, and the angular resolution is calculated as a function of depth.

The fabricated spin-multiplexed metalens array successfully demonstrated spin-dependent bifocality, with measured focal lengths aligning well with design values. The system achieved high transmission and focusing efficiency across the visible spectrum. The integrated light-field camera achieved a record DoF, seamlessly bridging near and far focusing ranges (3 cm to 1.7 km). The multiscale convolutional neural network effectively corrected for optical aberrations, producing high-quality, aberration-free images across this extreme DoF. The system's resolution closely matched theoretical diffraction limits. Experiments with a USAF 1951 resolution chart validated the high spatial resolution across the entire DoF. Real-world imaging of a scene encompassing a range of distances from 3 cm to 1.7 km demonstrated the camera's capabilities in capturing sharp images of both near and far objects.

This study successfully demonstrated a light-field camera with an unprecedented DoF, addressing the long-standing trade-off between DoF and resolution in light-field imaging. The bio-inspired design, using the trilobite's bifocal eyes as inspiration, coupled with advanced nanophotonic fabrication and a powerful deep learning-based aberration correction algorithm, resulted in a system surpassing the capabilities of existing technologies. The achievement of high-resolution imaging across such a broad range of distances has implications for numerous applications, from consumer photography to microscopy and machine vision. The neural network's ability to correct for complex aberrations simplifies the design and fabrication of metasurface optics, reducing the need for intricate designs to compensate for imperfections.

This research presents a significant advancement in light-field imaging technology. The development of a nanophotonic light-field camera inspired by the trilobite's visual system, combined with a deep learning-based aberration correction algorithm, has achieved unprecedented depth of field and high resolution. Future work could explore improvements in efficiency and miniaturization, as well as applications in specific fields like 3D microscopy and autonomous driving.

While the developed system demonstrates exceptional performance, some limitations exist. The current design relies on polarization-dependent focusing, meaning that unpolarized light will have reduced performance. Further optimization of the metalens design and fabrication process could improve efficiency. The training dataset for the neural network is specific to the fabricated optical system; adapting the network to different system configurations might require retraining.