Compact light field photography towards versatile three-dimensional vision

Three-dimensional (3D) imaging is crucial for understanding the physical world and has applications in diverse fields like navigation, robotics, and medical imaging. However, capturing 3D scenes with 2D sensors presents a dimensionality challenge. Current methods, such as multi-view techniques (stereo, structured light, light field cameras) and time-of-flight (ToF) sensing, each have limitations. Multi-view methods offer high accuracy at close range but degrade with distance and require object texture. ToF methods maintain resolution over large ranges but struggle with high-speed, dense depth mapping and robustness to motion. Combining multi-view and ToF measurements offers potential benefits: ultrafast imaging, enhanced sensing range, and occlusion-handling capabilities. However, existing approaches suffer from limited views, slow acquisition speeds, and massive data loads. This paper addresses these challenges by introducing compact light field photography (CLIP), a framework for efficient light field sampling.

The paper reviews existing 3D imaging techniques, highlighting the strengths and weaknesses of multi-view and time-of-flight methods. Multi-view methods, while accurate at close range, suffer from quadratic accuracy degradation with distance and reliance on object texture. Time-of-flight methods, while robust to texture and capable of long-range sensing, face challenges in achieving high-speed dense depth mapping. The combination of multi-view and time-of-flight approaches is noted as potentially disruptive, but existing methods struggle with the data volume and acquisition speed. The authors then discuss existing compressive light field cameras which require densely sampled 2D images and mention other techniques like coded aperture and wavefront-coding, which are limited by detector resolution in ultrafast or infrared imaging.

CLIP is presented as a systematic framework that transforms any imaging model using nonlocal data acquisition into a highly efficient light field imaging approach. It distributes the nonlocal acquisition process across different views and models the inherent correlations in 4D light fields. This allows for light field recovery or direct refocused image retrieval from a measurement dataset smaller than a single sub-aperture image. CLIP accommodates various sensor formats (single pixel, linear array, sparse 2D area detectors). The core concept involves representing image acquisition as a matrix equation (f = Ah + σ), where A is the system matrix, h is the image, and f is the measurement. CLIP modifies this to efficiently capture light fields by employing nonlocal acquisition for the rows of A and splitting measurements into different views. This results in a transformed equation (f = A'P + σ), where A' is a block-diagonal matrix and P is the 4D light field. CLIP further exploits the correlation among sub-aperture images by using a shearing operator to relate them, leading to a final model (f = F(d)h + σ), where F(d) is a depth-dependent function. This allows for refocusing and 3D imaging. The paper details implementations using single-pixel, linear array, and 2D area detectors, highlighting the nonlocal acquisition strategy's role in robustness against occlusions and defective pixels. The methodology also includes ToF integration, resulting in a system capable of snapshot 3D imaging, and real-time NLOS imaging. Specific experimental setups for ToF-CLIP using a streak camera and cylindrical lenslets are described. The image reconstruction process involves solving optimization problems using l1 norm minimization and regularization techniques (BM3D and TV denoisers). Camera calibration procedures and details regarding flash LiDAR and NLOS experiments are provided, including time-gain compensation, coordinate transformations, and a hybrid frequency-time domain NLOS reconstruction algorithm for efficient handling of curved surfaces. The algorithm uses wave propagation in the time domain to convert measurements on a curved surface to a virtual plane, followed by frequency-domain reconstruction.

The paper demonstrates CLIP's effectiveness through several key findings. First, it shows that CLIP enables single-shot 3D imaging through severe occlusions using time-of-flight (ToF) data. Objects completely hidden behind occluders in conventional images are successfully reconstructed in 3D, showcasing the benefits of dense multi-view ToF data. Second, CLIP extends the depth range of flash LiDAR imaging. It achieves all-in-focus images of scenes spanning approximately 2 meters, overcoming limitations of conventional flash LiDAR. Third, CLIP achieves real-time non-line-of-sight (NLOS) imaging with curved and disconnected surfaces. The system uses built-in flash LiDAR for real-time surface mapping and a hybrid time-frequency domain reconstruction algorithm for efficient processing. NLOS imaging results are presented for planar, disconnected, and curved walls, demonstrating the versatility of the approach. The authors also show successful dynamic imaging of rotating objects, capturing motions accurately even with cluttered backgrounds. The efficiency of CLIP is emphasized throughout the results section, showing how it significantly reduces the data load compared to conventional light field cameras while maintaining performance, Quantitative accuracy is presented in the supplementary materials, demonstrating that CLIP maintains a small imaging error (<10%) with a substantial reduction (>100 times) in light field measurement data.

CLIP addresses the limitations of existing 3D imaging methods by providing an efficient framework for capturing and processing light field data. The ability to handle occlusions, extend depth of field, and perform real-time NLOS imaging with complex surfaces represents significant advancements in 3D vision. The use of various sensor formats demonstrates the adaptability of the technique. The integration of ToF capabilities allows for high-speed imaging and robust depth estimation even in challenging scenarios. The results show CLIP’s potential for diverse applications in areas such as robotics, autonomous driving, and medical imaging. The reduced data load and computational efficiency of the proposed reconstruction algorithms are important contributions, enabling real-time processing for dynamic scenes.

Compact light field photography (CLIP) presents a novel and efficient framework for versatile 3D vision. Its ability to handle various sensor formats, combine multi-view and ToF data, and perform real-time NLOS imaging with complex surfaces is significant. Future research could explore applications in more complex scenarios, improvements in reconstruction algorithms for even greater efficiency and robustness, and investigations into the integration of additional modalities (e.g., spectral or polarization information).

While CLIP demonstrates significant advancements, some limitations exist. The current implementation for imaging through occlusions is limited to relatively simple geometries due to the compression factor. The extended depth-of-field in NLOS imaging is crucial for good performance; defocus effects degrade the results. Additionally, secondary laser inter-reflections in NLOS imaging with curved surfaces caused artifacts in some experiments. The computational complexity of CLIP reconstruction, while improved compared to conventional methods, could still be optimized further.