Compact light field photography towards versatile three-dimensional vision

The paper addresses the challenge of high-speed, accurate 3D imaging that works over large depth ranges and under severe occlusions. Conventional approaches measure along one extra axis of light: multi-view (angular) methods such as stereo, structured light, and light field cameras provide high accuracy at near range but degrade quadratically with distance and often rely on scene texture; time-of-flight (ToF) methods are texture-agnostic and maintain resolution over longer ranges but struggle with high-speed, dense depth mapping robust to motion. Combining multi-view with ToF promises benefits such as ultra-fast 3D imaging, extended sensing range, and seeing through occlusions, yet direct multi-view ToF acquisition either has limited views or is too time-consuming, and it exacerbates the big-data burden due to additional angular and temporal dimensions. The authors propose Compact Light Field Photography (CLIP) to efficiently sample dense light fields using simple optics and a small number of sensors (including single-pixel, 1D, and sparse/2D arrays). CLIP distributes nonlocal acquisition across views and models inter-view correlations, enabling recovery of 4D light fields or directly refocused images from a dataset smaller than a single sub-aperture image, thereby synergizing multi-view and ToF for extended-depth, occlusion-robust, and non-line-of-sight (NLOS) 3D vision.

Multi-view techniques (stereo, structured light, lens-array light field cameras) can achieve sub-100 μm depth accuracy at short range but suffer with distance and, except for structured light, depend on texture. ToF systems maintain depth resolution over longer ranges and are texture agnostic but face challenges in high-speed, dense, and motion-robust mapping. Previous multi-view ToF attempts have either few views or require slow scanning, and data volume becomes prohibitive when adding angular and temporal dimensions, complicating real-time processing, especially with large-format detectors (e.g., megapixel SPADs) or specialized sensors with limited element counts (infrared, ultrafast, terahertz). Earlier compressive light field approaches required dense 2D sampling to reconstruct 4D light fields. Wavefront coding and coded aperture can extend depth of field or enable 3D but still assume dense 2D sampling, limiting applicability to ultrafast or IR regimes where detectors are 0D/1D and sparse. CLIP generalizes and unifies these ideas by transforming nonlocal imaging models (single-pixel camera, x-ray CT/Radon models, diffuser cameras) into efficient light-field acquisition compatible with arbitrary sensor formats and camera arrays.

CLIP reformulates imaging as f = A h + σ, where f is the measurement vector, h the image, A the system matrix, and σ noise. Instead of acquiring all m measurements from a single view, CLIP splits the nonlocal measurements across l views by partitioning A into sub-matrices and forming a block-diagonal operator over the 4D light field P, yielding f = A' P + σ. It then explicitly models inter-view correlations via a shearing operator that relates each sub-aperture image Pk to a reference image h through Pk = Bk h. This yields a depth-dependent forward model f = F(d) h + σ, where changing d corresponds to refocusing. Nonlocal acquisition is vital: each small sub-measurement vector per view encodes global information about the scene, enabling compressive sampling (m < N^2) while preserving robustness to defective pixels and occlusions. CLIP supports several embodiments: (1) Single-pixel (0D) CLIP: sequential random coding with a bucket detector while scanning the detector along the angular (u,v) direction to split measurements into views; with binary codes, about 7 measurements per view suffice to cover all pixels with high probability. (2) Linear array (1D) CLIP: a cylindrical lens performs a Radon-like line integral along its invariant axis onto a 1D sensor; arrays of cylindrical lenses at distinct orientations acquire complementary angular encodings across views in parallel. (3) 2D detectors: a complex mask (e.g., random lens/diffuser) yields a wide-field, depth-dependent PSF to multiplex multi-view measurements; wavefront coding, coded aperture, and diffuser cameras are subsumed into CLIP, often avoiding full 4D reconstruction. Camera arrays can be incorporated by letting each camera capture a few nonlocal coefficients with overlapping FOVs. For ToF-CLIP, a streak camera (1D ultrafast sensor) is multiplexed by seven cylindrical lenslets at different angles to form a multi-view ToF dataset in one snapshot. The system captures a 125×125×7 light field with 1016 temporal samples at 100 Hz. A femtosecond laser provides programmable diverged or collimated illumination. Reconstruction is posed as sparse-regularized optimization: either estimating the refocused image h via minimizing ||f − F(d)h||_1 + μ||ψ(h)||_1 or reconstructing the 4D light field P via ||f − A^T P||_1 + μ||ψ(P)||_1, using RED with BM3D/TV denoisers and warm-start across refocus depths. Calibration (intrinsics via Zhang’s method) enables metric 3D. Polar-to-rectilinear coordinate conversion transforms LiDAR ranges to 3D points using camera intrinsics; time-gain compensation offsets r^2 decay in flash LiDAR. For NLOS, the relay wall geometry is mapped by flash LiDAR; the laser line is parameterized via two reference points measured by LiDAR. A hybrid time-frequency NLOS reconstruction first migrates the curved-wall measurement to a virtual plane in time domain (o(N^4)) and then applies frequency-domain phasor-field reconstruction (o(N^3 log N)), achieving overall o(N^4) compute and o(N^3) memory, with CUDA GPU acceleration.

- CLIP acquires large-scale light fields using simple optics and few sensors (0D/1D/2D), reducing data by orders of magnitude while enabling refocusing and 3D imaging. - Occlusion-robust 3D imaging: A ToF-CLIP setup with a 1D streak camera and seven cylindrical lenslets (baseline 15 mm; FOV 30 mm at 60 mm distance; ~1000 pixels) implicitly recorded a 125×125×7 light field and streamed 1000 spatial × 1016 temporal samples at 100 Hz. It reconstructed objects fully occluded in the front view, producing background-free 3D separation of objects and occluders. Despite a compression factor ~20 (relative to a single sub-aperture image), occluded objects with relatively simple geometry were recovered; synthetic studies reported <10% imaging error with >100× reduction in light-field data. - Extended-range flash LiDAR: Single-shot imaging of texture-less objects over ~2 m depth range at ~3 m standoff and 1.5 m × 1.5 m FOV. Digital refocusing shows defocus blur without CLIP; an all-in-focus LiDAR image produced by CLIP resolves the entire 3D scene. Estimated resolution: ~30 mm lateral, ~10 mm axial. Dynamic scenes (rotating letter V) captured at 100 Hz with faithful motion and shadow recovery in both simple and cluttered backgrounds. - Real-time NLOS imaging with non-planar relay surfaces: Using flash LiDAR to map the relay wall and a hybrid time-frequency reconstruction, CLIP performed single-shot NLOS imaging with planar, disconnected, and curved walls, recovering 3D position and shape of hidden objects placed >1 m from the wall spot. Extended depth of field was crucial; disabling it degraded reconstructions. Curved-wall experiments showed artifacts due to strong secondary laser inter-reflections. - Computational performance: GPU-accelerated hybrid NLOS reconstruction achieves ~0.03 s for a 128×128×128 volume with a 125×125×1016 data cube (~30 Hz). The iterative CLIP reconstruction of the wall data is the bottleneck (~2.0 s), reducible to ~0.01 s using an adjoint-operator approximation at the cost of noise robustness. - Robustness: Nonlocal acquisition confers resilience to defective pixels and severe occlusions because each measurement encodes global scene information. - Generality: CLIP unifies wavefront coding, coded aperture, and diffuser cameras, enabling light-field functionality without full 4D reconstruction and supporting arbitrary sensor formats and camera arrays.

The study demonstrates that distributing nonlocal measurements across views and exploiting inter-view correlations enables efficient sampling of the plenoptic function along angular and temporal dimensions. This addresses the core limitations of traditional multi-view (distance-dependent accuracy, occlusion) and ToF (speed, density, robustness) approaches by synergizing them in CLIP. In practice, CLIP reduces data requirements dramatically while retaining light-field capabilities like refocusing and depth-from-focus, thus extending LiDAR depth of field and enabling background-free 3D through occlusions. The hybrid NLOS solver and built-in flash LiDAR mapping of relay surfaces expand applicability to realistic, curved, and disconnected walls, moving towards field-ready, point-and-shoot NLOS systems. The framework’s compatibility with 0D/1D/2D sensors and camera arrays enhances robustness to sensor defects and occlusion, and facilitates deployment in regimes where high-resolution 2D detectors are unavailable (ultrafast, IR, terahertz). Beyond direct 3D sensing, CLIP’s efficient optical-domain dimensionality reduction enables modular extensions to spectral and polarization channels, potentially further improving depth sensing and reconstruction quality. Overall, the findings validate CLIP as a versatile, scalable approach to high-speed 3D vision with significantly lower data load and improved robustness.

The work introduces Compact Light Field Photography (CLIP), a general framework that transforms nonlocal imaging models into efficient light-field acquisition across arbitrary sensor formats. By modeling inter-view correlations and employing nonlocal multiplexing, CLIP recovers refocused images or 4D light fields from datasets smaller than a single sub-aperture image. Experiments demonstrate: (i) background-free 3D imaging through severe occlusions, (ii) snapshot flash LiDAR with extended depth of field over meter-scale ranges and dynamic scenes at 100 Hz, and (iii) real-time NLOS imaging with planar, disconnected, and curved relay surfaces using a hybrid time-frequency solver. Future directions include measuring and correcting optical aberrations by recovering full 4D light fields, integration with megapixel SPAD and other specialized detectors, extensions to spectral and polarization dimensions for enhanced 3D sensing, and further acceleration/robustness improvements in reconstruction (e.g., improved priors, learned denoisers, and optimized measurement designs).

- Occlusion complexity: With a compression factor ~20 and reduced effective measurements for occluded regions, well-recovered occluded objects must be relatively simple in geometry; occluded parts may appear with weaker intensity due to fewer contributing measurements. - NLOS artifacts: Curved-surface NLOS experiments suffered from strong secondary laser inter-reflections causing artifacts, despite algorithmic robustness to weaker multiple scattering. - Computational bottleneck: Iterative CLIP reconstruction of the wall’s spatiotemporal data (~2.0 s) limits real-time NLOS rates; a faster adjoint solution (~0.01 s) trades off noise robustness. - Model assumptions: Performance relies on structured sparsity and accurate depth-dependent shearing models; deviations (e.g., complex BRDFs, strong multipath) can degrade conditioning. - Hardware constraints: Current demonstrations use a streak camera and femtosecond laser; while representative, practical deployments may require alternative detectors/illumination and careful calibration (intrinsics, LiDAR coordinate transforms, laser line parameterization).