Snapshot multidimensional photography through active optical mapping

Conventional photography captures only 2D spatial light distribution, neglecting rich information contained in the plenoptic function (x, y, z, θ, φ, λ, τ). Multidimensional photography aims to capture this additional information, such as wavelength and time, but existing methods are limited by static optical architectures. Current snapshot multidimensional imagers, while offering parallel capture of light datacube voxels, suffer from fixed mapping relations between voxels and sensor pixels, limiting their adaptability. This paper addresses this limitation by introducing tunable multidimensional photography using an active optical mapper, a high-resolution SLM that dynamically permutes and maps light datacube voxels onto the sensor pixels. This allows the acquisition scheme to adapt to the scene, significantly enhancing flexibility and enabling seamless transitions between imaging modalities. The research's importance lies in overcoming the limitations of existing systems, opening new avenues for various applications in science and engineering where scene adaptability is crucial.

Existing multidimensional imaging techniques broadly fall into two categories: direct measurement and compressed measurement. Direct measurement establishes a one-to-one mapping between light datacube voxels and camera pixels, but is limited by the number of camera pixels. Compressed measurement allows multiple voxels to map onto a single pixel, improving detector utilization but increasing computational complexity and requiring sparse scenes. Snapshot multidimensional imagers offer parallel capture, improving light throughput, but most are passive, with fixed mapping relations. The lack of tunability in these systems restricts their applicability in dynamic or complex scenes. This paper reviews these existing techniques, highlighting their advantages and disadvantages in relation to the proposed method of active optical mapping.

The proposed system uses a high-resolution reflective-type SLM as an active optical mapper. The SLM dynamically adjusts the mapping between incident and emitted light. The system's operation involves projecting a high-dimensional light datacube onto the SLM, which then permutes and maps the voxels onto a 2D image sensor. The mapping relation is programmed, enabling reconstruction of the high-dimensional datacube from the 2D measurement by solving the inverse problem. The mapper allows flexible switching between direct and compressed measurement. In direct measurement (linear phase array on the SLM), a one-to-one mapping is created, enabling real-time reconstruction. In compressed measurement (pseudo-random phase pattern), multiple voxels map to a single pixel, requiring computational reconstruction under sparsity constraints. The paper details the optical architecture, including a multiscale lens design that converts angular separation created by the SLM into spatial separation, enabling the use of dispersion elements (diffraction grating for hyperspectral imaging, time-delay integration (TDI) camera for high-speed imaging) to fill the blank spaces created by the mapping. The system's adaptability is demonstrated through various experiments. For hyperspectral imaging, a 3D (x, y, λ) datacube is vertically sliced, spectrally dispersed, and imaged onto a 2D sensor. For high-speed imaging, a 3D (x, y, t) datacube is horizontally sliced and temporally dispersed using a TDI camera. The system also transitions seamlessly between direct and compressed measurements by simply changing the SLM's phase pattern. Finally, the system's ability to perform 4D (x, y, λ, t) imaging is demonstrated using a hybrid approach combining direct and compressed measurements.

The active optical mapping system successfully performs hyperspectral imaging using both direct and compressed measurement. Direct mapping enables real-time reconstruction of a 45 × 90 × 18 (x, y, λ) datacube, resolving spectrally overlapping images. Experiments with an occluded finger demonstrate real-time video-rate hyperspectral imaging, capturing changes in absorbance following occlusion release. Compressed measurement, using multiple encoding patterns, achieves high-fidelity reconstruction of a large 400 × 340 × 275 (x, y, λ) hyperspectral datacube. In high-speed imaging, direct mapping at 153.6 kHz frame rate resolves a fast-moving object, eliminating motion blur. Compressed measurement expands the time window, enabling reconstruction of a fast-moving object at 102.4 kHz with a compression ratio of 256. An application in high-throughput imaging flow cytometry demonstrates blur-free imaging of fluorescent beads at 200 kHz. Finally, a hybrid approach combines spectral and temporal shearing for 4D (x, y, λ, t) imaging, successfully capturing a spectrally and temporally varying scene. Quantitative evaluation in direct measurement mode using a color checker target shows an average root mean squared error (RMSE) of 0.11, with noise primarily from SLM imperfections. A numerical study comparing direct and compressed measurements shows that compressed sensing with multiple encoding patterns offers superior noise tolerance, particularly at higher compression ratios.

The active optical mapping system significantly advances multidimensional photography by offering tunability and flexibility previously unavailable. The ability to switch seamlessly between direct and compressed measurement modes optimizes data acquisition for diverse applications, mitigating the trade-offs inherent in existing techniques. The demonstrated capabilities in hyperspectral and ultrafast imaging, along with 4D imaging, open numerous avenues in biomedical imaging, material science, and other fields requiring high-speed, high-resolution, and multispectral data. The real-time capabilities of the direct mapping method are particularly significant for applications requiring immediate feedback, while the ability to capture large datasets using compressed sensing is crucial for comprehensive analysis of complex scenes. The system's adaptability addresses a critical limitation in current multidimensional imaging systems.

This paper presents a versatile snapshot multidimensional photography platform based on active optical mapping. The use of a high-resolution SLM enables dynamic control over the mapping of high-dimensional light datacubes onto a 2D sensor, providing unparalleled flexibility. The system's success in hyperspectral, ultrafast, and 4D imaging demonstrates its potential to revolutionize various applications. Future research could focus on further improving the SLM's resolution and developing advanced computational algorithms for even more efficient and robust reconstruction.

The current system's performance is limited by the SLM's space-bandwidth product, affecting the maximum size of the measurable datacube in direct measurement mode. Compressed measurement, while allowing larger datacubes, requires computationally intensive reconstruction and is sensitive to noise. Improvements in SLM technology and advanced reconstruction algorithms are needed to fully address these limitations.