Snapshot multidimensional photography through active optical mapping

The study addresses the limitation of conventional 2D image sensors that integrate away angular, spectral, and temporal information of light, losing rich scene information described by the plenoptic function. Existing strategies for mapping higher-dimensional light datacubes to 2D sensors often trade a spatial axis for spectral or temporal dimensions, requiring scanning (e.g., pushbroom hyperspectral, streak cameras) and thus assuming scene repeatability. Snapshot multidimensional imagers improve light throughput by parallel capture, using either direct one-to-one mappings (computationally simple but limited by sensor pixel count) or compressed multiplexed mappings (higher detector utilization but computationally intensive and requiring sparsity). Current devices are largely passive with fixed mappings, constraining applicability when prior scene information is lacking. The authors propose a tunable snapshot multidimensional photography platform that actively controls the mapping between datacube voxels and sensor pixels via a programmable spatial light modulator (SLM), enabling adaptive switching between direct and compressed measurement modes for diverse applications.

The paper reviews snapshot multidimensional imaging modalities and their trade-offs. Direct mapping methods (e.g., image mapping spectrometry; sequentially timed all-optical mapping photography) provide computationally efficient reconstruction but are limited by the number of camera pixels. Compressed measurement methods (e.g., coded aperture snapshot spectral imaging, adaptive feature-specific spectral imaging, programmable pixel compressive camera, coded exposure photography, compressed ultrafast photography) increase detector utilization and enable undersampled acquisition but require sparsity and heavy computation. Prior image mappers are passive (custom mirror facets or fiber bundles), with fabrication and throughput limitations. The need for tunable, application-adaptive mapping motivates the active optical approach. The paper also contextualizes benefits of snapshot over scanning in light throughput and references multiscale lens designs and temporal shearing devices such as TDI and streak cameras.

Core concept: Use a reflective phase-only high-resolution SLM as an active optical mapper to programmatically permute and map high-dimensional light datacube voxels to a 2D sensor. By applying different phase patterns, the mapper can operate in direct or compressed modes and target spectral or temporal dimensions.

Optical architecture: A front-end bi-telecentric imaging system forms an intermediate image on the SLM. The SLM imposes a spatial phase map that redirects segments of the image into different angles, effectively tearing the datacube apart in angle. A multiscale lens system (large objective with a secondary lenslet array) converts angular separation at the Fourier plane to spatial separation, thus rearranging image segments that are relayed to the sensor. The empty spaces between segments are filled by shearing along another dimension using dispersion elements: a diffraction grating for spectral shearing or a time-delay integration (TDI) camera for temporal shearing.

Direct mapping mode: Display linear phase ramps on the SLM to act as an array of tilted mirror facets that slice the image and redirect segments to designated lenslets, yielding a one-to-one voxel-to-pixel mapping. Reconstruction uses a precomputed lookup table (invertible reshaping), enabling real-time output with negligible computation.

Compressed mapping mode: Display pseudorandom binary phase patterns on the SLM to encode image segments spatially. After spectral or temporal shearing and integration, multiple datacube voxels are multiplexed onto single pixels, enabling acquisition beyond the Nyquist sampling of the sensor under sparsity assumptions. Reconstruction solves a linear inverse problem with regularization.

Implementations:

• Hyperspectral imaging (x,y,λ): The SLM vertically slices the image; a diffraction grating disperses spectra horizontally; a 3×3 lenslet array relays to a monochrome camera. Direct mode used 45 slices grouped into nine directions; compressed mode used nine complementary encoding patterns, increasing sampling rate ninefold over single-mask approaches.
• High-speed imaging (x,y,t): The SLM horizontally slices the image; a TDI camera performs temporal shearing and vertical integration, streaming line data at high line rates. Direct mode uses an optical chopper to confine the time window; compressed mode integrates over a longer window with complementary masks and reconstructs the video computationally.
• Hybrid 4D imaging (x,y,λ,t): Temporal information is directly mapped via TDI (horizontal slicing and vertical shearing), while spectral information is compressed via horizontal dispersion and coded patterns, reducing the 4D datacube to 2D measurement in a snapshot.

Hardware and parameters: Bi-telecentric lens (MVTC23100) to SLM via 4-f system (×5.33). SLM: Meadowlark HSP1920-488-800-HSP8, 1920×1152 pixels; linear polarizer for phase-only modulation. Achromatic lens (f=250 mm, 50 mm diameter) to diffraction grating (GT25-03, 300 grooves/mm) at Fourier plane. Custom 3×3 lenslet array (f=30 mm, 2.5 mm diameter). Cameras: monochrome CMOS (3376×2704, 3.69 µm) for (x,y,λ); TDI camera (4640×256, 5 µm; 256 TDI stages) for (x,y,t) and 4D. Pupil mask with nine apertures to suppress unwanted orders/DC. For flow cytometry: microfluidic channel 140 µm×25 µm×30 mm; illumination by 532 nm ns-pulsed laser; TDI at up to 200 kHz.

Phase pattern design: Maximum SLM deflection angle θ≈sin⁻¹(λ/2p); with λ=550 nm, p=9.2 µm, θ≈0.03 rad. Hyperspectral direct: 45 vertical slices (42×1152 SLM pixels per slice), divided into nine groups mapped to nine lenslets with angles distributed to avoid zero-order and aliasing overlaps; separation at Fourier plane ≈f·θ≈2.5 mm matching lenslet pitch; magnification SLM→sensor 0.12; slice width 46 µm on sensor; PSF ~8.4 µm; sensor sampled effectively. High-speed direct: 27 horizontal slices; three central lenslets used; temporal information fills vertical spacings via TDI.

Calibration: Spatial mapping at λ=532 nm using a speckle-reduced laser to form elemental lines; fit each line to correct distortions and misalignments. Spectral calibration using narrowband filters at 510, 532, 550, 590 nm to map wavelength to horizontal pixel positions; build full lookup by interpolation. Normalize SLM diffraction efficiency and sensor spectral QE using uniform illumination. For compressed sensing, measure actual encoding operators with HDR acquisitions to account for optical aberrations and nonidealities.

Image formation models: Direct mapping E = T S C I with invertible reshaping A; reconstruction by scalar remapping using lookup table. Compressed sensing: Nine complementary masks C = [C1…C9], measurements Ei acquired after shearing and integration; discretized forward model accounts for pixel pitch and shearing velocity. In TDI, vertical charge summation over P stages yields line outputs. Reconstruction solves argmin ||E−T S C I||² + λΦ(I) with total variation regularization and BM3D denoiser within a two-step iterative shrinkage/thresholding (TwIST) framework.

• Hyperspectral direct mapping: Resolved two heavily overlapping USAF target images illuminated at CWL 532 nm (FWHM 1 nm) and 550 nm (FWHM 10 nm), which were indistinguishable with a color camera. Captured 45×90×18 (x,y,λ) voxels in a single snapshot with real-time reconstruction via lookup table.
• Physiological demo: Monitored a finger during occlusion release under green illumination (CWL 550 nm, FWHM 40 nm). Immediately post-release, total absorbance increased (reactive hyperemia), with spectrum showing a valley near 560 nm consistent with oxy-hemoglobin molar attenuation; after ~10 s absorbance decreased as blood flow normalized.
• Hyperspectral compressed mapping: Using nine complementary encoding patterns increased sampling rate 9×. Reconstructed large datacube: sampled 400×340×275 voxels; resolvable modes 112×96×66. Spectral range 495–505 nm with 1.6 nm resolution. Measured averaged spectral irradiance matched a fiber spectrometer (STS-VIS-L-25-400-SMA, 1.5 nm resolution).
• High-speed direct mapping: Imaged a fast-moving object (~4 m s⁻¹) at high frame rates via TDI. At 10.24 kHz frames showed motion blur; at 153.6 kHz the object was clearly resolved; vertical discretization artifacts were due to limited number of slices (N=27). Full frames at 51.2 kHz reported in supplement.
• High-speed compressed mapping: Temporal compression ratio 256 (full TDI stages). Three complementary masks; line-scanning rate 102.4 kHz; reconstructed a moving rocket scene at 102.4 kHz. Demonstrated imaging flow cytometry: 15 µm fluorescent bead in microfluidic channel at ~0.8 m s⁻¹ imaged at 200 kHz with blur-free frames.
• Throughput and streaming advantage: System can stream compressed videos of 426×256 pixels at 200 kHz, 12-bit depth, overcoming on-chip memory constraints typical of conventional high-speed cameras (e.g., 32 GB memory yields ~1.05 s window for same scene).
• 4D hybrid mapping: Single-exposure snapshot of 27×54×33×10 (x,y,λ,t) voxels by combining direct temporal and compressed spectral mapping. Verified with a linear variable filter producing rainbow illumination and a moving chrome mask; reconstructed time series and per-pixel spectral irradiance.
• Technical evaluation (direct mode): Compared spectral measurements of a color checker against a fiber spectrometer; average RMSE of normalized spectra was 0.11. Noise sources included SLM diffraction efficiency nonuniformity and unwanted diffraction orders; RMSE increased at lower irradiance due to reduced SNR.
• Direct vs compressed trade-off: Reconstruction of a 400×340×275 datacube took ~20 min on an i7-8700 CPU. Simulations showed that with nine complementary masks (compression ratio ξ≈50/9), correlation with ground truth exceeded 0.95 at SNR≥25, approaching direct mapping performance with denoising; fewer masks or lower SNR reduced fidelity.

The active optical mapping approach directly addresses the core limitation of fixed, passive mappings in multidimensional photography by enabling programmable, on-demand mapping tailored to scene characteristics. This adaptability allows seamless switching between direct (invertible, real-time) and compressed (high-efficiency, large datacube) acquisition, extending applicability across hyperspectral and ultrafast imaging without hardware changes beyond SLM pattern updates. Empirical results demonstrate faithful hyperspectral separation of overlapping targets, physiological monitoring with spectra consistent with hemoglobin properties, high-frame-rate video beyond conventional sensor bandwidths, and snapshot 4D imaging via hybrid mapping. The method significantly improves light throughput compared to scanning techniques since voxels are captured in parallel. Trade-offs are tunable: phase patterns can emphasize spectral resolution (e.g., adding cylindrical/Fresnel phases to narrow slice width) at the cost of spatial resolution in direct mode; compressed mode boosts sampling but increases computational burden and noise sensitivity. System performance is currently bounded by the SLM’s space-bandwidth product and diffraction efficiency; larger-format sensors and multi-sensor arrays can further aid reconstruction, especially under high compression. Overall, active mapping enhances measurement flexibility, enabling real-time feedback in direct mode and high-dimensional capture in compressed mode, aligning acquisition strategies with scene sparsity and dynamics.

The paper introduces and experimentally validates a versatile snapshot multidimensional photography platform based on active optical mapping with a programmable SLM. The system adaptively maps high-dimensional datacubes to 2D sensors, supporting direct and compressed acquisition for hyperspectral, ultrafast, and hybrid 4D imaging. Demonstrations include real-time hyperspectral separation, physiological spectral monitoring, high-speed videos at up to 200 kHz streaming, and snapshot 4D capture. The approach improves light throughput and provides acquisition flexibility absent in passive systems. Future work should focus on increasing the SLM space-bandwidth product and diffraction efficiency, refining phase pattern designs to balance spatial/spectral/temporal resolutions, accelerating reconstruction algorithms, and leveraging larger or multiple sensors to enhance fidelity under higher compression.

• The achievable datacube size in direct mode is limited by the SLM’s space-bandwidth product and the number of sensor pixels; current SLM resolution constrains the number of controllable slices/angles.
• Finite SLM diffraction efficiency and unwanted diffraction orders introduce stripe artifacts and reduce SNR; nonuniform efficiency necessitates calibration and normalization.
• Compressed sensing reconstructions are computationally intensive (e.g., ~20 minutes for 400×340×275 on a desktop CPU) and sensitive to measurement noise and deviations from sparsity, potentially causing artifacts.
• In direct high-speed mode, the temporal measurement window is limited by slice spacing and TDI scanning rate, requiring an optical chopper to confine the time window; exceeding the window invalidates one-to-one mappings.
• Trade-offs between spatial and spectral/temporal resolution must be managed (e.g., increasing spectral resolution via phase patterns reduces spatial resolution).