Non-line-of-sight snapshots and background mapping with an active corner camera

The ability to 'see' around corners, or non-line-of-sight (NLOS) imaging, holds transformative potential for numerous fields, including search and rescue, autonomous navigation, and reconnaissance. Current active NLOS imaging techniques typically involve sequentially illuminating points on a relay surface (e.g., a wall) using a pulsed laser and employing time-resolved sensing to gather transient information. These methods, including raster scanning approaches, are limited by the scanning process itself, restricting speed and scalability. While snapshot methods exist, they are generally limited to reconstructing only point-like targets. This research addresses these limitations by presenting a snapshot-capable active NLOS system that generates accurate reconstructions of moving foreground objects and simultaneously maps the static background behind them. The challenge in NLOS imaging, both active and passive, lies in the multiple diffuse bounces light undergoes before reaching the sensor, leading to loss of directional information and signal attenuation. Passive methods leverage occluding structures (e.g., windows or moving objects) to separate light originating from different hidden scene directions. Active methods, on the other hand, commonly rely on laser scanning, which this study aims to overcome. This work builds upon previous research in edge-resolved transient imaging (ERTI), which utilizes a vertical edge occluder for azimuthal resolution and transient measurements for longitudinal resolution, but eliminates the scanning requirement, resulting in a significant improvement in acquisition speed and a new capability of background mapping.

Existing active NLOS imaging techniques predominantly employ laser scanning across a relay surface, limiting acquisition speed and scene size. Methods like raster scanning of a rectangular grid on a wall opposite the hidden scene, or using a horizontal relay surface, suffer from this inherent limitation. Snapshot approaches are restricted to identifying only one or two point targets. Edge-resolved transient imaging (ERTI) represents a notable advancement, combining an edge occluder with transient measurements. However, ERTI still relies on laser scanning. Previous work using a floor as a relay surface and a SPAD array improved acquisition time, but was confined to tracking the horizontal position of a point-like moving object. Most existing methods assume a static scene during acquisition, hindering the ability to handle moving objects in a dynamic environment. This work draws inspiration from passive corner-camera systems that exploit the known geometry of vertical edges for azimuthal resolution, and aims to leverage both active illumination and edge occlusion for accurate snapshot imaging of moving objects and background mapping.

This research utilizes a similar hardware setup as previous work, employing a floor as a relay surface and a SPAD array positioned adjacent to a vertical wall edge. A pulsed laser illuminates the hidden scene, and the SPAD camera measures the temporal response of the returning light. The vertical edge occluder provides azimuthal resolution, while temporal response yields longitudinal resolution. Crucially, the system acquires data in a single snapshot per frame, eliminating laser scanning. The acquisition process incorporates a reference measurement before the introduction of moving objects to characterize the stationary scene response. A Poisson distribution model (equation 1) describes the measurement rate, considering contributions from stationary scenery (*b*), foreground objects (*S_fg*), and occluded background regions (*S_oc*). The foreground objects and occluded regions are modeled as vertical, planar, rectangular facets. Parameters like albedo (*a*), range (*r*), and height (*h*) are estimated for each object and its corresponding occluded region. These parameters are recovered from the measurement data for each frame, processed independently. The occlusion is modeled as a shadow, and the reduction in photon counts due to the shadow of the moving object on the stationary background is used to reconstruct the occluded background region. By accumulating reconstructions from multiple frames as the object moves, a map of the static hidden scene is generated. The system's acquisition parameters involve an integration time for each frame (0.4 seconds for dynamic frames, 30 seconds for the reference frame) and a frame length of 10 µs. The experimental setup included a hidden room with white foam board walls, a black cloth ceiling, and various objects (white and gray facets, a mannequin, and a staircase) used to test the reconstruction algorithm.

The developed active NLOS system successfully reconstructs both moving objects and the stationary background in a single snapshot per frame. The algorithm accurately estimates the height, width, range, and reflectivity of multiple moving objects. The occluded background regions behind the moving objects are also accurately reconstructed, even when objects overlap. By accumulating data from multiple frames, a comprehensive map of the static hidden scene is created. Experimental results demonstrate the algorithm's robustness with objects of varying reflectivity and shapes that deviate from the assumed rectangular model (e.g., a mannequin and a staircase). Even with dimmer objects, range estimation remains accurate. The system effectively handles scenarios where multiple objects move through the hidden scene, resolving their individual positions and generating a background map that closely matches the ground truth. The method is robust to different hidden object locations, frame lengths, and lighting conditions, making it versatile for various real-world scenarios.

The results demonstrate the success of the proposed method in achieving both detailed reconstruction of moving objects and simultaneous mapping of the hidden static environment in a non-line-of-sight setting. This addresses the limitations of previous active NLOS systems that rely on scanning or are restricted to point-like reconstructions. The accurate estimation of the parameters for each object, including its occluded background region, showcases the effectiveness of the proposed model. The ability to generate a comprehensive map of the hidden scene enhances the system's potential applications significantly. This approach is particularly valuable in dynamic environments where previous methods struggle. The robustness observed with objects of different shapes and reflectivity expands the system's practicality for diverse scenarios. Future research could explore joint processing of multiple frames to leverage inter-frame priors, improving reconstruction quality and tracking accuracy. This would include considering motion continuity and the consistency of object properties across frames.

This paper introduces a novel active NLOS imaging method capable of reconstructing moving objects and simultaneously mapping the stationary hidden scene using a single snapshot per frame. This is achieved through careful modeling of the optical response and the exploitation of a vertical edge occluder. The results demonstrate accurate reconstruction and background mapping, even under challenging conditions, significantly advancing the state-of-the-art in NLOS imaging. Future work could involve improved algorithms that process frames jointly, incorporate wall thickness into the model, and enhance target resolution.

The current algorithm processes each frame independently, which could be improved by incorporating inter-frame priors to leverage temporal consistency of object properties and motion. The thin-wall assumption simplifies the model, but real-world walls have thickness, requiring more complex modeling. The spatial and temporal resolution of the current sensor limits the accuracy of parameter estimation, particularly for fast-moving or distant objects. Improvements in SPAD technology are expected to address these limitations.