Seeing around corners with edge-resolved transient imaging

The ability to 'see around corners' has significant implications across various fields, including medical imaging, surveillance, and autonomous navigation. Autonomous navigation, in particular, could greatly benefit from NLOS imaging by allowing vehicles to anticipate obstacles hidden from direct view, thus improving safety and efficiency. Current NLOS imaging techniques, however, suffer from limitations such as long acquisition times, limited field of view (FOV), and small scale, hindering their practical application. This paper introduces a novel framework for NLOS imaging, edge-resolved transient imaging (ERTI), designed to overcome these limitations and pave the way for more practical implementations. The primary challenge in NLOS imaging stems from the diffuse nature of light reflections. Light scattering from diffuse surfaces destroys directional information and significantly reduces signal strength. While line-of-sight (LOS) imaging largely avoids this issue through focused illumination and detection, NLOS imaging faces the added complexity of cascaded diffuse reflections, where light bounces off a relay surface before reaching the observer, further weakening the signal. Existing NLOS imaging techniques have made significant progress in addressing these challenges. Early methods utilized active optical illumination, scanning a pulsed laser over a relay surface and employing time-resolved sensing to collect light. While successful, these approaches often require long acquisition times and are limited to small-scale scenes due to the weak signals involved. Subsequent research focused on enhancing reconstruction algorithms, incorporating techniques such as back-projection, Fourier transform-based methods, and Bayesian approaches. However, even with improved algorithms, the extremely low levels of informative light usually necessitate the use of single-photon detectors and multiple illuminations, restricting the practical scale of the imaged scenes. Other NLOS imaging approaches have explored passive methods, leveraging ambient light and occlusions to infer information about hidden scenes. These methods are often less sensitive to radial falloff but inherently have greater uncertainty due to uninformative ambient light. Additionally, some methods avoid diffuse reflections altogether by using modalities like thermal, acoustic, or radar, but these often compromise resolution and provide information about physical properties different from optical reflectance. In summary, while existing methods show promise, most are not suitable for practical deployment due to their constraints in terms of acquisition time, FOV, scale, and reliance on specific environmental conditions.

The literature review extensively covers existing NLOS imaging techniques, categorizing them based on their illumination methods (active vs. passive), sensing modalities, and reconstruction algorithms. Active methods, utilizing pulsed laser illumination and time-resolved detection, are analyzed for their strengths and weaknesses. The review highlights advances in reconstruction algorithms, including back-projection, light-cone transform, f-k migration, and Bayesian methods. The limitations of these methods, such as the need for large scan apertures and long acquisition times, are discussed. Passive techniques, relying on ambient light and occlusions, are also reviewed, noting their advantages in terms of reduced radial falloff but increased uncertainty due to ambient light. The review further touches upon alternative modalities like thermal, acoustic, and radar imaging, comparing their advantages and limitations compared to optical methods. The overall assessment emphasizes the lack of a practical NLOS imaging solution suitable for large-scale scene reconstruction with high speed and minimal resources.

ERTI introduces a novel approach to NLOS imaging that combines the advantages of both active and passive methods. It leverages the occluding effect of vertical edges, a common feature in many environments, similar to passive corner cameras, but incorporates pulsed laser illumination and time-resolved detection for precise distance measurement. This dual configuration distinguishes ERTI from existing corner camera approaches. The ERTI system sequentially illuminates 45 spots along a semicircle (1.5 cm radius) centered around a vertical edge using a 532-nm pulsed laser. A single-photon avalanche diode (SPAD) detector, positioned beyond the edge, collects light from both the visible and hidden scenes. The time-correlated single photon counting (TCSPC) module records photon detection times, generating histograms for each illumination spot. Temporal gating is used to eliminate the overwhelming direct reflection from the ground plane. Crucially, ERTI computes the differences between successive histograms. This histogram differencing technique effectively isolates the light contributions from a specific wedge-shaped portion of the hidden scene illuminated by a given laser spot, removing the constant contributions from the visible scene and ambient light. This process significantly reduces noise and enhances directional information, counteracting the inherent ambiguity caused by diffuse reflections. The light transport model in ERTI uses a 2.5D representation of the scene, approximating objects as planar facets extending vertically from the ground plane. This simplification is justified by the prevalence of such objects in real-world environments and greatly reduces the computational complexity of the reconstruction process. The model incorporates parameters such as distance, height, orientation angle, and albedo for each facet. Closed-form expressions for the transient response of these planar facets are derived, significantly improving computational efficiency compared to previous methods that relied on Monte Carlo simulations. The reconstruction algorithm employs a Bayesian framework with a tailored Markov chain Monte Carlo (MCMC) sampling approach. This framework incorporates prior knowledge about the structure of typical scenes, such as spatial clustering of facets and the tendency for facets belonging to the same object to share similar parameters (height, albedo, orientation). This Bayesian approach effectively handles the uncertainty inherent in NLOS imaging and the unknown number of surfaces within each wedge. The algorithm iteratively modifies the facet configuration, proposing changes and accepting them based on a predefined rule (Green ratio), efficiently incorporating non-linear effects like occlusions.

The key findings of this research demonstrate the effectiveness of ERTI in reconstructing large-scale NLOS scenes with high accuracy despite using a small aperture and a limited number of measurements. The histogram differencing technique, combined with the 2.5D planar facet model, significantly reduces uncertainty and enhances directional information, allowing for accurate localization of objects in the hidden scene. Experiments conducted on various indoor scenes containing different objects confirmed the accuracy of the ERTI system. Reconstructions accurately capture the positions and orientations of foreground objects, ceiling height, and most wall components. The planar staircase used as a test object demonstrated height estimation accuracy within approximately 10 cm. The system successfully resolved challenging scenes with multiple objects, varying depths, heights, and albedos. Results showcased the robustness of the ERTI algorithm to noise and low signal strength, even with short acquisition times. Comparisons with conventional NLOS methods revealed a significant advantage for ERTI in reconstructing large scenes using small apertures and fewer measurements. While conventional methods struggle with large-scale scene reconstruction due to the need for large apertures and numerous measurements, ERTI effectively overcomes these limitations by utilizing the existing vertical edges. The ERTI methodology provides a more efficient and practical alternative for NLOS imaging, addressing limitations present in existing techniques.

The results demonstrate the feasibility and practical advantages of ERTI for NLOS imaging. The combination of edge resolution and transient imaging allows for the reconstruction of large-scale scenes using a significantly smaller aperture and fewer measurements compared to conventional methods. This reduction in measurement requirements translates to faster acquisition times and less stringent hardware demands, making ERTI more practical for real-world applications. The choice of a planar facet representation improves the reconstruction efficiency and allows for the effective incorporation of non-linear effects such as occlusions. The Bayesian framework, with its tailored MCMC algorithm and informed priors, provides a robust and accurate method for estimating the scene parameters despite noise and ambiguity. The success of ERTI hinges on the exploitation of readily available vertical edges, making it applicable to diverse environments without the need for specially engineered relay surfaces. This characteristic contrasts sharply with conventional methods which often require large, uniformly reflective surfaces, limiting their applicability. The findings of this study suggest a significant advancement in NLOS imaging, offering a potential pathway towards robust and practical systems for diverse applications such as autonomous navigation.

ERTI presents a novel approach to NLOS imaging that addresses the limitations of existing techniques. By combining edge resolution with transient imaging, ERTI achieves accurate reconstruction of large-scale scenes using a significantly reduced number of measurements and a smaller aperture. The use of a planar facet model and a Bayesian reconstruction framework further enhances efficiency and robustness. Future research directions could focus on improving temporal resolution for real-time applications, incorporating multi-wavelength illumination for color information, and extending the model to handle more complex occluders and object shapes. The enhanced efficiency and practicality of ERTI make it a promising technique for various applications, particularly in the domain of autonomous navigation and robotics.

While ERTI demonstrates significant advancements in NLOS imaging, several limitations exist. The current system's resolution is limited by the relatively small number of illumination positions. The assumption of vertically extending planar facets for objects, while valid for many scenarios, may not be accurate for all object types (e.g., cantilevered or floating objects). The system currently uses a single laser wavelength, thus neglecting spectral information that could aid in object identification. Finally, acquisition and reconstruction times, while improved compared to conventional methods, are not yet optimized for real-time applications.