Seeing around corners with edge-resolved transient imaging

The study addresses the challenge of imaging objects outside the line of sight, where diffuse reflections erase directional information and severely attenuate signals. Practical applications such as autonomous navigation require rapid acquisition and reconstruction over large-scale scenes with wide field of view, but existing NLOS methods are hindered by long acquisition times, large required scan apertures, limited FOV, and constrained scene scale. The authors aim to develop a method that leverages ubiquitous vertical edges and time-resolved sensing to recover actionable 2.5D information (plan view plus height) over a wide (≈180°) FOV using few measurements and a small aperture, moving toward practical deployment for tasks like collision avoidance.

Early active NLOS work scanned a relay wall with pulsed illumination and time-resolved detection to reconstruct 3D shape, but required large apertures, many measurement points, and high optical powers, often limited to scenes within ~1 m of the relay surface. Algorithmic advances include improved back-projection, light-cone transforms, f-k migration, Fermat paths, Bayesian approaches, phasor fields, and inverse rendering. Passive approaches exploit occlusions (e.g., corner camera) to reduce directional ambiguity using ambient light, typically yielding 2D or 1D estimates in small-scale scenes. Other modalities (thermal, acoustic, radar) maintain directionality via specular reflections but have lower resolution and different measured properties. Despite progress, conventional active methods suffer FOV and scale limitations tied to aperture size and sampling density, while passive methods are sensitive to ambient illumination and generally cannot infer full 3D structure. Only a few active methods avoid large apertures, often requiring object motion or strong geometric assumptions (e.g., empty polyhedron).

The proposed edge-resolved transient imaging (ERTI) combines active time-resolved sensing with occlusions from vertical edges. Acquisition: A 532 nm, 120 mW picosecond pulsed laser at 20 MHz sequentially illuminates 45 spots along a semicircle (radius 1.5 cm) centered at a vertical edge, sweeping θ from 0 to π. A SPAD detector (focused to a small spot ~20 cm beyond the edge) collects returns from both visible and hidden scenes. Time-correlated single-photon counting records detection times with 16 ps resolution; temporal gating turns on 3 ns after the ground reflection to avoid overwhelming signals. For spot i, the histogram m_i comprises hidden (h_i), visible (v_i≈constant), and background (b). Differencing successive histograms y_i=m_{i+1}−m_i isolates new hidden contributions u_i from a distinct hemispherical wedge determined by the edge occlusion; y_i entries follow a Skellam distribution with conditionally independent bins. Scene model and assumptions: Hidden objects visible from the edge are approximated as vertical planar facets extending from the ground; a ceiling component is optionally included. Within each wedge, a facet is parameterized by radial distance ρ, height γ, orientation θ (facet angle), and albedo α (yielding a 2.5D representation). Light transport: Under a confocal approximation (illumination and detection co-located at the edge origin), the transient photon flux L(t) integrates albedo-weighted Lambertian cosine factors with a δ-function enforcing time-of-flight isochrons. The response of a fronto-parallel half-facet (width w, height η, distance d, albedo α) is derived in closed form by integrating over the circular annulus formed by intersecting a time-of-flight sphere with the facet during a TCSPC bin [t, t+Δt]. The full facet response doubles the half-facet; arbitrary orientations are handled by combining two adjusted half-facet responses. Ceiling contributions are computed similarly. Multiple facets within a wedge are combined nonlinearly by removing occluded components from farther facets. Reconstruction: A Bayesian framework with a spatial point process prior encourages clustered, 1D manifold-like facet configurations across wedges (e.g., walls), with correlated parameters among neighboring facets. The posterior p(Φ,x|{y_i})∝p({y_i}|Φ,x)p(Φ,x) is explored via a reversible-jump MCMC algorithm that proposes local birth/death/move updates, efficiently evaluating the forward model per wedge using the closed-form expressions and accounting for occlusions. Execution times are ~100 s, less than acquisition times. Experimental setup: Hardware is ~2 m from the edge; SPAD with a 25 mm lens and narrowband filter (>90% transmission at 532 nm, 2 nm FWHM); SPAD gating window 42 ns starting 3 ns after the direct ground return; galvos steer the beam; TCSPC (HydraHarp 400) timestamps detections. Data and code are available on GitHub.

• ERTI achieves 2.5D reconstruction (plan view plus facet heights) over a 180° FOV using a small scan aperture (1.5 cm radius arc) and only 45 illumination positions.
• Experimental reconstructions of room-scale hidden scenes (up to ~3 m in each dimension) recover foreground objects, ceiling height, and most wall components with correct positions and orientations, even under high noise and background conditions.
• Height estimation accuracy: For a planar staircase, average facet heights for 30, 60, and 90 cm steps were measured as 41.1, 54.3, and 92.1 cm, respectively (≈10 cm accuracy).
• Simulations show conventional confocal NLOS with limited apertures and measurements fails to reconstruct large scenes (severe transverse blur, limited FOV), whereas ERTI accurately reconstructs ceilings, walls, and cylindrical objects outside the conventional FOV with far fewer measurements.
• Processing runtime is ~100 s, shorter than acquisition (20–60 s per spot in experiments), indicating computational feasibility.
• Histogram differencing reduces uncertainty from 2D to 1D within each wedge and mitigates visible-scene/ambient contributions (mean removed, variance accounted for via Skellam noise).

ERTI addresses the core NLOS challenge—loss of directionality and low signal—by exploiting vertical-edge occlusions to recover angular information and time-of-flight to recover depth, reducing positional uncertainty and measurement requirements. The facet-based model concentrates signal over larger surfaces than voxel methods, improving SNR, enabling explicit occlusion handling, and reducing memory/computation. Closed-form facet responses allow efficient likelihood evaluations within an RJ-MCMC framework, facilitating inference of both the number of facets and their parameters across wedges with structural priors. Compared with conventional relay-wall methods requiring large apertures and many points, ERTI attains wide FOV and room-scale reconstructions with a small arc and few measurements, improving practicality for applications like autonomous navigation and collision avoidance. The method is robust to noise and background, but performance degrades for distant, highly oblique, or occluded walls; vertical resolution is coarser due to the model and acquisition density. The approach can integrate with passive corner cameras and be extended spectrally for richer scene understanding.

The paper introduces edge-resolved transient imaging (ERTI), an active NLOS framework that combines vertical-edge occlusions, pulsed time-of-flight sensing, closed-form facet light transport, and Bayesian RJ-MCMC reconstruction to form 2.5D representations of hidden scenes over a 180° FOV using few measurements and a minimal aperture. Simulations and experiments demonstrate accurate room-scale reconstructions, including object localization and approximate height estimation, outperforming conventional methods under small-aperture constraints. Future work includes: increasing angular sampling for finer resolution; relaxing modeling assumptions (non-vertical, non-planar, varying albedo surfaces; thick/nonlinear occluders); multi-wavelength or supercontinuum illumination and spectrally filtered detection; fusion with passive RGB corner-camera data; higher-power or longer-wavelength, eye-safe hardware for faster acquisition; algorithmic acceleration and dedicated processing; and adaptive/multiresolution acquisition strategies.

• Coarse angular/vertical resolution due to the small number of illumination positions and the facet-based 2.5D model.
• Scene model assumes vertical planar facets extending from the ground and constant albedo; errors can occur for floating, cantilevered, curved, or complex objects, and for thick/nonlinear occluders.
• Single-wavelength operation lacks spectral information; color/material cues are not captured.
• Not real-time: acquisition requires 20–60 s per spot and reconstruction ~100 s; faster hardware and software are needed for real-time applications.
• Sensitivity to noise and ambient/visible-scene contributions increases variance, especially at higher illumination angles; distant or highly oblique/occluded walls are most challenging to recover accurately.
• Assumes near-confocal geometry and a thin, linear vertical edge; deviations may require modified modeling and calibration.