Real-time molecular imaging of near-surface tissue using Raman spectroscopy

Clinical imaging modalities such as MRI, CT, PET, and ultrasound are routinely used to screen for cancer, localize suspicious lesions, and guide surgery, often providing morphological or anatomical contrasts while overlooking molecular composition. Raman spectroscopy offers label-free, non-contact, and non-destructive access to intrinsic molecular fingerprints and has shown strong potential to detect and delineate cancer from healthy tissue. Despite this promise, handheld fiber-optic Raman probes lack imaging capability because spectral measurements are not co-registered with probe position; analyses are often performed post-acquisition; and 3D surface topology is typically ignored. Prior attempts to enable imaging have used multi-fiber arrays to sample multiple points simultaneously, or mechanical/robotic arms to physically track probe position, but these solutions are bulky, complex, or unsuitable for large-area or clinical use. Computer vision-based positional tracking offers a compact alternative by tracking an aiming laser with conventional cameras. Real-time data processing combined with augmented reality (AR) and mixed reality (MR) could make molecular information perceptible directly in the surgical field. To address these challenges, the authors propose a fiber-based Raman imaging approach combining real-time spectral analysis, computer vision positional tracking, photometric stereo for 3D topology, and AR/MR visualization for real-time, near-surface molecular imaging and boundary demarcation.

The paper situates its contribution within several lines of prior work: (1) Raman spectroscopy has been widely studied for in vivo clinical applications and cancer delineation due to its label-free molecular specificity. (2) Imaging via fiber probes has been attempted with multi-fiber arrays to sample multiple points simultaneously, but these approaches impose high power densities and limited fields, hindering translation. (3) Probe position tracking for large-scale imaging has used passively coordinated mechanical arms and robotic arms, which are bulky and complex. (4) Computer vision-based tracking with conventional cameras can register the aiming laser position without complex mechanics and has been applied in other optical modalities. (5) While some Raman systems offer online analysis, they are typically limited to single-point diagnostics rather than image formation for tasks like margin delineation. (6) AR/MR techniques have enhanced guidance in modalities such as CT, OCT, MRI, and FLIM, motivating their integration with molecular imaging to improve real-time visualization and surgical precision.

System overview: A handheld fiber-optic Raman probe acquires spectra while a brightfield camera simultaneously tracks the laser spot position on the sample. A photometric stereo module (four light sources) reconstructs the 3D surface topology prior to Raman scanning. Real-time processing links spectral and positional data to build molecular images, which are visualized as AR on a screen (2D overlay or 3D overlay on the reconstructed surface) and as MR by back-projecting the molecular map onto the physical sample using a projector. Data processing (Block 1): A reference database of Raman spectra is assembled for relevant components (e.g., lipid, bone, protein, collagen; and additional sample-specific components via sparse sampling). Each acquired spectrum undergoes ALS-based baseline correction and unity normalization. The corrected spectra are fit as non-negative linear combinations of database spectra using non-negatively constrained least squares, yielding a coefficient vector representing relative component concentrations (s ≈ Pc + ε). These coefficients drive molecular concentration map construction for each database component. Positional tracking (Block 2a): During acquisition, the brightfield camera stream is processed to detect the excitation laser spot via color segmentation, thresholding, and ellipse fitting. The intensity threshold preserves the bright Rayleigh-scattered spot and mitigates defocused measurements; frames failing ellipse fitting are discarded. From the fitted ellipse, the center and minor radius define the measurement location and an adaptive drawing radius for updating molecular maps. 3D surface reconstruction (Block 2b): Photometric stereo using four illumination directions generates a 3D height map of the sample surface before Raman scanning. The reconstructed surface is later used to map the molecular distributions for 3D AR visualization. Real-time image formation and visualization (Block 3): For each new frame, a circle centered at the tracked laser position is drawn in each component’s 8-bit image plane with an auto-scaled diameter determined by the fitted ellipse minor radius r and the inter-frame center displacement D (scaled roughly as 0.5r, r, or 2r depending on D). Pixel intensities are updated from the mean of non-negative fit coefficients at that location, normalized to the maximum observed coefficient and scaled to 0–255. A data gridding function fills gaps to produce continuous maps. AR overlays molecular maps on the live brightfield view or on the 3D surface model. MR projects the molecular map onto the sample, registered via the camera-projector geometry. Acquisition parameters and performance: The system achieves a spectral sampling frequency of 10 Hz. Representative measurements used laser power ~100 mW and 0.1 s acquisition per point. Spatial resolution is 0.5 mm in-plane, and topology (height) resolution is 0.6 mm via photometric stereo. The handheld scanning supports large-area coverage within minutes. Experimental demonstrations: (1) Porcine brain with two coated regions (lipid-rich compound and N-acetyl-4-aminophenol) plus native gray matter; (2) freshly excised ex vivo tumor tissue (sarcoma), with components including collagen and epithelial tissue; (3) additional ex vivo lipoma tissue (supplementary).

- Achieved real-time handheld Raman molecular imaging with AR/MR visualization mapped onto 3D surfaces. - System performance: spatial resolution 0.5 mm (transverse), topology resolution 0.6 mm, spectral sampling frequency 10 Hz; capable of imaging large tissue areas within minutes. - Porcine cerebrum demo: 856 points acquired in 260 s at ~100 mW and 0.1 s/point; successful delineation of lipid-rich coating, gray matter, and pharmaceutical compound (N-acetyl-4-aminophenol) with molecular maps overlaid on reconstructed 3D topology; AR and MR enabled intuitive boundary visualization. - Ex vivo tumor demo (approx. 5.2 cm × 4.1 cm sample): 994 points in 184 s with the same acquisition settings; clear molecular boundaries identified (e.g., collagen, epithelial tissue) with AR overlays and MR projection; data gridding produced continuous boundary maps. - Mixed reality projection allowed direct visualization of molecular boundaries on the specimen; for more transparent samples, MR projection was applied to the final image rather than updated in real time. - Demonstrations indicate feasibility for rapid, intuitive assessment of macromolecular distributions and boundary demarcation in clinically relevant tissues.

The study addresses the lack of imaging capability in handheld fiber-optic Raman probes by integrating computer vision-based positional tracking with real-time spectral unmixing and 3D surface reconstruction. This enables co-registered, near-surface molecular imaging at clinically practical speeds and resolutions without bulky mechanical tracking systems. The AR overlays enhance interpretability by fusing molecular maps with native visual context, and MR projection brings molecular boundaries directly into the surgical field, potentially improving precision in margin assessment and resection. Demonstrations on porcine brain and ex vivo tumor tissue show that the approach can differentiate molecularly distinct regions in minutes, supporting its suitability for intraoperative guidance. By leveraging a component database and NNLS fitting, the system provides flexible, extensible molecular assessments tailored to target tissues. While detection of very low concentrations remains challenging and MR updating may be limited by sample transparency, the pipeline demonstrates a practical path toward real-time, image-guided molecular diagnostics and surgery.

The work presents a handheld, fiber-based Raman imaging system that unifies real-time spectral analysis, computer vision positional tracking, and photometric stereo to generate AR/MR molecular visualizations on 3D tissue surfaces. It achieves sub-millimeter spatial and topological resolution with 10 Hz spectral sampling and demonstrates rapid, accurate delineation of molecular boundaries on porcine brain and ex vivo tumor tissue. These results highlight the potential for clinical translation to real-time, image-guided diagnostics and surgical margin assessment. Future directions include optimizing optical parameters to improve sensitivity for very low analyte concentrations, enhancing MR real-time performance in transparent tissues, refining projector-sample registration, and expanding component databases for broader tissue types and disease states.

- Detection of very low concentrations is challenging in the current implementation, as noted in supplementary results; optical parameter optimization is needed for improved sensitivity. - Mixed reality visualization was not updated in real time for more transparent samples due to increased sample transparency; only the final molecular image was projected. - Demonstrations are ex vivo or on phantoms; in vivo performance and robustness were not shown here. - The approach relies on accurate ellipse fitting of the laser spot; significant defocus leads to discarded frames, potentially reducing sampling density in challenging conditions.