Current medical imaging techniques like MRI, CT, PET, and ultrasound primarily provide morphological information, neglecting molecular composition. Raman spectroscopy, a label-free, non-invasive technique providing molecular fingerprints, offers a promising alternative. However, current fiber optic probe-based Raman spectroscopy lacks real-time imaging capabilities. Existing approaches using multiple fibers or physical tracking are either limited in resolution or complex. This research addresses these limitations by proposing a computer vision-based positional tracking system, integrated with photometric stereo and augmented/mixed reality for real-time visualization of molecular distributions on 3D surfaces. This system aims to improve the speed and ease of assessing molecular boundaries for applications like tumor margin detection.

Numerous studies have demonstrated the potential of Raman spectroscopy for detecting and delineating cancerous from healthy tissues. However, current fiber optic probe-based Raman systems primarily acquire data from individual locations, hindering image formation. Previous attempts to create Raman images have employed multiple optical fibers or physical tracking methods, which suffer from drawbacks like high power densities or increased system complexity. Computer vision-based positional tracking offers a more flexible and less cumbersome alternative, already successfully implemented in other imaging modalities like fluorescence lifetime imaging. Combining real-time data processing with augmented and mixed reality visualization can enhance the diagnostic and surgical applications of Raman spectroscopy.

The developed system integrates Raman spectral acquisition, computer vision-based positional tracking, photometric stereo for 3D surface reconstruction, and real-time data processing. A handheld fiber-optic Raman probe acquires spectra, while a brightfield camera tracks the laser point for positional information. Photometric stereo reconstructs the sample's 3D surface. Real-time data analysis involves baseline correction, normalization, and non-negative least squares fitting of acquired spectra against a reference spectral database. The resulting concentration maps are visualized in augmented reality (overlayed on brightfield images or 3D models) and mixed reality (projected onto the sample). A formula is provided to calculate the diameter of a circle representing each measurement point based on ellipse fitting parameters and Euclidean distance between consecutive frames. The intensity of each pixel in the component image planes is calculated based on the mean value of the non-negative estimation results, normalized, and scaled to 8-bit for visualization.

The system successfully imaged 3D structured surfaces and ex-vivo tumor tissues. A porcine cerebrum coated with lipid-rich and pharmaceutical compounds was scanned, demonstrating the ability to visualize the spatial distribution of these compounds. The system achieved a spatial resolution of 0.5 mm and a topology resolution of 0.6 mm with a spectral sampling frequency of 10 Hz. In 260 seconds, 856 data points were acquired and processed. The 3D representation, combining molecular information with photometric stereo data, offered intuitive visualization. Measurements on an ex-vivo tumor sample (5.2 cm x 4.1 cm) similarly showed distinct molecular boundaries within the pathological tissue, with 994 points acquired in 184 seconds. Results on ex-vivo cancer lipoma tissues are also included. The mixed reality visualization effectively projected molecular information directly onto the sample surface. The entire data acquisition and visualization process, including augmented and mixed reality, was demonstrated in supplementary videos. The system successfully differentiated various biomedical components (lipid, bone, protein, collagen) in real-time.

The developed system successfully addresses the challenges of real-time Raman imaging using fiber optic probes. The combination of computer vision-based tracking, photometric stereo, and augmented/mixed reality provides a robust and user-friendly platform for real-time molecular visualization. The high spatial resolution, fast acquisition speed, and intuitive visualization make it a valuable tool for clinical applications. The successful application to various biological samples, including ex-vivo tumor tissues, highlights its potential for image-guided diagnosis and surgical resection. The ability to visualize molecular boundaries in real-time offers significant advantages over conventional post-acquisition analysis methods, allowing for more informed decision-making during surgical procedures. The system's capabilities provide a significant advancement in the field of Raman spectroscopy for biomedical imaging.

This research demonstrates a novel real-time fiber optic probe-based Raman imaging system combining computer vision, photometric stereo, and augmented/mixed reality visualization. This innovative approach overcomes limitations of previous methods, offering high-resolution, fast acquisition, and intuitive 3D visualization of molecular distributions in biological samples. The system shows great potential for clinical translation, enabling improved image-guided diagnosis and surgical resection. Future work could focus on improving the system's sensitivity and expanding the range of detectable molecules for broader clinical applicability.

While the current system demonstrates significant advancements, some limitations remain. The current implementation uses high concentrations of the compounds for demonstration purposes. Future improvements in optical parameters are needed to enhance sensitivity for detecting lower concentrations relevant in clinical settings. Real-time mixed reality projection was not perfectly synchronized in all examples due to limitations in sample transparency. Further development and optimization of the system are necessary to address these points and improve the robustness for routine clinical usage.