Histopathology, crucial for diagnostics, traditionally relies on labor-intensive and error-prone staining procedures like H&E staining. While label-free microscopy offers alternatives, limitations in visualization and clinical applicability remain. This research addresses these limitations by developing a DL-based framework for automated analysis of label-free photoacoustic histology (PAH) images, specifically focusing on human liver cancers. The existing methods have several shortcomings. For example, traditional deep learning methods for virtual staining require paired images, introducing challenges in image registration. Unsupervised methods like CycleGAN, while effective with unpaired data, can struggle when one domain contains more information than the other. CUT methods offer an improvement but still lack explainability in their results. Existing deep learning-based histological image analysis (HIA) often relies on conventional histopathological images and are limited to single tasks. This work aims to overcome these limitations by presenting a unified framework performing virtual staining (using E-CUT, improving upon CUT by integrating saliency loss and integrated gradients for explainability), segmentation (using U-Net), and classification (using StepFF, a novel stepwise feature fusion method). This integrated approach aims to translate label-free PAH images into clinically interpretable, H&E-like images while providing accurate and sensitive analysis results. The expected benefits include faster and more efficient pathological analysis and reduction in cost and error associated with traditional staining methods.

Several optical microscopy techniques attempt to address the challenges of sample preparation and staining quality. Light-sheet microscopy, while rapid, requires additional chemical procedures. Label-free modalities like bright-field microscopy, optical coherence tomography (OCT), and autofluorescence microscopy simplify preparation but lack the specificity and information richness of H&E staining. Raman microscopy and spectroscopic OCT offer biochemical composition analysis but suffer from low signal sensitivity. Photoacoustic microscopy (PAM), particularly UV-PAM, offers a solution by selectively highlighting biomolecules based on optical absorption, visualizing cell nuclei without staining. However, translating label-free PAH images into clinically useful information remains a challenge. Recent advances in deep learning (DL)-based image processing, including virtual staining and HIA, have shown potential. DL-based virtual staining aims to mimic the characteristics of various staining styles, but traditional methods require paired images, making them cumbersome. Unsupervised methods like CycleGAN offer a solution, but can suffer from inconsistencies and are computationally expensive. CUT has improved upon these, but the lack of explainability limits its application in safety-critical medical contexts. DL-based HIA methods are often designed for conventional images and lack compatibility with label-free images or are limited to a single task.

This study employed a three-stage deep learning framework for automated histological image analysis of label-free PAH images of liver cancer. The first stage involves virtual staining using the proposed E-CUT method. E-CUT, built upon CUT, incorporates saliency loss and integrated gradients to improve explainability and the accuracy of virtual H&E (VHE) staining, transforming grayscale PAH images into color-coded images resembling H&E-stained tissue. The saliency loss helps preserve image content and visualize important morphological features. Integrated gradients reveal the features contributing to the discriminator’s decisions, enhancing traceability. This VHE image is then used in the second stage for segmentation using a U-Net architecture to extract relevant features such as cell area, cell count, and the distance between cell nuclei. The third stage uses a novel stepwise feature fusion method (StepFF) for classification of cancerous and non-cancerous cells. StepFF combines deep feature vectors (DFVs) from PAH, VHE, and the segmented images to improve the classification accuracy. The performance of StepFF is compared with traditional H&E-based classification. A UV-PAM system with a lateral resolution of ~1.2 µm was used to acquire PAH images of human liver samples. These images were then processed using the E-CUT, U-Net, and StepFF models. The performance of the virtual staining models was quantitatively assessed using Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) metrics, comparing the generated VHE images to real H&E images. The classification accuracy and sensitivity of StepFF were evaluated using a dataset of liver cancer samples, and its performance was compared to a traditional PAH-based classifier. Three pathologists independently evaluated the results to confirm the clinical applicability of the method.

The proposed E-CUT method for virtual staining significantly outperformed CycleGAN, E-CycleGAN, and CUT, as measured by FID and KID scores. E-CUT achieved the lowest FID (50.91) and KID (0.2451), indicating a high degree of similarity between the generated VHE images and real H&E images. The U-Net based segmentation successfully extracted key features from the VHE images. The StepFF method, which combines DFVs from PAH, VHE, and segmented images, achieved a 98.00% classification accuracy, a considerable improvement over the 94.80% accuracy of conventional PAH classification. Importantly, StepFF achieved a sensitivity of 100% based on the evaluation of three pathologists, demonstrating its potential for clinical application. The visual comparisons clearly showed that E-CUT effectively preserved the morphology of cell nuclei in the VHE images, making them clinically useful. The superior performance of the entire framework highlights the synergy between the virtual staining, segmentation, and classification components, leading to a more accurate and robust diagnostic tool.

The results demonstrate the success of the proposed DL-based framework in improving the diagnostic capabilities of label-free PAH. The combination of E-CUT, U-Net, and StepFF provides a significant improvement over conventional PAH analysis, achieving high accuracy and sensitivity. The explainability incorporated into E-CUT contributes to trustworthiness and transparency. The high sensitivity (100%) is particularly encouraging, indicating the framework's potential for reliable clinical diagnosis. The improvements in accuracy and sensitivity are attributable to the synergistic effect of combining multiple features from different modalities, leading to a more comprehensive and robust analysis. The study’s focus on liver cancer provides a strong foundation for future applications to other types of cancer and tissue types. Future research could explore the generalizability of the model to larger and more diverse datasets, further enhancing its clinical applicability.

This research successfully developed a novel DL-based framework for automated HIA in label-free PAH, combining virtual staining (E-CUT), segmentation (U-Net), and classification (StepFF). The framework demonstrates high accuracy and sensitivity in classifying liver cancers. E-CUT's explainability enhances trust and transparency. This approach shows substantial promise as a practical clinical strategy for digital pathology, potentially replacing laborious traditional staining methods with a faster, more cost-effective, and objective analysis.

The study's findings are based on a relatively limited dataset of liver cancer samples. Future work should involve validation on a larger and more diverse dataset, including different types of cancers and tissue types. The perfect registration between PAH and H&E images was not guaranteed because two adjacent tissue slices were used for comparison. Further research could explore more sophisticated image registration techniques to address this limitation. The generalizability of the proposed framework to other imaging modalities or clinical settings also needs to be investigated.