Deep learning-based virtual staining, segmentation, and classification in label-free photoacoustic histology of human specimens

The study addresses the challenge that conventional histopathology relies on labor-intensive, time-consuming, and error-prone staining procedures such as hematoxylin and eosin (H&E), which can be limiting when sample quantity is insufficient and throughput demands are high. Although label-free imaging modalities (e.g., bright-field, OCT, autofluorescence, Raman microscopy, spectroscopic OCT, deep-UV, photoacoustic microscopy/remote sensing) reduce or eliminate staining, they frequently lack the molecular specificity, sensitivity, and familiar color-coded visualization that pathologists rely on. Photoacoustic histology (PAH), especially UV-PAM targeting strong DNA/RNA absorption, can visualize nuclei label-free with high sensitivity, but still does not yield H&E-like whole-slide images directly suitable for routine diagnosis. Deep learning-based virtual staining and histological image analysis hold promise; however, prior methods often require paired data (difficult registration), or unsupervised approaches like CycleGAN can struggle with structural preservation and are resource intensive. There is a need for a unified, explainable DL framework that converts label-free PAH into clinically interpretable, H&E-like images and performs downstream analysis (segmentation and cancer classification) with high sensitivity and accuracy.

Prior work on virtual staining leveraged supervised networks requiring paired images, which are hard to obtain due to registration challenges. Unsupervised CycleGAN-based approaches enabled translation between unpaired domains but can suffer from weak structural fidelity, potential failure when domain information content is imbalanced, and computational overhead from two generators/discriminators. Contrastive unpaired translation (CUT) improved quality and efficiency via patch-wise contrastive learning by maximizing mutual information between input and generated images. Explainability remains a concern for safety-critical medical applications, motivating integration of saliency and attribution methods. In histological image analysis, deep learning has been applied to classification, detection, and nuclei segmentation predominantly on standard stained images, with limited compatibility with label-free modalities and often focusing on single tasks. These gaps motivate an explainable, multi-task framework tailored to label-free PAH.

Data acquisition: Human liver tissue sections (FFPE) were imaged with a UV-PAM system for label-free PAH. A 266 nm pulsed UV laser (20 kHz repetition) illuminated the sample; the laser and acoustic detection beams were co-scanned via a MEMS mirror, and acoustic signals were collected through an opto-ultrasound combiner and a 20 MHz ultrasound transducer with an acoustic lens. The system achieved ~1.2 µm lateral resolution and imaged a 700 × 1000 µm^2 field in ~35 s at 1.0 µm/pixel step size. Whole-slide mosaics were formed by stitching 123 tiles (total area ~10.5 × 8.0 mm^2). Corresponding H&E whole-slide images were obtained from adjacent slices. Virtual staining (E-CUT): An unsupervised explainable CUT framework was developed to translate grayscale PAH to virtual H&E (VHE). The base CUT architecture used patch-wise contrastive learning (PatchNCE loss) and an adversarial discriminator. Explainability components included (1) saliency masks and a saliency loss to enforce preservation of salient structural content between PAH and VHE and improve training stability, and (2) integrated gradients applied to the discriminator to attribute which input features influence real/fake decisions, highlighting nuclear features as training progresses. E-CUT was compared against CycleGAN, explainable CycleGAN (E-CycleGAN), and standard CUT. Performance was assessed with Fréchet Inception Distance (FID) and Kernel Inception Distance (KID) on 100 test tiles. Segmentation: A U-Net-based model segmented histological features on VHE images to enable automated extraction of quantitative descriptors, including cell area, cell count, and inter-nuclear distance. Classification (StepFF): A deep learning classifier using a stepwise feature fusion (StepFF) strategy combined deep feature vectors (DFVs) derived from PAH, VHE, and segmentation outputs to classify tiles as cancerous versus noncancerous liver tissue. Performance was compared to a conventional PAH-only classifier and evaluated with accuracy and sensitivity, including assessment by three pathologists. Evaluation and visualization: Visual comparisons assessed morphological preservation of nuclei and cytoplasm versus ground truth H&E from adjacent sections, acknowledging imperfect registration. Explainability outputs included saliency maps (content preservation) and integrated gradients (feature importance in discrimination).

- Virtual staining quality: E-CUT produced VHE images that more closely resemble real H&E than competing methods, with improved preservation of nuclear morphology and overall staining quality. Quantitatively on 100 test tiles, FID (lower is better): CycleGAN 67.76, E-CycleGAN 61.91, CUT 54.87, E-CUT 50.91. KID×100 (lower is better): CycleGAN 2.2900, E-CycleGAN 1.6314, CUT 0.6428, E-CUT 0.2451. - Explainability: Saliency loss maintained consistent salient structures from PAH to VHE, and integrated gradients in the discriminator increasingly focused on nuclear features over training, aligning with histological relevance. - Segmentation: U-Net successfully segmented key features in VHE images, enabling extraction of cell area, cell counts, and inter-nuclear distances. - Classification: StepFF combining DFVs from PAH, VHE, and segmentation achieved 98.00% accuracy, outperforming conventional PAH-only classification at 94.80% accuracy. Sensitivity reached 100% in evaluations by three pathologists, demonstrating clinical applicability. - PAH imaging performance: Achieved ~1.2 µm lateral resolution; ~35 s per 700 × 1000 µm^2 tile; whole-slide mosaic of 123 tiles (10.5 × 8.0 mm^2).

The proposed framework directly addresses the need to render label-free PAH images into diagnostically familiar and interpretable forms while enabling downstream quantitative analysis and robust classification. By uniting contrastive learning with saliency-based constraints and attribution, E-CUT preserves critical nuclear and cytoplasmic morphology and yields VHE images with distributional similarity to real H&E, as evidenced by substantially lower FID/KID scores than CycleGAN-based methods. The explainability modules increase trust by visualizing which structures are emphasized and how the discriminator forms its decisions. Segmentation on VHE images reliably extracts biologically meaningful features, supporting quantitative histology. Critically, stepwise fusion of DFVs from complementary sources (PAH raw signal-derived content, VHE morphology/color cues, and segmentation-driven structure) boosts classification performance, achieving 98% accuracy and 100% sensitivity in pathologist-based evaluation, which highlights the method’s potential for sensitive clinical screening and diagnosis. Together, these results indicate that a unified, explainable virtual staining–segmentation–classification pipeline can improve the utility of label-free PAH for digital pathology.

This work introduces an explainable, multi-task deep learning framework that converts label-free PAH images of human liver tissue into virtual H&E, segments salient histological features, and classifies cancer with high accuracy. E-CUT advances virtual staining by combining contrastive learning with saliency loss and integrated gradients, improving structural preservation and interpretability. The subsequent U-Net segmentation and StepFF classification leverage complementary information to enhance diagnostic performance, achieving 98% accuracy and perfect sensitivity in pathologist evaluation. These advances suggest strong potential for practical clinical deployment of label-free PAH in digital pathology. Future work can extend validation across broader cohorts, tissue types, and institutions, and integrate the pipeline into end-to-end clinical workflows.

Virtual staining results do not perfectly match real H&E morphology due to the use of adjacent, not perfectly registered, tissue sections as ground truth. While E-CUT improves structural preservation, differences inherent to unpaired translation and inter-section variability remain. The reported evaluations focus on human liver samples; broader multi-tissue, multi-center validation is not detailed here.