Multimodal graph representation learning for robust surgical workflow recognition with adversarial feature disentanglement

Robot-assisted minimally invasive surgery (RMIS) has revolutionized modern medicine, offering enhanced precision and control. The increasing adoption of RMIS necessitates intelligent systems capable of recognizing and understanding surgical workflows to automate tasks, support decision-making, and train surgeons. Vision-based systems, utilizing high-resolution cameras, can identify instruments and gestures, but are susceptible to occlusions and varying lighting. Kinematic-based systems, analyzing instrument movements, are less affected by visual interference but lack contextual information. Multimodal approaches, fusing vision and kinematic data, offer a promising solution by combining the strengths of both modalities. However, current methods often lack robustness against data corruption and don't fully explore the rich features within each modality. This research addresses these limitations by proposing a novel framework that leverages graph representation learning, adversarial feature disentanglement, and contextual calibration to achieve robust and accurate surgical workflow recognition.

Existing research on surgical workflow recognition utilizes various techniques. Early methods relied on hand-crafted features and Hidden Markov Models (HMMs). Deep learning approaches, employing Convolutional Neural Networks (CNNs) for spatial feature extraction and Recurrent Neural Networks (RNNs) like LSTMs and Temporal Convolutional Networks (TCNs) for temporal modeling, have significantly improved performance. Multimodal methods, integrating vision and kinematics, have also emerged, often employing simple fusion techniques. Graph representation learning has shown promise in multimodal fusion tasks, but its application to surgical workflow recognition is still limited, particularly concerning robustness to data corruption. Network calibration techniques aim to improve the reliability of predictions by aligning predicted probabilities with true outcome likelihoods, but these methods haven't been extensively adapted to the complexities of graph neural networks in surgical data.

The proposed GRAD framework consists of five key modules: (1) Multimodal Disentanglement Graph Network (MDGNet): This module extracts features from both vision and kinematic data. For vision, it uses a ResNet-18 backbone followed by TCNs, and disentangles features across spatial, wavelet, and Fourier domains. For kinematics, it uses parallel LSTM and TCNs to capture temporal patterns. (2) Vision-Kinematic Adversarial (VKA) Training: This module utilizes adversarial learning to align the feature distributions of the vision and kinematic modalities in a shared embedding space, enhancing robustness to noise. (3) Multimodal Graph Learning: A Graph Attention Network (GAT) models the relationships between visual and kinematic features represented as nodes, allowing for dynamic weighting of inter-modal relationships. (4) Calibrated Prediction Decoder: This module integrates the original multimodal features with the graph embeddings, then uses a calibrated cross-entropy loss function with a minimal-entropy regularization term to increase prediction confidence and robustness against corrupted data. (5) Workflow Prediction Decoder: Finally, a fully connected layer maps the combined embeddings to the predicted surgical workflow. The entire framework is trained end-to-end.

Extensive experiments were conducted on two publicly available datasets: MISAW and CUHK-MRG. On the MISAW dataset, GRAD achieved an accuracy of 86.87%, outperforming state-of-the-art methods by a significant margin. The Edit Score, evaluating temporal consistency, reached 97.50%. Similarly, on the CUHK-MRG dataset, GRAD achieved 92.38% accuracy. Ablation studies demonstrated the effectiveness of each module: visual representation disentanglement improved accuracy, VKA training increased robustness, and the calibrated decoder boosted confidence. Robustness experiments showed that GRAD maintained high accuracy even under various types of data corruption (noise, blur, occlusion, and digital corruptions) across different severity levels, significantly outperforming the baseline MRG-Net model. For example, at the highest corruption level, GRAD consistently maintained accuracy above 75% across various corruption types whereas MRG-Net fell consistently below 75%.

The results demonstrate the efficacy of GRAD's multimodal approach and its robust design. The integration of vision and kinematic data, coupled with feature disentanglement and adversarial training, effectively addresses the limitations of single-modality approaches and enhances performance in challenging scenarios. The high accuracy and robustness to data corruption highlight the potential of GRAD for real-world applications in surgical assistance and training. The superior performance compared to existing state-of-the-art methods suggests that GRAD's novel approach of combining multimodal graph representation learning with adversarial feature disentanglement and contextual calibration is a significant advancement in surgical workflow recognition.

GRAD offers a robust and accurate framework for surgical workflow recognition. Key contributions include the novel integration of multimodal graph representation learning, adversarial feature disentanglement, and contextual calibration. Future research should focus on addressing data imbalance issues, exploring more efficient training strategies, and validating the model on larger, more comprehensive datasets that incorporate real-world clinical scenarios and variability.

While GRAD shows superior performance, several limitations should be noted. The datasets used contain a limited number of samples and potentially exhibit class imbalance, affecting the generalizability of the results. Training the model is computationally expensive. The evaluation was performed on benchmark datasets, and further validation on diverse real-world surgical data is needed to fully assess its clinical applicability. The limited duration of some surgical gestures in existing datasets could lead to inaccurate recognition. Addressing these limitations would further strengthen the clinical relevance of the GRAD model.