Multimodal graph representation learning for robust surgical workflow recognition with adversarial feature disentanglement

Surgical workflow recognition is crucial for automating tasks, aiding decision-making, and training surgeons. However, data corruption (e.g., occlusion, transmission errors) hinders performance. This paper proposes GRAD, a robust multimodal graph-based approach integrating vision and kinematic data. GRAD uses a Multimodal Disentanglement Graph Network (MDGNet) to capture fine-grained visual information and model vision-kinematic relationships. A Vision-Kinematic Adversarial (VKA) framework aligns feature spaces, and a Contextual Calibrated Decoder enhances robustness. Experiments on two public datasets show high accuracy (86.87% and 92.38%) and robustness to data corruption.

Long Bai, Boyi Ma, Ruohan Wang, Guankun Wang, Beilei Cui, Zhongliang Jiang, Mobarakol Islam, Zhe Min, Jiewen Lai, Nassir Navab, Hongliang Ren