Al co-pilot bronchoscope robot

The study addresses disparities in access to high-quality bronchoscopy stemming from a shortage of experienced practitioners, especially in underdeveloped regions. Bronchoscopy is critical for detecting and managing lung diseases but requires substantial skill to navigate airways safely, avoid mucosal injury, and maintain central positioning. Existing robotic systems improve reach and stability yet retain steep learning curves and costs, limiting widespread adoption. The research aims to develop an AI-assisted bronchoscope robot that enables novice doctors to perform safe, efficient, and expert-level steering during lung examinations, thereby improving equity in care and reducing training burdens.

Recent robotic bronchoscopy platforms, notably the Monarch Platform and the Ion Endoluminal System, extend reach into peripheral bronchi (approximately 9th vs. 6th generations) with favorable diagnostic yields (about 81.7%–92% for nodules 14.8–21.9 mm) and low complication rates. Additional academic systems address sensing and control challenges. Concurrently, AI and computer vision methods have been explored for image-guided navigation, real-time localization, tracking, and path planning, potentially reducing cognitive load. However, methods relying on preoperative CT registration can suffer from misregistration and field-of-view mismatches, raising safety concerns (e.g., pneumothorax, bleeding). Overall, despite advances, telerobotic bronchoscopy still faces a steep learning curve and limited autonomy, motivating AI-shared control approaches.

Hardware: The bronchoscope robot integrates with a robotic arm for adjusting intubation posture and is teleoperated via a remote console. The tendon-driven steering system uses four linear motors for actuation and four force sensors for force measurement. A magnetic adsorption interface enables rapid, user-friendly catheter replacement. The catheter has a stiff proximal section (braided mesh) and a flexible distal section (small hinge joints), both covered with TPU. The tip includes two LEDs and a microcamera. Two catheter types are provided: 3.3 mm diameter with a 1.2 mm working channel, and 2.1 mm diameter without a working channel, enabling deep bronchial access.

AI-human shared control: During procedures, a doctor issues discrete commands (left, right, up, down, forward). A policy network takes the live bronchoscopic image and the human command to predict continuous steering actions (pitch/yaw angle rates), which are converted via inverse kinematics and a low-level controller into tendon displacements, closing the control loop and maintaining lumen centering for safety.

Policy network: A multi-task architecture performs main-task steering action prediction and side-task depth estimation. It uses a ResNet-34 feature extractor, a transposed-convolution depth decoder with skip connections, and five action heads (MLPs) selected by a five-way switch according to the discrete command. Features are flattened to 512-d for action heads.

Training data and virtual environment: An airway model segmented from preoperative CT establishes a virtual bronchoscopy environment with extracted centrelines as reference paths. A simulated robot renders bronchoscopic images and depths. An Artificial Expert Agent (AEA) automatically generates human commands and ground-truth steering actions using privileged pose information and centreline waypoints, producing (image, depth, command, action) samples. Dataset aggregation (DAgger) enables on-policy imitation, mitigating distribution mismatch, with automatic labels eliminating human intervention.

Sim2Real adaptation and randomisation: A structure-preserving unpaired image translation GAN with a depth constraint translates Sim-style (pink-textured) images into realistic styles (e.g., Real-, Phantom-, Clinical-style) while preserving bronchial structure to keep action labels valid. Domain randomisation augments variability by random roll rotations, varying light intensity, adding command noise when robot-wall distance <1 mm, and generic image augmentations. This improves generalisation from simulation to real procedures.

Evaluation: Simulation training and tests used airway models with 5th-generation bronchi. Training environments contained 74 and 84 reference paths; a test environment with realistic texture had 60 paths. In vitro evaluations used realistic human bronchial phantoms with simulated respiratory behavior. In vivo validation used live porcine lungs, with an expert and an attending doctor (the latter assisted by the AI co-pilot) steering along two porcine bronchial paths; actuation displacement/force, operation error (centering), number of interventions, and workload (NASA-TLX) were measured across repeated trials.

Simulation: The policy trained with Sim-style images plus domain adaptation and randomisation (Sim+A+R) achieved the best performance on 60 test paths: success rate ~93.3%, successful path ratio ~98.9 ± 4.7%, and lowest trajectory error ~1.04 ± 0.21 mm. This outperformed Sim+A (~80.0% success; ~96.4 ± 7.4% path ratio; ~1.37 ± 0.26 mm error), Real (~81.8%; ~96.5 ± 8.0%; ~1.23 ± 0.28 mm), Sim+A(baseline) (~71.8%; ~92.9 ± 12.4%; ~2.57 ± 0.54 mm), and Sim (~31.8%; ~75.2 ± 22.6%; ~3.36 ± 0.66 mm). The domain adaptation method preserved structure and improved image similarity (SSIM: Real 0.91 vs 0.80; Phantom 0.95 vs 0.70; Clinical 0.96 vs 0.78) and PSNR (Real 25.16 dB vs 12.25 dB) compared with AttentionGAN.

In vivo (porcine): Both the expert and the AI-assisted attending doctor navigated beyond 5th-generation bronchi (~2.5 mm diameter) with nearly identical visualization. AI assistance yielded smoother actuation displacement/force profiles and overall lower means and fluctuation ranges. Across eight repeated trials, the attending doctor with AI achieved lower operation error (11.38 ± 0.16 pixels) than the expert (16.26 ± 0.27 pixels), corresponding to a mean 3D positioning error <0.73 mm. The AI-assisted group required significantly fewer interventions, indicating reduced physical and cognitive load, corroborated by lower NASA-TLX workload scores.

Findings demonstrate that an AI-human shared control paradigm can safely and efficiently guide bronchoscopic navigation, enabling novice operators to attain expert-level performance. The structure-preserving Sim2Real strategy effectively bridges the gap between simulated and real bronchoscopic imagery, yielding robust generalisation across different textures and settings. Hardware innovations (rapid magnetic catheter exchange; thin, flexible catheters) combined with learned policies maintain lumen centering, reduce actuation effort variability, and decrease the number of operator interventions and perceived workload. Explainability analyses (GradCAM) showed the policy focuses on bronchial lumens and distances to walls, aligning with safety goals. Collectively, the system addresses key barriers—training demands and resource disparities—by lowering the learning curve and standardizing procedural quality.

The AI co-pilot bronchoscope robot combines cost-effective hardware with an AI-human shared control algorithm to enhance safety, accuracy, and efficiency in bronchoscopy. It empowers novice doctors to navigate deep bronchial airways with expert-level precision in simulation, in vitro, and in vivo porcine studies. The Sim2Real training pipeline generalizes across styles while preserving critical structure. Future work includes expanding clinical validation, integrating broader interventional capabilities (e.g., biopsy workflows), and adapting the approach to other procedures requiring precise endoluminal navigation, thereby contributing to reduced healthcare disparities.