Swimming microorganisms exhibit sophisticated navigation strategies involving gait switching, such as run-and-tumble or reverse-and-flick, to explore their environment and locate targets. This adaptive, multimodal gait-switching is highly desirable for developing smart artificial microswimmers capable of autonomous navigation for biomedical tasks like targeted drug delivery and microsurgery. Current artificial microswimmers often rely on fixed locomotory gaits and manual control, limiting their ability to navigate unpredictable environments. This research addresses this challenge by employing a deep reinforcement learning (RL) approach to enable a model microswimmer to self-learn effective gait-switching strategies. The use of AI offers the potential to overcome the complexities of designing locomotory gaits for sophisticated maneuvers in the low-Reynolds-number regime characteristic of microscale environments. Previous work has utilized machine learning for navigating active particles in various conditions (flows, thermal fluctuations, obstacles), but often without accounting for the complex gait adjustments needed for adaptive locomotion. This study builds upon recent work exploring the use of machine learning techniques for enabling reconfigurable microswimmers to evolve effective gaits for self-propulsion and chemotactic response, extending these efforts to continuous action spaces for increased versatility.

The study draws upon existing research on the hydrodynamics of swimming microorganisms and the development of artificial microswimmers. Purcell's seminal work and subsequent studies have demonstrated how simple reconfigurable systems with carefully designed locomotory gaits can achieve net translation and rotation at low Reynolds numbers. However, designing gaits for more complex maneuvers or navigating perturbed environments becomes increasingly challenging. Existing microswimmers are typically designed with fixed gaits and require manual intervention for navigation. The rapid advancements in artificial intelligence and its application to locomotion problems have spurred the development of 'smart' microswimmers. Various machine learning approaches have been used to guide the navigation of active particles, but typically using simplified models that lack the complexity of adaptive gait switching. This research combines reinforcement learning with an artificial neural network to bridge this gap and enable a simple reconfigurable system to perform complex maneuvers in a low-Reynolds-number environment.

The researchers employed a deep reinforcement learning framework with an Actor-Critic neural network structure and the Proximal Policy Optimization (PPO) algorithm. The model microswimmer consists of three spheres connected by two arms with variable lengths and orientations. The hydrodynamic interactions between the spheres and the surrounding viscous fluid are modeled using the Stokes equation at low Reynolds numbers. Unlike traditional approaches where locomotory gaits are prescribed, the RL framework allows the system to self-learn gaits. The state of the system is defined by the sphere centers, arm lengths, and orientations. The observation used by the AI agent is a subset of the state information, including arm lengths, the angle between arms, and the angle difference between the target direction and the swimmer's orientation. The AI agent decides the swimmer's next action (actuation of the arms) based on the observation using the Actor neural network. The reward is determined by the swimmer's displacement along the target direction. The training process consists of multiple episodes, each with multiple learning steps, where the initial state and target direction are randomized to ensure complete exploration of the state space. The Critic neural network updates the AI to maximize the expected long-term rewards. The hydrodynamic interactions between the spheres are governed by the Stokes equations and the Oseen tensor. The kinematics of the swimmer are determined by applying force-free and torque-free conditions. The arm lengths and intermediate angle actuation rates are expressed in terms of the relative velocities of the spheres. The system is non-dimensionalized to scale lengths, velocities, time, and forces.

The deep RL framework enabled the microswimmer to learn a multimodal navigation strategy reminiscent of biological gait switching. The swimmer autonomously switches between three distinct locomotory gaits: steering, transition, and translation. Steering gaits primarily focus on re-orientation, transition gaits combine rotation and translation, and translation gaits maximize forward motion. The performance of the swimmer improved significantly with increased training episodes, achieving 100% success rates in tests of translation and rotation after sufficient training. The AI-powered swimmer successfully navigated a complex path tracing the word "SWIM", demonstrating autonomous path following without explicit programming of the gaits for each segment. The AI-powered swimmer also exhibited robustness against flow perturbations, outperforming an untrained swimmer (a Najafi-Golestanian swimmer) in the presence of a rotlet flow. Even with a strong rotlet flow, the AI-powered swimmer maintained its orientation and successfully navigated towards the target direction, while the untrained swimmer lost control. The study highlights the emergence of adaptive locomotory gaits from the RL framework, without explicit programming or prior knowledge of low-Reynolds-number locomotion. The gait-switching strategy is similar to run-and-tumble behavior observed in bacteria.

The findings demonstrate the potential of deep reinforcement learning for designing adaptive microswimmers capable of navigating complex and unpredictable environments. The AI-powered swimmer's ability to self-learn multimodal navigation strategies, its robustness to flow perturbations, and its capacity to perform complex tasks such as path tracing without explicit programming represents a significant advancement in the field of micro-robotics. This approach offers a more autonomous and versatile alternative to traditional methods that rely on detailed hydrodynamic calculations and manual interventions. The adaptive behavior of the AI-powered swimmer is analogous to the run-and-tumble behavior in bacteria, suggesting that RL can be a powerful tool for mimicking the sophisticated locomotion strategies evolved in biological systems. The successful navigation in the presence of flow demonstrates the robustness of the learned strategy and its potential for real-world applications.

This research successfully demonstrates the use of deep reinforcement learning to enable adaptive gait switching in an artificial microswimmer, resulting in robust and versatile navigation capabilities. The AI-powered swimmer's ability to learn distinct locomotory gaits for steering, transitioning, and translation, combined with its robustness to flow perturbations and capacity for complex path tracing, showcases the potential of this approach for creating next-generation micro-robots for biomedical applications. Future work could explore extending this approach to three-dimensional navigation, applying it to different microswimmer designs, incorporating flow perturbations into the training process, and investigating the effects of Brownian noise.

The current study focuses on planar motion and does not account for three-dimensional effects. The model microswimmer is a simplified representation and does not incorporate the complexities of real biological microswimmers. The effects of Brownian noise were not explicitly included in this study, although future research is planned to address this. While the swimmer demonstrates robustness to some flow perturbations, the extent of this robustness to various types and strengths of flows requires further investigation. The training process requires substantial computational resources.