The increasing use of UAVs in diverse civilian applications, including communication infrastructure provision in areas lacking existing networks, necessitates efficient network topology control for UAV swarms. While using multiple UAVs to form a network offers advantages in speed and cost-effectiveness, scalability issues arise in multi-hop connectivity and UAV control, particularly when covering large areas. The challenge lies in efficiently controlling the network topology as the number of UAVs increases. Traditional approaches, such as connecting each UAV to all others within transmission range, lead to high network overhead and energy consumption. Although centralized controllers and MST-based approaches exist, they lack adaptability to dynamically changing network conditions such as user density and UAV characteristics (power consumption). This paper addresses this limitation by proposing a decentralized, RL-based solution that enables each UAV to optimize its connectivity based on real-time environmental factors and network conditions. This approach enhances adaptability and scalability, unlike traditional methods.

Existing research on UAV networks focuses on various aspects like optimal UAV positioning for tasks such as collision avoidance, flight information sharing, and surveillance. Studies utilizing deep learning for precise localization and ultra-wideband positioning systems for high accuracy and low latency are also prevalent. Furthermore, research exists on optimizing UAV formation for communication, addressing issues like energy efficiency through fuel consumption optimization and genetic algorithms for large-scale deployments. Game theory has also been applied to optimize energy efficiency. This paper builds upon this existing research by focusing on the specific problem of dynamic topology control in UAV networks. The Deep Deterministic Policy Gradient (DDPG) algorithm, known for its effectiveness in continuous action spaces, is selected for this task, with the ADAM optimizer used for parameter learning. Existing research on ADAM's efficiency and ongoing refinements in convergence speed and handling of local minima are noted.

The proposed system comprises three modules: network topology control, RL (using DDPG), and step optimization. The network topology control module uses beacon messages for real-time location updates, calculates distances between UAVs, and partitions the 3D space around each UAV into subspaces. Minimum Spanning Tree (MST) based topology is constructed for each subspace. The number of subspaces per UAV is dynamically determined by the RL module. The RL module employs the DDPG algorithm, learning to optimize the number of subspaces for each UAV to maximize energy efficiency and network throughput. The state comprises average hop count, power consumption, and degree of the network topology. The action is the number of sectors into which each UAV partitions its surrounding space. The reward function penalizes disconnections and actions exceeding a predefined boundary, while rewarding efficient topologies. The step optimization module dynamically adjusts the number of steps in the RL training based on the observed rewards. If recent average rewards remain high, the episode ends early. This accelerates learning speed and improves the system’s adaptability to changing network conditions. The ADAM optimizer is utilized for parameter learning within the DDPG algorithm. The system is evaluated through extensive simulations using Python and MATLAB, with the OpenCV library used for visualizing network topologies. Simulations consider various UAV formations (sphere, cube, pyramid) and random formations with varying numbers of UAVs (10, 20, 30, 40, 50). Performance is evaluated using metrics such as connectivity, energy consumption, and training time. Furthermore, the system is verified in the AirSim simulator, which closely mirrors real-world flight scenarios.

Simulations using typical UAV formations (sphere, cube, pyramid) showed that the proposed system successfully learns an optimal network topology, significantly reducing unnecessary links and improving connectivity compared to fully connected topologies. The system quickly converges to a near-optimal state, achieving high episodic rewards within a few episodes. Simulations with random UAV formations and varying numbers of UAVs (10-50) demonstrated the system's scalability and robustness. Even with randomly placed UAVs, the system consistently learned to form efficient, connected topologies. The analysis of randomly selected nodes within the network validated that all nodes participate effectively in the learning process, achieving similar results in terms of subspace partitioning and reward attainment. The step optimization module demonstrated significant reductions in training time (approximately 86% reduction when adjusting both step size and number). The optimized system maintains performance comparable to the baseline DDPG approach while dramatically reducing computation time. AirSim simulations further confirmed that the learned optimal topology is maintained in a dynamic real-world-like environment.

The findings demonstrate the effectiveness of the proposed RL-based approach for dynamic topology control in UAV networks. The use of DDPG enables adaptation to continuous changes in UAV positions and network conditions, addressing the limitations of previous centralized or static methods. The decentralized nature of the proposed system enhances scalability, avoiding the bottlenecks associated with centralized control. The step optimization strategy significantly improves the real-time adaptability of the system, making it suitable for environments with rapidly changing conditions. The results validate the use of RL as an effective technique for autonomous network management in dynamic UAV scenarios. The positive results in the AirSim environment suggest a strong potential for the system to function effectively in real-world UAV deployments.

This paper presents a novel RL-based topology control system for UAV networks, effectively optimizing network connectivity while minimizing energy consumption and training time. The system's adaptability and scalability are demonstrated through extensive simulations and AirSim validations. Future work will focus on exploring more complex RL algorithms, designing more sophisticated state, action, and reward schemes, and optimizing the variable step learning strategy.

The study relies on simulations and AirSim environments, which might not fully capture the complexity of real-world UAV deployments. Uncertainties such as unpredictable environmental factors (wind, interference) and potential hardware failures are not explicitly modeled. Furthermore, the performance of the proposed system under extremely high UAV densities or extreme environmental conditions remains to be fully investigated. Future research should address these aspects through real-world experiments and more comprehensive simulations.