Spectrum-aware Multi-hop Task Routing in Vehicle-assisted Collaborative Edge Computing

Multi-access edge computing (MEC) is a promising technology for delivering low-latency services by offloading computation to nearby edge servers (ESs). Collaborative edge computing, where multiple ESs share resources, further improves efficiency. However, most existing collaborative edge computing research focuses on one-hop offloading, limiting resource sharing due to poor channel conditions or overloaded ESs. This paper introduces a novel multi-hop MEC framework that utilizes ubiquitous vehicles in a city to create a data transportation network for multi-hop task delivery. This approach addresses the limitations of one-hop offloading by enabling tasks to be offloaded to powerful ESs even if they are multiple hops away. The dynamic nature of vehicular networks and channel variability poses challenges, such as the trade-off between communication overhead and computing capabilities and ensuring quality of service (QoS). To overcome these hurdles, this paper proposes a reinforcement learning-based task offloading approach using MADDPG to optimize task routing and server assignment, ultimately maximizing system throughput while adhering to end-to-end latency, spectrum, and computing resource constraints. Existing approaches predominantly focus on single-hop offloading, either optimizing resources for a single ES or employing cooperative MEC over multiple ESs, but still relying on the assumption of sufficient resources within one hop. These methods lack the adaptability and robustness needed to handle the dynamic and unpredictable nature of vehicular networks. Multi-hop relaying, as explored in 5G+ systems and 3GPP standards, is crucial for reliability, particularly in scenarios with limited coverage, like public safety applications. Inspired by the benefits of multi-hop D2D transmissions, this paper extends prior work that explored vehicle-assisted multi-hop transmissions in simpler scenarios, to a more general case with multiple users and multiple ESs. The use of vehicles as relays offers economic advantages over fixed infrastructure and minimizes interference through low-power, short-range transmissions. This paper addresses the complexity inherent in multi-hop offloading, particularly the joint optimization of communication overhead and computing capabilities while satisfying QoS requirements, in the face of dynamic network topology due to vehicle mobility. Traditional optimization methods are inadequate for this complex problem, motivating the use of deep reinforcement learning.

Existing research on computation offloading in MEC largely focuses on single-hop offloading from mobile devices (MDs) to ESs. This research can be categorized into two groups: resource optimization for a single ES and cooperative MEC over multiple ESs. In the single-ES scenario, studies like Cao et al. [8] addressed computation partitioning and scheduling, Poularakis et al. [9] tackled joint optimization of service placement and offloading, and Deng et al. [10] focused on throughput maximization under latency constraints. However, these works ignore inter-ES cooperation. Cooperative MEC approaches, such as Li et al. [11] and Li et al. [12], leverage collaboration among ESs for task processing and workload balancing. However, they often assume ample bandwidth in backbone/backhaul links. Prior works on multi-hop task offloading, such as Hui et al. [29] and Deng et al. [21], considered vehicle-assisted multi-hop transmissions but with limitations. Hui et al. [29] focused on minimizing relay service costs without considering resource constraints, while Deng et al. [21] addressed a simpler scenario with a single MD. This paper differentiates itself by addressing the comprehensive problem of spectrum-aware task offloading in a vehicle-assisted multi-hop MEC system with multiple MDs and ESs, explicitly considering the dynamic network topology, resource constraints, and QoS requirements.

The paper presents a multi-hop task offloading framework for a vehicle-assisted MEC network. The framework involves four steps: offloading from MD to vehicle, relaying among vehicles, uploading from vehicle to ES, and computing at the ES. The path loss between nodes is calculated using a 3GPP propagation model, and data rates are determined using the Shannon capacity theorem. The end-to-end latency comprises transmission latency, computing latency, and queueing latency. The primary optimization objective is to maximize the aggregated throughput of completed tasks while meeting end-to-end latency requirements, subject to communication and computing resource constraints. This optimization problem presents significant challenges due to the complex, non-linear relationship between latency and resource usage, and the dynamic, unpredictable nature of the vehicular network. Traditional optimization techniques are unsuitable for this problem due to the curse of dimensionality, resulting from the large state and action spaces. To address this, the problem is formulated as a Markov Decision Process (MDP). The state space encapsulates vehicle status (feasible relays and channel states), server status (computing capability and available bandwidth), and system workload (input data from MDs and queued tasks). The action space represents the routing path selection for each MD. The reward function reflects the size of successfully completed tasks within the deadlines. Instead of directly solving the MDP using traditional dynamic programming, which is computationally intractable, the paper employs the Multi-Agent Deep Deterministic Policy Gradient (MADDPG) algorithm, a model-free deep reinforcement learning (DRL) method. MADDPG is chosen because it can handle high-dimensional state and action spaces effectively and learn the network dynamics through interactions with the environment. Each MD acts as an agent, making individual decisions that contribute to the overall system objective. The MADDPG architecture consists of actor and critic networks for each agent, along with their corresponding target networks for improved stability. The critic network learns a Q-value function, estimating the value of taking an action given a state, and is trained using a global perspective, incorporating observations from all agents. The actor network determines the actions, maximizing the Q-value, and is trained locally using only the individual MD’s state. The training process involves iteratively updating the actor and critic networks using experience replay to improve the offloading policy. The selection of the destination ES implicitly defines the multi-hop route, often using the shortest path based on travel distance.

The simulation results demonstrate the effectiveness of the proposed MADDPG-based task offloading scheme compared to benchmark algorithms: single-hop offloading and a greedy multi-hop approach. The MADDPG approach consistently achieves significantly higher average throughput (the total size of completed tasks) and success rate (the ratio of completed tasks to generated tasks). This improvement is attributed to the algorithm's ability to effectively balance the workload across multiple ESs via multi-hop transmissions. The average throughput of the MADDPG algorithm increases with increasing task arrival rates, numbers of MDs and ESs, showcasing its scalability. Interestingly, the throughput remains relatively stable even with varying numbers of vehicles, highlighting the algorithm's adaptability to network dynamics. The success rate of MADDPG approaches 100% as the number of ESs increases or when task arrival rates are low, further illustrating its effectiveness. In contrast, single-hop offloading is severely limited by the resources of immediately available ESs, while the greedy multi-hop approach suffers from uncontrolled workload distribution. These results confirm that the proposed MADDPG approach dynamically balances the workloads of both communication and computing, leading to optimal resource utilization and task completion.

The findings confirm the effectiveness of the proposed multi-hop task offloading framework and the MADDPG-based solution for optimizing resource utilization in dynamic vehicular MEC networks. The superior performance of MADDPG compared to single-hop and greedy multi-hop methods directly addresses the limitations of those approaches. The multi-hop capability enables leveraging resources from ESs that would otherwise be unreachable in a one-hop scenario, which is crucial in dynamically changing network conditions. The use of MADDPG allows for adaptability to unpredictable vehicle movement and varying channel conditions, leading to improved resource allocation and task completion rates. This research contributes to the advancement of efficient and robust task offloading mechanisms for future MEC systems in vehicular environments. The successful implementation of MADDPG highlights its potential for solving complex, high-dimensional optimization problems in dynamic network settings.

This paper presents a novel multi-hop task offloading framework for vehicle-assisted MEC, addressing the limitations of one-hop approaches. A MADDPG-based algorithm effectively optimizes task routing and server assignment, maximizing throughput while adhering to latency and resource constraints. Simulation results demonstrate the significant performance improvement compared to benchmark schemes. Future research could explore more sophisticated routing algorithms, incorporating factors like energy consumption and security considerations. Investigating alternative reinforcement learning approaches and exploring decentralized control strategies would also be valuable extensions.

The current study assumes a centralized controller with global network knowledge, which might be unrealistic in large-scale deployments. The simulation environment, while comprehensive, is still a simplification of real-world conditions. The accuracy of the propagation model and traffic patterns directly impact the results. Future work should consider more realistic scenarios, exploring decentralized control and potentially incorporating uncertainty in the network dynamics and model parameters.