Real-time outage management in active distribution networks using reinforcement learning over graphs

The increasing unreliability of power distribution networks (DNs), especially during extreme weather events or cyber-physical attacks, has highlighted the need for self-healing capabilities. Traditional outage mitigation methods, such as reconfiguration through switching control and emergency load shedding, are often too slow and computationally inefficient for smart grids. This paper addresses this challenge by proposing a novel approach that leverages the power of graph reinforcement learning (GRL) to achieve real-time, near-optimal outage management. The shift towards smart grids, characterized by the integration of distributed energy resources (DERs) and advanced automation technologies, necessitates rapid and autonomous response mechanisms during outages. DNs are no longer viewed merely as passive links but as active entities capable of independent operation, even when disconnected from the main grid. This paradigm shift demands intelligent control systems that can swiftly detect, isolate, and restore power, minimizing disruptions and maximizing energy supply. The core challenge lies in the combinatorial complexity of network reconfiguration and the inherent interdependencies between various network variables. Conventional optimization techniques often struggle to provide timely solutions in such dynamic and high-dimensional environments. This research aims to demonstrate the effectiveness of GRL in addressing these challenges and enhancing the resilience of active distribution networks.

Existing literature on outage management in distribution networks explores various techniques, including heuristic, meta-heuristic, and mixed-integer programming (MIP) approaches. Heuristic and meta-heuristic methods, while offering solutions, often suffer from computational expense and scalability issues, especially for large-scale networks. MIP-based techniques, such as mixed-integer nonlinear programming (MINLP), face similar limitations concerning real-time applicability. Linear programming approximations are often inadequate for handling the complexities of three-phase unbalanced DNs with various DER types. While reinforcement learning (RL) has emerged as a promising tool for power system control, its application to outage management in DNs is still in its early stages. Previous RL-based studies primarily focus on dynamic distribution network reconfiguration (DNR) under normal operating conditions, often lacking the real-time capabilities and adaptability needed during emergencies. Many existing RL approaches for DNR rely on deep Q-learning or similar methods which suffer from limitations in terms of scalability. This paper builds upon the existing literature by proposing a novel GRL approach, tailored specifically for real-time outage management, which addresses the limitations of existing methods and provides more accurate, effective solution with enhanced scalability for larger networks. This work also considers both grid-connected and islanding reconfiguration strategies, a significant improvement over methods limited to one type of reconfiguration.

The proposed methodology represents the distribution network (DN) as a graph, where nodes represent buses and edges represent lines or transformers. State variables, such as demand/generation estimates and voltage/current measurements, are superimposed on this graph. The complex interdependencies between these variables are captured using a Graph Capsule (GCAPS) neural network, a type of graph neural network (GNN). The GCAPS network is chosen for its ability to effectively capture the structural information of the graph and provide enhanced state representation compared to simpler architectures like Multi-Layer Perceptrons (MLPs). The outage management problem is formulated as a Markov Decision Process (MDP) over graphs, with the state comprising node and edge variables, network topology, and outage information. The action space includes switching actions (opening/closing switches) and load shedding actions. The transition probability is learned during training, and the reward function is designed to maximize energy supplied while minimizing voltage violations. Proximal Policy Optimization (PPO), a policy gradient method, is employed to train the GCAPS-based policy network. The environment is simulated using OpenDSS, a power system simulator, with a Python-based API to interface with the GRL agent. The proposed approach involves a three-step process: 1) Representing the DN as a graph; 2) Formulating outage management as a Markov Decision Process (MDP) in the graph domain; and 3) Employing a GCAPS-based GRL agent to learn optimal control policies. The GCAPS network efficiently captures the network topology and state variables, allowing for effective learning of complex relationships. The training process involves generating diverse outage scenarios through randomized edge removal in the graph representation, varying load and generation points, and incorporating different failure scenarios. PPO is used to train the model, optimizing the policy network to maximize the cumulative reward. This method allows the model to learn optimal strategies for reconfiguration and load shedding in various outage conditions.

The proposed GCAPS-based GRL model demonstrates superior performance compared to several baseline methods across three test networks (13-bus, 34-bus, and 123-bus modified IEEE networks). The key findings are: 1. **Near-optimal, real-time performance:** The GCAPS model achieves near-optimal solutions for outage management in real-time (millisecond-scale response times), significantly outperforming traditional optimization methods (MISOCP and BPSO) which require seconds or even minutes for larger networks. The speed is crucial for preventing cascading failures. 2. **Enhanced resilience:** The model demonstrates a substantial improvement in network resilience, measured by the reduction in loss of energy. For example, in the 13-bus and 34-bus networks, the reduction in energy loss when compared to MISOCP is 607.45 kWs and 596.52 kWs respectively. 3. **Generalizability:** The model generalizes well across a wide range of outage scenarios and network sizes, showcasing its adaptability to unforeseen events. The ability to respond effectively to diverse and unexpected outage situations is demonstrated through the multiple scenarios considered in each of the test networks. 4. **Effective topology integration:** The utilization of GCAPS, unlike MLP, effectively integrates the network's topological information into the decision-making process, leading to superior performance and improved adherence to operational constraints. The comparison against an MLP-based RL agent highlights the value of this graph-based approach. 5. **Scalability:** The GCAPS model scales well to larger networks, indicating its suitability for real-world applications. The tests conducted on the 123-bus network confirm the scalability of the approach and show that near optimal performance could still be maintained. The experimental results consistently show that the GCAPS model outperforms MLP and approaches the performance of the significantly slower optimization-based methods (MISOCP and BPSO). The training convergence plots show that GCAPS achieves higher rewards and faster convergence than MLP. The detailed comparison of switching and load shedding actions for various scenarios across different networks further underscores the superior performance and generalizability of the GCAPS model.

The results demonstrate the effectiveness of the proposed GRL model for real-time outage management in active distribution networks. The model's real-time performance, coupled with its ability to handle diverse outage scenarios and network sizes, positions it as a viable self-healing tool for smart grids. The significant improvement in resilience compared to traditional optimization techniques underlines the benefits of integrating graph-based learning into the outage management process. The superior performance of GCAPS over MLP highlights the importance of explicitly incorporating topological information into the learning framework. The model's ability to handle both grid-connected and islanded reconfiguration schemes further enhances its versatility. The findings of this study have significant implications for the design and implementation of resilient smart grids. The ability to quickly and effectively manage outages is crucial for ensuring the reliability and stability of the power system, especially in the face of increasingly frequent and severe extreme weather events. This work contributes to the growing body of research on the application of RL and GNNs to power system control, advancing the development of autonomous and adaptive grid management systems.

This paper presents a novel GRL-based model for real-time outage management in active distribution networks. The model uses a GCAPS network to learn optimal control policies, achieving near-optimal performance on various network sizes and outage scenarios. The model's real-time capability and generalizability make it a promising solution for enhancing grid resilience. Future research could explore the model's performance under more complex scenarios, such as considering the dynamics of DERs, communication network limitations, and cascading failures. Investigating advanced GNN architectures or integrating other machine learning techniques could further enhance the model's capabilities.

While the proposed model shows significant promise, some limitations exist. The training process requires considerable computational resources, which could pose a challenge for smaller organizations or regions. The model's performance relies on the accuracy of the power flow simulations and state variable estimations, and inaccuracies here could affect decision-making. The study also uses idealized models of DERs and assumes perfect communication, which might not always hold in real-world scenarios. Future work should address these aspects by considering more realistic models for DERs and communication networks.