PNNL

Abstract

Improving system-level resiliency of networked microgrids is gaining importance with the increasing integration of inverter-based resources (IBRs). This paper (1) presents resilient control design in the presence of adversarial cyber-events, and proposes a novel federated reinforcement learning (Fed-RL) approach to tackle (a) model complexities, unknown dynamical behaviors of IBR devices, (b) privacy issues regarding data sharing in multi-party-owned networked grids, and (2) transfers learned controls from simulation to test-bed, thereby helping to bridge the gap between simulation and real world. With these multi-prong objectives, first, we formulate a reinforcement learning (RL) training setup generating episodic trajectories with adversaries (attack signal) injected at the primary controllers of the grid forming (GFM) inverters where RL agents (or controllers) are being trained to mitigate the injected attacks. For networked microgrids, the horizontal Fed-RL method involving distinct independent environments are not appropriate, leading us to develop vertical variant Federated Soft Actor-Critic (FedSAC) algorithm to grasp the interconnected dynamics of networked microgrid. Next, utilizing the OpenAI Gym interface, we built a custom simulation set-up in GridLAB-D/HELICS co-simulation platform, named Resilient RL Co-simulation (ResRLCoSIM), to train the RL agents on numerical instances of a modified IEEE 123-bus benchmark test systems comprising 3 interconnected microgrids. Finally, the learned policies in the simulation world are transferred to the real-time test-bed set-up developed using high-fidelity software Hypersim to mimic real-world implementation. Experiments show that the simulator-trained RL controllers produce convincing results with the real-time test-bed set-up, validating the minimization of the sim-to-real gap.