Skip to main content

PNNL

  • About
  • News & Media
  • Careers
  • Events
  • Research
    • Scientific Discovery
      • Biology
        • Chemical Biology
        • Computational Biology
        • Ecosystem Science
        • Human Health
          • Cancer Biology
          • Exposure Science & Pathogen Biology
          • Metabolic Inflammatory Diseases
        • Integrative Omics
          • Advanced Metabolomics
          • Chemical Biology
          • Mass Spectrometry-Based Measurement Technologies
          • Spatial and Single-Cell Proteomics
          • Structural Biology
        • Microbiome Science
          • Biofuels & Bioproducts
          • Human Microbiome
          • Soil Microbiome
          • Synthetic Biology
      • Chemistry
        • Computational Chemistry
        • Chemical Separations
        • Chemical Physics
        • Catalysis
      • Earth & Coastal Sciences
        • Atmospheric Science
          • Atmospheric Aerosols
          • Human-Earth System Interactions
          • Modeling Earth Systems
        • Coastal Science
        • Ecosystem Science
        • Plant Science
        • Subsurface Science
        • Terrestrial Aquatics
      • Materials Sciences
        • Materials in Extreme Environments
        • Precision Materials by Design
        • Science of Interfaces
        • Solid Phase Processing
          • Cold Spray
          • Friction Stir Welding & Processing
          • ShAPE
      • Nuclear & Particle Physics
        • Dark Matter
        • Flavor Physics
        • Fusion Energy Science
        • Neutrino Physics
      • Quantum Information Sciences
    • Sustainable Energy
      • Electric Grid Modernization
        • Emergency Response
        • Grid Analytics
          • AGM Program
          • Tools and Capabilities
        • Grid Architecture
        • Grid Cybersecurity
        • Grid Energy Storage
        • Grid Resilience and Decarbonization
          • Earth System Modeling
          • Energy System Modeling
        • Transmission
        • Distribution
      • Energy Efficiency
        • Appliance and Equipment Standards
        • Building Energy Codes
        • Building Technologies
          • Advanced Building Controls
          • Advanced Lighting
          • Building-Grid Integration
        • Building and Grid Modeling
        • Commercial Buildings
        • Federal Buildings
          • Federal Performance Optimization
          • Resilience and Security
        • Grid Resilience and Decarbonization
        • Residential Buildings
          • Building America Solution Center
          • Energy Efficient Technology Integration
          • Home Energy Score
        • Energy Efficient Technology Integration
      • Energy Storage
        • Electrochemical Energy Storage
        • Flexible Loads and Generation
        • Grid Integration, Controls, and Architecture
        • Regulation, Policy, and Valuation
        • Science Supporting Energy Storage
        • Chemical Energy Storage
      • Environmental Management
        • Waste Processing
        • Radiation Measurement
        • Environmental Remediation
      • Fossil Energy
        • Subsurface Energy Systems
        • Carbon Management
          • Carbon Capture
          • Carbon Storage
          • Carbon Utilization
        • Advanced Hydrocarbon Conversion
      • Nuclear Energy
        • Fuel Cycle Research
        • Advanced Reactors
        • Reactor Operations
        • Reactor Licensing
      • Renewable Energy
        • Solar Energy
        • Wind Energy
          • Wind Resource Characterization
          • Wildlife and Wind
          • Wind Systems Integration
          • Wind Data Management
          • Distributed Wind
        • Energy Equity & Health
        • Marine Energy
          • Environmental Monitoring for Marine Energy
          • Marine Biofouling and Corrosion
          • Marine Energy Resource Characterization
          • Testing for Marine Energy
          • The Blue Economy
        • Hydropower
          • Environmental Performance of Hydropower
          • Hydropower Cybersecurity and Digitalization
          • Hydropower and the Electric Grid
          • Materials Science for Hydropower
          • Pumped Storage Hydropower
          • Water + Hydropower Planning
        • Grid Integration of Renewable Energy
        • Geothermal Energy
      • Transportation
        • Bioenergy Technologies
          • Algal Biofuels
          • Aviation Biofuels
          • Waste-to-Energy and Products
        • Hydrogen & Fuel Cells
        • Vehicle Technologies
          • Emission Control
          • Energy-Efficient Mobility Systems
          • Lightweight Materials
          • Vehicle Electrification
          • Vehicle Grid Integration
    • National Security
      • Cybersecurity
        • Discovery and Insight
        • Proactive Defense
        • Trusted Systems
      • Nuclear Material Science
      • Nuclear Nonproliferation
        • Radiological & Nuclear Detection
        • Nuclear Forensics
        • Ultra-Sensitive Nuclear Measurements
        • Nuclear Explosion Monitoring
        • Global Nuclear & Radiological Security
      • Stakeholder Engagement
        • Disaster Recovery
        • Global Collaborations
        • Legislative and Regulatory Analysis
        • Technical Training
      • Systems Integration & Deployment
        • Additive Manufacturing
        • Deployed Technologies
        • Rapid Prototyping
        • Systems Engineering
      • Threat Analysis
        • Advanced Wireless Security
          • 5G Security
          • RF Signal Detection & Exploitation
        • Climate Security
        • Grid Resilience and Decarbonization
        • Internet of Things
        • Maritime Security
      • Chemical & Biothreat Signatures
        • Contraband Detection
        • Pathogen Science & Detection
        • Explosives Detection
        • Threat-Agnostic Biodefense
    • Data Science & Computing
      • Artificial Intelligence
      • Graph and Data Analytics
      • Software Engineering
      • Computational Mathematics & Statistics
      • High-Performance Computing
      • Visual Analytics
    • Lab Objectives
    • Publications & Reports
    • Featured Research
  • People
    • Inventors
    • Diversity & Inclusion
    • Lab Leadership
    • Lab Fellows
    • Staff Accomplishments
  • Partner with PNNL
    • Academia
      • Distinguished Graduate Research Programs
      • Internships
      • Visiting Faculty Program
      • Joint Appointments
      • Joint Institutes
      • Linus Pauling Distinguished Postdoctoral Fellowship
      • Minority-serving Institutions
    • Community
      • Regional Impact
      • STEM Education
        • Resources
        • Student STEM Ambassadors
        • STEM Ambassadors in the Classroom
      • Philanthropy
      • Volunteering
    • Industry
      • Available Technologies
      • Industry
      • Industry Partnerships
      • Licensing & Technology Transfer
      • Entrepreneurial Leave
  • Facilities & Centers
    • All Facilities
      • Atmospheric Radiation Measurement User Facility
      • Electricity Infrastructure Operations Center
      • Energy Sciences Center
      • Environmental Molecular Sciences Laboratory
      • Grid Storage Launchpad
      • Institute for Integrated Catalysis
      • Interdiction Technology and Integration Laboratory
      • PNNL Seattle Research Center
      • PNNL-Sequim (Marine and Coastal Research)
      • Radiochemical Processing Laboratory
      • Shallow Underground Laboratory

Physics-Informed Machine Learning for Energy and Environment

  • Extreme Weather & Power Outages
  • Control-Oriented Modeling for Buildings
  • Machine Learning for Fuel Cell Modeling
  • Optimal Control by Transfer Learning
  • Reinforcement Learning for Grid Control
  • Surrogate Modeling for Thermal Dynamics
  • Subsurface Reduced-Order Modeling
  • News
  • Contacts

Reinforcement Learning for Grid Control

Grid control

Team

Xinya Li, leader; Yunzhi Huang, Xiaoyuan Fan, Malachi Schram, Qiuhua Huang 

Problem

In environments that are only partially observable, with control tasks occurring over a long-time horizon in highly dimensional states and action spaces, deep reinforcement learning (DRL) with agents have been shown to achieve promising results. In energy system domains, several DRL core challenges remain to be explored: scalability, learning from limited samples, and high-dimensional continuous state and action spaces. In this task, we focus on the scalability challenge in coordination with ongoing projects investigating sample efficiency and state space requirements for controlling energy transmission on the electric grid. 

Approach

The dataset for the study is generated for the purpose of reinforcement learning for grid control. In terms of scalability and sampling efficiency, our team will apply an augmented random search (ARS) agent on a generator bus of a two-bus system environment hosting the swing-equation simulation tool. The selected generator’s rotor angle and speed will remain as the observation states, which are the DRL agent’s neural network inputs. Physics-based information from the normal generator operation state formula will be provided to the reinforcement learning (RL) agent’s neural network loss function during training. Faulty conditions will also be considered.  

Results

A performance comparison of the physics-informed RL agent and traditional RL agent will be completed to identify improvements in training speed, reward convergence, sampling efficiency, scalability, and transferability.  

Physics-informed reinforcement learning for Grid Control

Code, Data, and Related Links

In progress.

Publications

In progress.

PNNL

  • Get in Touch
    • Contact
    • Careers
    • Doing Business
    • Environmental Reports
    • Security & Privacy
    • Vulnerability Disclosure Program
  • Research
    • Scientific Discovery
    • Sustainable Energy
    • National Security
Subscribe to PNNL News
Department of Energy Logo Battelle Logo
Pacific Northwest National Laboratory (PNNL) is managed and operated by Battelle for the Department of Energy
  • YouTube
  • Facebook
  • Twitter
  • Instagram
  • LinkedIn