Transactions of China Electrotechnical Society  2024, Vol. 39 Issue (5): 1436-1453    DOI: 10.19595/j.cnki.1000-6753.tces.230015
Current Issue| Next Issue| Archive| Adv Search |
Optimal Scheduling of Integrated Energy Multi-Microgrid System Based on Hierarchical Constraint Reinforcement Learning
Dong Lei1, Yang Zimin1, Qiao Ji2, Chen Sheng2, Wang Xinying2, Pu Tianjiao2
1. School of Electrical and Electronics Engineering North China Electric Power University Beijing 102206 China;
2. China Electric Power Research Institute Beijing 100192 China

Download: PDF (2182 KB)   HTML (1 KB) 
Export: BibTeX | EndNote (RIS)      
Abstract  

The optimization of the integrated energy multi-microgrid system is a complex task, with numerous variables and challenges including data privacy protection and uncertainties of power generation and load, posing significant challenges for the efficient implementation of traditional mathematical optimization methods. Recently, many scholars have turned their attention to deep reinforcement learning (DRL) methods, which rely on data-driven principles and exhibit strong adaptability to uncertainties of power generation and load. Nevertheless, the difficulty of convergence persists with increasing system scale, and traditional DRL methods that handle constraints by adding penalty terms to the reward function may obscure the boundary between objectives and constraints, making it difficult to ensure that constraints are fully satisfied and resulting in excessively conservative learning strategies or suboptimal solutions. To address these issues, this paper proposed a hierarchical constraint reinforcement learning optimization method.
Firstly, this paper proposed a hierarchical DRL optimization framework for multi-microgrid systems. The proposed framework divides the optimization problem into two layers: an upper layer and a lower layer. The upper layer does not require obtaining all the operating status information of each microgrid. Instead, it utilizes net load prediction information and energy storage state information to provide energy storage optimization strategies and power interaction strategies. On the other hand, the lower layer enables each microgrid to autonomously optimize the output of its internal devices based on its own status information through mathematical programming, with the upper layer strategy as a constraint. The proposed framework leverages cooperation between the upper and lower layers to achieve overall optimization of the multi-microgrid system. This framework fully utilizes the advantages of DRL based on data-driven principles and effectively considers the solution accuracy of mathematical programming. Based on this hierarchical framework, a constraint DRL method is proposed that combines DRL methods with Lagrange multiplier methods. This method transforms the constraint optimization problem into an unconstrained optimization problem, enabling the agent to find the optimal strategy while strictly satisfying the constraints. Compared to traditional centralized optimization methods, the proposed method dynamically responds to the fluctuations of power generation and load to meet online optimization requirements and protects microgrid data privacy by not requiring the aggregation of all microgrid status information. Compared to general DRL methods, our approach effectively solves the problem of constraint violation and significantly improves both the convergence speed and accuracy.
The following conclusions can be drawn from the case studies: (1) A hierarchical design approach is proposed to simplify the optimization of multi-microgrid systems. The approach does not require information exchange between microgrids and only necessitates uploading net load and energy storage state information. Microgrids can independently and parallelly solve the optimization problem based on their own status information. This approach can provide scheduling results in real-time consistent with the optimal solution when local status information is available. (2) The proposed approach combines data-driven principles with traditional methods, simplifying the complexity of action space and reward design. It effectively balances the rapid solving ability of DRL and the solution accuracy of mathematical programming. Compared to traditional DRL methods, the proposed approach significantly improves both convergence speed and accuracy. (3) The approach combines DRL methods with Lagrange multiplier methods to transform the constrained optimization problem into an unconstrained one. This ensures that the agent can find the optimal strategy while strictly satisfying the constraints. The approach avoids convergence difficulties and constraint violation issues caused by manually setting the penalty coefficient in traditional DRL methods. (4) The model exhibits robustness and can effectively adapt to the fluctuations of power generation and load, making rapid decisions on power interactions of each microgrid.

Key wordsMulti-microgrid      hierarchical constraint reinforcement learning      uncertainty      data privacy protection     
Received: 06 January 2023     
PACS: TM73  
Service
E-mail this article
Add to my bookshelf
Add to citation manager
E-mail Alert
RSS
Articles by authors
Dong Lei
Yang Zimin
Qiao Ji
Chen Sheng
Wang Xinying
Pu Tianjiao
Cite this article:   
Dong Lei,Yang Zimin,Qiao Ji等. Optimal Scheduling of Integrated Energy Multi-Microgrid System Based on Hierarchical Constraint Reinforcement Learning[J]. Transactions of China Electrotechnical Society, 2024, 39(5): 1436-1453.
URL:  
https://dgjsxb.ces-transaction.com/EN/10.19595/j.cnki.1000-6753.tces.230015     OR     https://dgjsxb.ces-transaction.com/EN/Y2024/V39/I5/1436
Copyright © Transactions of China Electrotechnical Society
Supported by: Beijing Magtech