基于安全强化学习和多能源惯性协调的综合能源系统优化调度

doi:10.19595/j.cnki.1000-6753.tces.251725

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (14993 KB)
输出: BibTeX | EndNote (RIS)

摘要

针对综合能源系统中多能源惯性特性迥异、多时间尺度协调复杂导致系统难以实现最优安全经济运行的问题,该文提出一种基于事件触发-安全强化学习的综合能源系统多能源惯性多时间尺度协调优化调度方法。首先,分析电、热、气子系统不同能源惯性的对应协调关系,建立考虑多能源设备参与电网惯性支撑的多能源惯性协调模型。然后,分析电、热、气网络传输时间尺度差异性,建立基于双层安全强化学习的综合能源系统多能源惯性协调优化调度模型,并提出考虑事件触发的安全强化学习求解算法。最后,基于修改的IEEE 30节点电力子系统、20节点天然气子系统和14节点热力子系统进行仿真验证。结果表明,该文提出的多能源惯性多时间尺度协调优化调度方法能够在满足综合能源系统安全约束的前提下保持较好的经济性。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	孙鹏
	杨茂
	滕云
	陈哲
	王令

关键词 ：综合能源系统, 多能源惯性, 多时间尺度, 强化学习, 优化调度

Abstract：

The integrated energy system (IES), which synergistically couples electricity, heat, and natural gas, is pivotal for achieving high-quality energy development and carbon neutrality goals. However, the divergent inertial characteristics and multi-time-scale dynamics of these energy subsystems pose significant challenges to secure and economically optimal operation. The declining inertia in power grids, a consequence of high penetration of power-electronic-interfaced renewables, undermines system frequency stability and anti-disturbance capability. Concurrently, the disparate response times-with electricity operating on a second/minute scale and heat/gas on a minute/hour scale-complicate coordinated scheduling. Existing studies often have limitations: some overlook the full potential of multi-energy coupling equipment in providing grid inertia support, others fail to adequately model the impact of device participation on source-network energy balance, and many cannot effectively resolve the security-economic trade-off in high-dimensional decision-making across multiple time scales. To holistically address these intertwined challenges, this paper proposes a novel multi-time-scale coordinated optimization scheduling method for IES based on event-triggered safe reinforcement learning (SRL), designed to coordinate multi-energy inertia for enhanced security and economy.
The proposed methodology is structured around three key innovations. First, a comprehensive "equipment-inertia-energy" model is established to characterize how multi-energy coupling devices participate in grid inertia support. Unlike existing models that simplify devices as fixed inertia sources or focus solely on energy balance, this model meticulously analyzes the participation degree of devices like CHP units, power-to-gas/hydrogen, fuel cells, and thermal storage, and maps their operational states to equivalent grid inertia constants, considering the energy buffering provided by thermal and gas networks. Second, a multi-time-scale energy coordination strategy is developed that leverages the inherent tolerance of thermal and gas systems to short-term power fluctuations. This strategy explicitly treats the power grid's second/minute-level inertia regulation demands as an energy buffer that can be absorbed and managed within the longer minute/hour-level dispatch cycles of the thermal and gas networks, eliminating the need for highly precise short-term predictions. Third, a bi-level SRL optimization framework with an event-triggering mechanism is constructed to solve the model efficiently. The upper-level SRL optimizes short-term grid economic operation and inertia security, while the lower-level SRL minimizes long-term thermal and gas subsystem operational costs. Crucially, the event-triggering mechanism, based on a Lyapunov safety function, activates a computationally intensive safety recovery strategy only when the system state approaches or violates security boundaries; otherwise, a more efficient evolutionary policy is executed, thus balancing computational load and safety assurance.
Simulation studies were conducted on a modified IES testbed comprising an IEEE 30-bus power system, a 20-node gas system, and a 14-node thermal system. Five different schemes were compared to validate the proposed method's effectiveness. The results demonstrate that the proposed method (Scheme 5) effectively maintains grid inertia within a high-inertia security zone across various time periods, significantly reducing the probability and associated cost of frequency and Rate-of-Change-of-Frequency (RoCoF) limit violations. Specifically, the frequency/RoCoF violation cost was reduced to nearly zero, and the thermal/gas network violation cost was cut by over 80% compared to the sub-optimal scheme. Furthermore, the multi-time-scale coordination reduced total multi-energy operational costs, with the comprehensive cost in Scheme 5 being at least 12.7% lower than schemes without full coordination. A comparative analysis with other reinforcement learning algorithms, including PPO, SAC, and CPO, showed that the proposed event-triggered bi-level SRL achieved higher and more stable reward values and a significantly lower cost function, indicating superior security constraint handling. Sensitivity analysis on the conversion gain coefficient confirmed its critical role in frequency stability, while computational efficiency tests proved that the event-triggering mechanism reduced the average cycle computation time by a factor of 3.5, making it highly suitable for the online dispatch of complex IES.

Key words： Integrated energy system multi-energy inertia multiple time scales reinforcement learning optimal scheduling

收稿日期: 2025-10-09

PACS:

TM73

基金资助:

国家重点研发计划项目（2017YFB0902100）; 国家重点研发计划项目(2022YFB2403000); 东北电力大学博士科研启动基金资助（BSJXM-2024205）

通讯作者: 杨茂男,1982年生,教授,博导,研究方向为新能源发电功率预测,微网优化调度,综合能源系统优化。E-mail：yangmao820@163.com

作者简介: 孙鹏男,1994年生,博士,讲师,硕士生导师,研究方向为综合能源系统优化调度与运行控制,多能源惯性评估与优化。E-mail：sunpeng@neepu.edu.cn

引用本文:

孙鹏, 杨茂, 滕云, 陈哲, 王令. 基于安全强化学习和多能源惯性协调的综合能源系统优化调度[J]. 电工技术学报, 0, (): 20251725-. Sun Peng, Yang Mao, Teng Yun, Chen Zhe, Wang Ling. Optimal Scheduling of Integrated Energy System Based on Safety Reinforcement Learning and Multi-Energy Inertia Coordination. Transactions of China Electrotechnical Society, 0, (): 20251725-.

链接本文:

https://dgjsxb.ces-transaction.com/CN/10.19595/j.cnki.1000-6753.tces.251725 https://dgjsxb.ces-transaction.com/CN/Y0/V/I/20251725

[1] 张苏涵,顾伟,俞睿智,等.综合能源系统建模与仿真:综述、思考与展望[J].电力系统自动化,2024,48(17):1-21.
ZHANG Suhan, GU Wei, YU Ruizhi, et al.Modeling and simulation of integrated energy system: review, reflection and prospects[J]. Automation of Electric Power Systems, 2024, 48(17): 1-21.
[2] 刘硕,滕云,陈哲.融合减碳型多能源微网的城市能源系统环境-经济协调优化模型[J/OL].电工技术学报,1-24[2025-10-09].https://doi.org/10.19595/ j.cnki.1000-6753.tces.242075.
Liu Shuo, Teng Yun, Chen Zhe.An Environment-Economic Coordination Optimization Model for Urban Energy Systems Integrating Carbon-Reducing Multi-Energy Microgrids[J]. Transactions of China Electrotechnical Society. 1-24[2025-10-09].
[3] S. Wang, W. Wu, C. Lin, et al.A Dynamic Equivalent Energy Storage Model of Natural Gas Networks for Joint Optimal Dispatch of Electricity-Gas Systems, IEEE Transactions on Sustainable Energy, vol. 15, no. 1, pp. 621-632, Jan. 2024.
[4] N. Yanget al., "Two-Stage Coordinated Robust Planning of Multi-Energy Ship Microgrids Considering Thermal Inertia and Ship Navigation," in IEEE Transactions on Smart Grid, vol. 16, no. 2, pp. 1100-1111, March 2025.
[5] Z. Houet al. Privacy-Preserved Aggregate Thermal Dynamic Model of Buildings. IEEE Transactions on Smart Grid, vol. 15, no. 6, pp. 5653-5664, Nov. 2024.
[6] Xu Li, Jianhua Deng, Jichun Liu, A two-layer and three-stage dynamic demand response game model considering the out of sync response for gases generators. Renewable Energy, Volume 228, 2024, 120681.
[7] Weijia Sun, Qi Wang, Yujian Ye, Yi Tang, Unified modelling of gas and thermal inertia for integrated energy system and its application to multitype reserve procurement. Applied Energy, Vol. 305, 2022, 117963.
[8] Qirun Sun, Zhi Wu, Wei Gu, et al.Resilience Assessment for Integrated Energy System Considering Gas-Thermal Inertia and System Interdependency. IEEE Transactions on Smart Grid, vol. 15, no. 2, pp. 1509-1524, March 2024.
[9] 陶然,赵冬梅,徐辰宇,等.考虑电-气-热-交通相互依存的城市能源系统韧性评估与提升方法[J]. 电工技术学报,2023,38(22):6133-6149.
TAO Ran, ZHAO Dongmei, XU Chenyu, et al.Resilience assessment and enhancement methods for urban energy system considering electricity-gas-heat-transport interdependency[J]. Transactions of China Electrotechnical Society, 2023, 38(22): 6133-6149.
[10] 张帅,刘文霞,张艺伟,等.计及多重热惯性特征的区域综合能源系统可靠性评估[J].电工技术学报,2023,38(12):3289-3305.
ZHANG Shuai, LIU Wenxia, ZHANG Yiwei, et al.Reliability evaluation of regional integrated energy system with multiple thermal inertia characteristics[J]. Transactions of China Electrotechnical Society, 2023, 38(12): 3289-3305
[11] 孙鹏,滕云,陈哲,等.考虑异质能流输运特性的多能源系统惯量极限优化[J].中国电机工程学报,2022,42(10):3642-3656.
SUN Peng, TENG Yun, HUI Qian, CHEN Zhe.Inertia Limit Optimization of Multi-energy System Considering the Transport Characteristics of Heterogeneous Energy Flow[J]. Proceedings of the CSEE, 2022, 42(10): 3642-3656.
[12] 项颂,苏鹏,吴坚,等.基于多源储能协同的交直流送端系统惯量优化控制模型[J].中国电力,2023,56(04):68-76.
XIANG Song, SU Peng, WU Jian, Inertia optimization control model of AC/DC transmission side system based on multi-source energy storage coordination[J]. Electric Power. 2023, 56(04): 68-76.
[13] 张怡静,李智,时艳强,等.基于储能惯量支撑的受端电网频率优化控制方法[J].电工技术学报,2024,39(11):3556-3568.
Yijing Zhang, Xiwen Cui, Yanqiang Shi, Optimal frequency control method of receiving power grid based on energy storage inertia support[J]. Transactions of china electrotechnical society. 2024, 39(11): 3556-3568.
[14] 缪蔡然,朱姚培,王琦,等.考虑气热惯性的综合能源系统参与辅助服务策略[J].电力建设,2023,44(11):128-137.
MIAO Cairan, ZHU Yaopei, WANG Qi.Ancillary Service Strategies for Integrated Energy Systems Considering Gas-Thermal Inertia[J]. Electric Power Construction. 2023, 44(11): 128-137.
[15] 王欣,崔承刚,王想想,等.基于安全强化学习的热电联产机组经济调度策略研究[J].系统仿真学报,2025,37(04):968-981.
Wang Xin, Cui Chenggang, Wang Xiangxiang.Research on Economic Dispatching Strategy of CHP Units Based on Safe Reinforcement Learning[J]. Journal of System Simulation, 2025, 37(04): 968-981.
[16] 焦昊,殷岩岩,吴晨,等.基于安全强化学习的主动配电网有功-无功协调优化调度[J].中国电力,2024,57(03):43-50.
JIAO Hao, YIN Yanyan, WU Chen.Coordinated Optimization of Active and Reactive Power of Active Distribution Network Based on Safety Reinforcement Learning. Electric Power, 2024, 57(03): 43-50.
[17] G. Hao, Y. Li, Y. Li, et al.Lyapunov-Based Safe Reinforcement Learning for Microgrid Energy Management. IEEE Transactions on Neural Networks and Learning Systems, vol. 36, no. 6, pp. 9985-9999, June 2025.
[18] SUN Peng, TENG Yun, CHEN Zhe.Robust coordinated optimization for multi-energy systems based on multiple thermal inertia numerical simulation and uncertainty analysis[J]. Applied Energy, 2021, 296: 116982.
[19] Qinran Hu, Rushuai Han, Xiangjun Quan, et al.Grid-Forming Inverter Enabled Virtual Power Plants With Inertia Support Capability. IEEE Transactions on Smart Grid, vol. 13, no. 5, pp. 4134-4143, Sept. 2022.
[20] 张祥宇,朱永健,付媛.基于系统惯量需求的风储协同快速频率响应技术[J].中国电机工程学报,2023,43(14):5415-5429.
ZHANG Xiangyu, ZHU Yongjian, FU Yuan.Wind-storage Cooperative Fast Frequency Response Technology Based on System Inertia Demand. Proceedings of the CSEE. 2023, 43(14): 5415-5429.
[21] 袁铁江,张江飞,滕越.基于虚拟同步机的新能源制氢系统协调控制策略[J/OL].中国电机工程学报,2025, 45(01): 163-175.
YUAN Tiejiang, ZHANG Jiangfei, TENG Yue.Coordinated Control Strategy of the Renewable Energy Hydrogen Production System Based on VSG[J]. Proceedings of the CSEE, 2025, 45(01): 163-175.
[22] 张苏涵,顾伟,陆帅,等.综合能源系统分析——从数值到解析(三):混合时间尺度经济调度[J/OL].中国电机工程学报,1-13[2025-11-21].https://link.cnki. net/urlid/11.2107.tm.20241202.1409.006.
ZHANG Suhan, GU Wei, LU Shuai, et al.Integrated Energy System Analysis——From Numerical to Analytical (I): Analytical Method in Natural Gas System[J]. Proceedings of the CSEE, 1-13[2025-11-21].
[23] J. Hu, Z. Yan, X. Xu and S. Chen, Inertia Market: Mechanism Design and Its Impact on Generation Mix, Journal of Modern Power Systems and Clean Energy, vol. 11, no. 3, pp. 744-756, May 2023.
[24] 作者. 附录-基于安全强化学习和多能源惯性协调的综合能源系统优化调度[CP/OL]. https://www.kdocs.cn/l/cfqvIHrvuS53
[25] 张薇,王浚宇,杨茂,等.基于分布式双层强化学习的区域综合能源系统多时间尺度优化调度[J].电工技术学报,2025,40(11):3529-3544.
Zhang Wei, Wang Junyu, Yang Mao.The Multi-Time-Scale Optimal Scheduling for Regional Integrated Energy System Based on the Distributed Bi-Layer Reinforcement[J]. Transactions of China Electrotechnical Society, 2025, 40(11): 3529-3544.
[26] 陈明昊,朱月瑶,孙毅,等.计及高渗透率光伏消纳与深度强化学习的综合能源系统预测调控[J].电工技术学报,2024,39(19):6054-6071+6103.
CHEN Minghao, ZHU Yueyao, SUN Yi, et al.The predictive-control optimization method for park integrated energy system considering the high penetration of photovoltaics and deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2024, 39(19): 6054-6071, 6103.
[27] Ye Y J, Wang H, Chen P, et al.Safe deep reinforcement learning for microgrid energy management in distribution networks with leveraged spatial-temporal perception[J]. IEEE Trans on Smart Grid, 2023, 14(5): 3759-3775.