Optimal Scheduling of Integrated Energy System Based on Safety Reinforcement Learning and Multi-Energy Inertia Coordination
Sun Peng1, Yang Mao1, Teng Yun2, Chen Zhe3, Wang Ling4
1. Key Laboratory of Modern Power System Simulation and Control & Renewable Energy Technology of Ministry of Education Northeast Electric Power University Jilin 132012 China; 2. School of Electrical Engineering Shenyang University of Technology Shenyang 110870 China; 3. Department of Energy Technology Aalborg University Aalborg DK-9220 Denmark; 4. Kezhou Power Supply Company of Xinjiang Electric Power Co. Kizilsu Kirgiz Autonomous 845350 China
Abstract：The integrated energy system (IES), which synergistically couples electricity, heat, and natural gas, is pivotal for achieving high-quality energy development and carbon neutrality goals. However, the divergent inertial characteristics and multi-time-scale dynamics of these energy subsystems pose significant challenges to secure and economically optimal operation. The declining inertia in power grids, a consequence of high penetration of power-electronic-interfaced renewables, undermines system frequency stability and anti-disturbance capability. Concurrently, the disparate response times-with electricity operating on a second/minute scale and heat/gas on a minute/hour scale-complicate coordinated scheduling. Existing studies often have limitations: some overlook the full potential of multi-energy coupling equipment in providing grid inertia support, others fail to adequately model the impact of device participation on source-network energy balance, and many cannot effectively resolve the security-economic trade-off in high-dimensional decision-making across multiple time scales. To holistically address these intertwined challenges, this paper proposes a novel multi-time-scale coordinated optimization scheduling method for IES based on event-triggered safe reinforcement learning (SRL), designed to coordinate multi-energy inertia for enhanced security and economy. The proposed methodology is structured around three key innovations. First, a comprehensive “equipment-inertia-energy” model is established to characterize how multi-energy coupling devices participate in grid inertia support. Unlike existing models that simplify devices as fixed inertia sources or focus solely on energy balance, this model meticulously analyzes the participation degree of devices like CHP units, power-to-gas/hydrogen, fuel cells, and thermal storage, and maps their operational states to equivalent grid inertia constants, considering the energy buffering provided by thermal and gas networks. Second, a multi-time-scale energy coordination strategy is developed that leverages the inherent tolerance of thermal and gas systems to short-term power fluctuations. This strategy explicitly treats the power grid's second/minute-level inertia regulation demands as an energy buffer that can be absorbed and managed within the longer minute/hour-level dispatch cycles of the thermal and gas networks, eliminating the need for highly precise short-term predictions. Third, a bi-level SRL optimization framework with an event-triggering mechanism is constructed to solve the model efficiently. The upper-level SRL optimizes short-term grid economic operation and inertia security, while the lower-level SRL minimizes long-term thermal and gas subsystem operational costs. Crucially, the event-triggering mechanism, based on a Lyapunov safety function, activates a computationally intensive safety recovery strategy only when the system state approaches or violates security boundaries; otherwise, a more efficient evolutionary policy is executed, thus balancing computational load and safety assurance. Simulation studies were conducted on a modified IES testbed comprising an IEEE 30-bus power system, a 20-node gas system, and a 14-node thermal system. Five different schemes were compared to validate the proposed method' s effectiveness. The results demonstrate that the proposed method (Scheme 5) effectively maintains grid inertia within a high-inertia security zone across various time periods, significantly reducing the probability and associated cost of frequency and rate-of-change-of-frequency (RoCoF) limit violations. Specifically, the frequency/RoCoF violation cost was reduced to nearly zero, and the thermal/gas network violation cost was cut by over 80% compared to the sub-optimal scheme. Furthermore, the multi-time-scale coordination reduced total multi-energy operational costs, with the comprehensive cost in Scheme 5 being at least 12.7% lower than schemes without full coordination. A comparative analysis with other reinforcement learning algorithms, including PPO, SAC, and CPO, showed that the proposed event-triggered bi-level SRL achieved higher and more stable reward values and a significantly lower cost function, indicating superior security constraint handling. Sensitivity analysis on the conversion gain coefficient confirmed its critical role in frequency stability, while computational efficiency tests proved that the event-triggering mechanism reduced the average cycle computation time by a factor of 3.5, making it highly suitable for the online dispatch of complex IES.
孙鹏, 杨茂, 滕云, 陈哲, 王令. 基于安全强化学习和多能源惯性协调的综合能源系统优化调度[J]. 电工技术学报, 2026, 41(11): 3718-3741.
Sun Peng, Yang Mao, Teng Yun, Chen Zhe, Wang Ling. Optimal Scheduling of Integrated Energy System Based on Safety Reinforcement Learning and Multi-Energy Inertia Coordination. Transactions of China Electrotechnical Society, 2026, 41(11): 3718-3741.
[1] 张苏涵, 顾伟, 俞睿智, 等. 综合能源系统建模与仿真: 综述、思考与展望[J]. 电力系统自动化, 2024, 48(17): 1-21. Zhang Suhan, Gu Wei, Yu Ruizhi, et al.Modeling and simulation of integrated energy system: review, reflection and prospects[J]. Automation of Electric Power Systems, 2024, 48(17): 1-21. [2] 刘硕, 滕云, 陈哲. 融合减碳型多能源微网的城市能源系统环境-经济协调优化模型[J]. 电工技术学报, 2025, 40(23): 7532-7553. Liu Shuo, Teng Yun, Chen Zhe.An environment-economic coordination optimization model for urban energy systems integrating carbon-reducing multi-energy microgrids[J]. Transactions of China Electro-technical Society, 2025, 40(23): 7532-7553. [3] Wang Siyuan, Wu Wenchuan, Lin Chenhui, et al.A dynamic equivalent energy storage model of natural gas networks for joint optimal dispatch of electricity-gas systems[J]. IEEE Transactions on Sustainable Energy, 2024, 15(1): 621-632. [4] Yang Nan, Xu Guobin, Fei Zhineng, et al.Two-stage coordinated robust planning of multi-energy ship microgrids considering thermal inertia and ship navigation[J]. IEEE Transactions on Smart Grid, 2025, 16(2): 1100-1111. [5] Hou Zeyin, Lu Shuai, Xu Yijun, et al.Privacy-preserved aggregate thermal dynamic model of buildings[J]. IEEE Transactions on Smart Grid, 2024, 15(6): 5653-5664. [6] Li Xu, Deng Jianhua, Liu Jichun.A two-layer and three-stage dynamic demand response game model considering the out of sync response for gases generators[J]. Renewable Energy, 2024, 228: 120681. [7] Sun Weijia, Wang Qi, Ye Yujian, et al.Unified modelling of gas and thermal inertia for integrated energy system and its application to multitype reserve procurement[J]. Applied Energy, 2022, 305: 117963. [8] Sun Qirun, Wu Zhi, Gu Wei, et al.Resilience assessment for integrated energy system considering gas-thermal inertia and system interdependency[J]. IEEE Transactions on Smart Grid, 2024, 15(2): 1509-1524. [9] 陶然, 赵冬梅, 徐辰宇, 等. 考虑电-气-热-交通相互依存的城市能源系统韧性评估与提升方法[J]. 电工技术学报, 2023, 38(22): 6133-6149. Tao Ran, Zhao Dongmei, Xu Chenyu, et al.Resilience assessment and enhancement methods for urban energy system considering electricity-gas-heat-transport interdependency[J]. Transactions of China Electro-technical Society, 2023, 38(22): 6133-6149. [10] 张帅, 刘文霞, 张艺伟, 等. 计及多重热惯性特征的区域综合能源系统可靠性评估[J]. 电工技术学报, 2023, 38(12): 3289-3305. Zhang Shuai, Liu Wenxia, Zhang Yiwei, et al.Reliability assessment of regional integrated energy system considering with multiple thermal inertia characteristics[J]. Transactions of China Electrotechnical Society, 2023, 38(12): 3289-3305. [11] 孙鹏, 滕云, 回茜, 等. 考虑异质能流输运特性的多能源系统惯量极限优化[J]. 中国电机工程学报, 2022, 42(10): 3642-3656. Sun Peng, Teng Yun, Hui Qian, et al.Inertia limit optimization of multi-energy system considering the transport characteristics of heterogeneous energy flow[J]. Proceedings of the CSEE, 2022, 42(10): 3642-3656. [12] 项颂, 苏鹏, 吴坚, 等. 基于多源储能协同的交直流送端系统惯量优化控制模型[J]. 中国电力, 2023, 56(4): 68-76. Xiang Song, Su Peng, Wu Jian, et al.Inertia optimization control model of AC/DC sending-end system based on multi-source energy storage coordination[J]. Electric Power, 2023, 56(4): 68-76. [13] 张怡静, 李智, 时艳强, 等. 基于储能惯量支撑的受端电网频率优化控制方法[J]. 电工技术学报, 2024, 39(11): 3556-3568. Zhang Yijing, Li Zhi, Shi Yanqiang, et al.Optimal frequency control method of receiving power grid based on energy storage inertia support[J]. Transactions of China Electrotechnical Society, 2024, 39(11): 3556-3568. [14] 缪蔡然, 朱姚培, 王琦, 等. 考虑气热惯性的综合能源系统参与辅助服务策略[J]. 电力建设, 2023, 44(11): 128-137. Miao Cairan, Zhu Yaopei, Wang Qi, et al.Ancillary service strategies for integrated energy systems considering gas-thermal inertia[J]. Electric Power Construction, 2023, 44(11): 128-137. [15] 王欣, 崔承刚, 王想想, 等. 基于安全强化学习的热电联产机组经济调度策略研究[J]. 系统仿真学报, 2025, 37(4): 968-981. Wang Xin, Cui Chenggang, Wang Xiangxiang, et al.Research on economic dispatching strategy of CHP units based on SRL[J]. Journal of System Simulation, 2025, 37(4): 968-981. [16] 焦昊, 殷岩岩, 吴晨, 等. 基于安全强化学习的主动配电网有功-无功协调优化调度[J]. 中国电力, 2024, 57(3): 43-50. Jiao Hao, Yin Yanyan, Wu Chen, et al.Coordinated optimization of active and reactive power of active distribution network based on safety reinforcement learning[J]. Electric Power, 2024, 57(3): 43-50. [17] Hao Guokai, Li Yuanzheng, Li Yang, et al.Lyapunov-based safe reinforcement learning for microgrid energy management[J]. IEEE Transactions on Neural Networks and Learning Systems, 2025, 36(6): 9985-9999. [18] Sun Peng, Teng Yun, Chen Zhe.Robust coordinated optimization for multi-energy systems based on multiple thermal inertia numerical simulation and uncertainty analysis[J]. Applied Energy, 2021, 296: 116982. [19] Hu Qinran, Han Rushuai, Quan Xiangjun, et al.Grid-forming inverter enabled virtual power plants with inertia support capability[J]. IEEE Transactions on Smart Grid, 2022, 13(5): 4134-4143. [20] 张祥宇, 朱永健, 付媛. 基于系统惯量需求的风储协同快速频率响应技术[J]. 中国电机工程学报, 2023, 43(14): 5415-5428. Zhang Xiangyu, Zhu Yongjian, Fu Yuan.Wind-storage cooperative fast frequency response technology based on system inertia demand[J]. Proceedings of the CSEE, 2023, 43(14): 5415-5428. [21] 袁铁江, 张江飞, 滕越. 基于虚拟同步机的新能源制氢系统协调控制策略[J]. 中国电机工程学报, 2025, 45(1): 163-174. Yuan Tiejiang, Zhang Jiangfei, Teng Yue.Coordinated control strategy of the renewable energy hydrogen production system based on VSG[J]. Proceedings of the CSEE, 2025, 45(1): 163-174. [22] 张苏涵, 顾伟, 陆帅, 等. 综合能源系统分析——从数值到解析(三):混合时间尺度经济调度[J]. 中国电机工程学报, 2026, 46(1): 173-185. Zhang Suhan, Gu Wei, Lu Shuai, et al.Integrated energy system analysis—from numerical to analytical (Ⅲ): hybrid timescale economic dispatch[J]. Proceedings of the CSEE, 2026, 46(1): 173-185 [23] Hu Jingwei, Yan Zheng, Xu Xiaoyuan, et al.Inertia market: mechanism design and its impact on generation mix[J]. Journal of Modern Power Systems and Clean Energy, 2023, 11(3): 744-756. [24] 张薇, 王浚宇, 杨茂, 等. 基于分布式双层强化学习的区域综合能源系统多时间尺度优化调度[J]. 电工技术学报, 2025, 40(11): 3529-3544. Zhang Wei, Wang Junyu, Yang Mao, et al.The multi-time-scale optimal scheduling for regional integrated energy system based on the distributed bi-layer reinforcement learning[J]. Transactions of China Electrotechnical Society, 2025, 40(11): 3529-3544. [25] 陈明昊, 朱月瑶, 孙毅, 等. 计及高渗透率光伏消纳与深度强化学习的综合能源系统预测调控[J]. 电工技术学报, 2024, 39(19): 6054-6071, 6103. Chen Minghao, Zhu Yueyao, Sun Yi, et al.The predictive-control optimization method for park integrated energy system considering the high penetration of photovoltaics and deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2024, 39(19): 6054-6071, 6103. [26] Ye Yujian, Wang Hongru, Chen Peiling, et al.Safe deep reinforcement learning for microgrid energy management in distribution networks with leveraged spatial-temporal perception[J]. IEEE Transactions on Smart Grid, 2023, 14(5): 3759-3775.
2026, Vol. 41 
