The Multi-Time-Scale Optimal Scheduling for Regional Integrated Energy System Based on the Distributed Bi-Layer Reinforcement Learning
Zhang Wei1,2, Wang Junyu2, Yang Mao1,2, Yan Gangui1,2
1. Key Laboratory of Modern Power System Simulation and Control & Renewable Energy Technology Ministry of Education Northeast Electric Power University Jilin 132012 China 2. School of Electrical Engineering Northeast Electric Power University Jilin 132012 China
Abstract：Under the background of the dual carbon goals, the regional integrated energy system (RIES) can achieve interconversion between heterogeneous energy sources due to its multi-energy coupling characteristics, providing new technical support for energy-saving and efficient operation of modern energy systems. Due to the differences in the flow of heterogeneous energy sources in transmission pipelines, existing research usually adopts convex relaxation techniques or linearization methods to model and solve the RIES for multi-time-scale, and relies on high-precision source-load forecasting results and equipment mathematical modeling to improve the reliability of scheduling decisions. However, the increasingly complex internal energy coupling structure of the RIES has increased the difficulty of its refined mathematical modeling and solution, posing challenges to the real-time scheduling decisions and safe optimal operation of the RIES. Therefore, this paper proposes an improved distributed bi-layer proximal policy optimization (DBLPPO) deep reinforcement learning scheduling model. This model can achieve multi-time-scale optimization management of various energy networks in the RIES and avoid the optimization difficulties caused by non-convex nonlinear model structures in scheduling solutions. Firstly, the power output, storage, and transformation of internal energy in the RIES are constructed into a high-dimensional space Markov decision process mathematical model. Secondly, based on the improved distributed proximal policy optimization (DPPO) algorithm, a sequential decision description is made for it, and a control model of the internal bi-layer proximal policy optimization (PPO) is constructed. The local network adopts the "coupling first, then decoupling" solution approach to carry out multi-time-scale optimization decision-making for the cold-heat system and the power system. In the early stage of long time scale, the inner and outer models perform coupled solutions, and the RIES cold-heat system and power system achieve coordinated optimal operation. In the remaining short time scales, the inner and outer models perform decoupled solutions and carry out short-term flexible regulation of the power system. The inner and outer models interact with each other and fluctuating convergence towards the reward maximization direction, eventually achieving multi-time-scale optimization scheduling of the RIES cold-heat system and power system. This paper conducts simulation experiments with a cold-heat-electric RIES as the scheduling scenario, and compares the scheduling results of the DBLPPO scheduling model with those of a single time scale scheduling model (PPO, DPPO). The results show that the DBLPPO scheduling model can flexibly regulate the system's adjustable resources in the short time scale, meet the power fluctuation requirements of electricity, heat, and cold loads in the short time scale, and has the lowest comprehensive operating cost, which is 24.47% lower than that of the DPPO scheduling model and 28.54% lower than that of the PPO scheduling model. In addition, simulation experiments are conducted with the DBLPPO scheduling model and the bi-layer PPO scheduling model in the same scenario, and the results show that the distributed structure of the DBLPPO scheduling model still has a significant advantage in improving model training efficiency, which can effectively shorten the training time, 10.01% shorter than that of the dual-layer PPO scheduling model. Through case analysis, it is verified that the proposed scheduling model can achieve coordinated optimal management of various energy networks in the RIES at different time scales, accelerate the optimal decision-making speed of the multi-time-scale scheduling model, and by virtue of the fast adaptability of the deep reinforcement learning algorithm, efficiently solve random optimization problems in complex RIES scenarios, and improve the economic benefits of system operation. The next step of work will be to improve the model to enhance the environmental awareness ability of the inner model, so that its decision-making scheme is always the optimal scheduling decision in the long time scale.
张薇, 王浚宇, 杨茂, 严干贵. 基于分布式双层强化学习的区域综合能源系统多时间尺度优化调度[J]. 电工技术学报, 2025, 40(11): 3529-3544.
Zhang Wei, Wang Junyu, Yang Mao, Yan Gangui. The Multi-Time-Scale Optimal Scheduling for Regional Integrated Energy System Based on the Distributed Bi-Layer Reinforcement Learning. Transactions of China Electrotechnical Society, 2025, 40(11): 3529-3544.
[1] 陈艳波, 张宁, 李嘉祺, 等. 零碳园区研究综述及展望[J]. 中国电机工程学报, 2024, 44(14): 5496-5517. Chen Yanbo, Zhang Ning, Li Jiaqi, et al.Review and prospect of zero carbon park research[J]. Proceedings of the CSEE, 2024, 44(14): 5496-5517. [2] 高玉, 王琦, 陈严, 等. 考虑需求响应和能量梯级利用的含氢综合能源系统优化调度[J]. 电力系统自动化, 2023, 47(4): 51-59. Gao Yu, Wang Qi, Chen Yan, et al.Optimal dispatch of integrated energy system with hydrogen considering demand response and cascade energy utilization[J]. Automation of Electric Power Systems, 2023, 47(4): 51-59. [3] 杨龙, 张沈习, 程浩忠, 等. 区域低碳综合能源系统规划关键技术与挑战[J]. 电网技术, 2022, 46(9): 3290-3304. Yang Long, Zhang Shenxi, Cheng Haozhong, et al.Regional low-carbon integrated energy system planning: key technologies and challenges[J]. Power System Technology, 2022, 46(9): 3290-3304. [4] 潘超, 范宫博, 王锦鹏, 等. 灵活性资源参与的电热综合能源系统低碳优化[J]. 电工技术学报, 2023, 38(6): 1633-1647. Pan Chao, Fan Gongbo, Wang Jinpeng, et al.Low-carbon optimization of electric and heating integrated energy system with flexible resource participation[J]. Transactions of China Electrotechnical Society, 2023, 38(6): 1633-1647. [5] 吴孟雪, 房方. 计及风光不确定性的电-热-氢综合能源系统分布鲁棒优化[J]. 电工技术学报, 2023, 38(13): 3473-3485. Wu Mengxue, Fang Fang.Distributionally robust optimization of electricity-heat-hydrogen integrated energy system with wind and solar uncertainties[J]. Transactions of China Electrotechnical Society, 2023, 38(13): 3473-3485. [6] Li Canbing, Yang Hanyu, Shahidehpour M, et al.Optimal planning of islanded integrated energy system with solar-biogas energy supply[J]. IEEE Transactions on Sustainable Energy, 2020, 11(4): 2437-2448. [7] 任洲洋, 王皓, 李文沅, 等. 基于氢能设备多状态模型的电氢区域综合能源系统可靠性评估[J]. 电工技术学报, 2023, 38(24): 6744-6759. Ren Zhouyang, Wang Hao, Li Wenyuan, et al.Reliability evaluation of electricity-hydrogen regional integrated energy systems based on the multi-state models of hydrogen energy equipment[J]. Transactions of China Electrotechnical Society, 2023, 38(24): 6744-6759. [8] Wang Yudong, Hu Junjie, Liu Nian.Energy management in integrated energy system using energy-carbon integrated pricing method[J]. IEEE Transactions on Sustainable Energy, 2023, 14(4): 1992-2005. [9] 董雷, 李扬, 陈盛, 等. 考虑多重不确定性与电碳耦合交易的多微网合作博弈优化调度[J]. 电工技术学报, 2024, 39(9): 2635-2651. Dong Lei, Li Yang, Chen Sheng, et al.Multi-microgrid cooperative game optimization scheduling considering multiple uncertainties and coupled electricity-carbon transactions[J]. Transactions of China Electrotechnical Society, 2024, 39(9): 2635-2651. [10] 曾艾东, 王佳伟, 邹宇航, 等. 考虑供热管网储热的综合能源系统多时间尺度优化调度[J]. 高电压技术, 2023, 49(10): 4192-4202. Zeng Aidong, Wang Jiawei, Zou Yuhang, et al.Multi-time-scale optimal scheduling of integrated energy system considering heat storage characteristics of heating network[J]. High Voltage Engineering, 2023, 49(10): 4192-4202. [11] 栗海润, 穆云飞, 贾宏杰, 等. 考虑量化储热的多区域电-热综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(增刊1): 16-27. Li Hairun, Mu Yunfei, Jia Hongjie, et al.Optimal dispatching of multi-regional electricity-heat integrated energy system considering quantitative thermal storage[J]. Proceedings of the CSEE, 2021, 41(S1): 16-27. [12] 徐卫君, 张伟, 胡宇涛, 等. 先进绝热压缩空气储能多能流优化调度模型[J]. 电工技术学报, 2022, 37(23): 5944-5955. Xu Weijun, Zhang Wei, Hu Yutao, et al.Multi energy flow optimal scheduling model of advanced adiabatic compressed air energy storage[J]. Transactions of China Electrotechnical Society, 2022, 37(23): 5944-5955. [13] 汤翔鹰, 胡炎, 耿琪, 等. 考虑多能灵活性的综合能源系统多时间尺度优化调度[J]. 电力系统自动化, 2021, 45(4): 81-90. Tang Xiangying, Hu Yan, Geng Qi, et al.Multi-time-scale optimal scheduling of integrated energy system considering multi-energy flexibility[J]. Automation of Electric Power Systems, 2021, 45(4): 81-90. [14] 韩丽, 王晓静, 鲁盼盼, 等. 考虑风电功率预测误差分时补偿的电热联合系统多时间尺度调度[J]. 电力系统自动化, 2023, 47(1): 74-85. Han Li, Wang Xiaojing, Lu Panpan, et al.Multi-time-scale dispatching of integrated electricity and thermal system considering time-sharing compensation of wind power forecasting error[J]. Automation of Electric Power Systems, 2023, 47(1): 74-85. [15] 李勇, 凌锋, 乔学博, 等. 基于网侧资源协调的自储能柔性互联配电系统日前-日内优化[J]. 电工技术学报, 2024, 39(3): 758-773, 923. Li Yong, Ling Feng, Qiao Xuebo, et al.Day-ahead and intra-day optimization of flexible interconnected distribution system with self-energy storage based on the grid-side resource coordination[J]. Transactions of China Electrotechnical Society, 2024, 39(3): 758-773, 923. [16] 陈志颖, 温步瀛, 朱振山. 计及风电相关性的区域综合能源系统多时间尺度优化调度[J]. 电力自动化设备, 2023, 43(8): 25-32. Chen Zhiying, Wen Buying, Zhu Zhenshan.Multi-time scale optimal scheduling of regional integrated energy system considering wind power correlation[J]. Electric Power Automation Equipment, 2023, 43(8): 25-32. [17] 南斌, 姜春娣, 董树锋, 等. 计及源荷不确定性的综合能源系统日前-日内协调优化调度[J]. 电网技术, 2023, 47(9): 3669-3683. Nan Bin, Jiang Chundi, Dong Shufeng, et al.Day-ahead and intra-day coordinated optimal scheduling of integrated energy system considering uncertainties in source and load[J]. Power System Technology, 2023, 47(9): 3669-3683. [18] 陈静, 杨坤漓, 张岩, 等. 多时间尺度下计及综合需求响应和碳捕集-电转气联合运行的综合能源系统优化调度[J]. 电力自动化设备, 2024, 44(6): 68-76. Chen Jing, Yang Kunli, Zhang Yan, et al.Optimal scheduling of integrated energy system considering integrated demand response and joint operation of carbon capture and power-to-gas under multiple time scales[J]. Electric Power Automation Equipment, 2024, 44(6): 68-76. [19] 李天格, 胡志坚, 陈志, 等. 计及电-气-热-氢需求响应的综合能源系统多时间尺度低碳运行优化策略[J]. 电力自动化设备, 2023, 43(1): 16-24. Li Tiange, Hu Zhijian, Chen Zhi, et al.Multi-time scale low-carbon operation optimization strategy of integrated energy system considering electricity-gas-heat-hydrogen demand response[J]. Electric Power Automation Equipment, 2023, 43(1): 16-24. [20] 崔杨, 曾鹏, 王铮, 等. 计及电价型需求侧响应含碳捕集设备的电-气-热综合能源系统低碳经济调度[J]. 电网技术, 2021, 45(2): 447-461. Cui Yang, Zeng Peng, Wang Zheng, et al.Low-carbon economic dispatch of electricity-gas-heat integrated energy system with carbon capture equipment considering price-based demand response[J]. Power System Technology, 2021, 45(2): 447-461. [21] 李壹民, 邵振国, 黄圆皓, 等. 电-热综合能源系统仿射型区间多能流算法[J]. 中国电机工程学报, 2023, 43(9): 3429-3444. Li Yimin, Shao Zhenguo, Huang Yuanhao, et al.Interval multi-energy flow algorithm for integrated heat and electricity system based on affine arithmetic[J]. Proceedings of the CSEE, 2023, 43(9): 3429-3444. [22] 林旗斌. 基于模型预测控制的含压缩空气储能微能网多时间尺度优化调度方法[J]. 电气技术, 2023, 24(7): 11-19, 63. Lin Qibin.Multi-time scale optimal scheduling strategy for micro energy network with compressed air energy storage based on model predictive control[J]. Electrical Engineering, 2023, 24(7): 11-19, 63. [23] Yi Zonggen, Luo Yusheng, Westover T, et al.Deep reinforcement learning based optimization for a tightly coupled nuclear renewable integrated energy system[J]. Applied Energy, 2022, 328: 120113. [24] Zhou Kunshu, Zhou Kaile, Yang Shanlin.Reinforcement learning-based scheduling strategy for energy storage in microgrid[J]. Journal of Energy Storage, 2022, 51: 104379. [25] Li Jifeng, He Xingtang, Li Weidong, et al.Low-carbon optimal learning scheduling of the power system based on carbon capture system and carbon emission flow theory[J]. Electric Power Systems Research, 2023, 218: 109215. [26] 乔骥, 王新迎, 张擎, 等. 基于柔性行动器-评判器深度强化学习的电-气综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(3): 819-833. Qiao Ji, Wang Xinying, Zhang Qing, et al.Optimal dispatch of integrated electricity-gas system with soft actor-critic deep reinforcement learning[J]. Proceedings of the CSEE, 2021, 41(3): 819-833. [27] Li Yang, Bu Fanjin, Li Yuanzheng, et al.Optimal scheduling of island integrated energy systems considering multi-uncertainties and hydrothermal simultaneous transmission: a deep reinforcement learning approach[J]. Applied Energy, 2023, 333: 120540. [28] 彭春华, 陈思畏, 徐佳璐, 等. 综合能源系统混合时间尺度多目标强化学习低碳经济调度[J]. 电网技术, 2022, 46(12): 4914-4925. Peng Chunhua, Chen Siwei, Xu Jialu, et al.Low carbon economic scheduling for integrated energy systems with mixed timescale & multi-objective reinforcement learning[J]. Power System Technology, 2022, 46(12): 4914-4925. [29] 陈明昊, 孙毅, 谢志远. 基于双层深度强化学习的园区综合能源系统多时间尺度优化管理[J]. 电工技术学报, 2023, 38(7): 1864-1881. Chen Minghao, Sun Yi, Xie Zhiyuan.The multi-time-scale management optimization method for park integrated energy system based on the bi-layer deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2023, 38(7): 1864-1881. [30] 李鹏, 王加浩, 黎灿兵, 等. 计及源荷不确定性与设备变工况特性的园区综合能源系统协同优化运行方法[J]. 中国电机工程学报, 2023, 43(20): 7802-7812. Li Peng, Wang Jiahao, Li Canbing, et al.Collaborative optimal scheduling of the community integrated energy system considering source-load uncertainty and equipment off-design performance[J]. Proceedings of the CSEE, 2023, 43(20): 7802-7812. [31] 郑诗程, 许浩, 郎佳红, 等. 计及光伏不确定性的多区域综合能源系统多场景分布鲁棒优化调度[J]. 太阳能学报, 2024, 45(3): 460-469. Zheng Shicheng, Xu Hao, Lang Jiahong, et al.Multi-scenario distributed robust optimal scheduling of multi-area integrated energy systems considering photovoltaic uncertainty[J]. Acta Energiae Solaris Sinica, 2024, 45(3): 460-469. [32] 冯斌, 胡轶婕, 黄刚, 等. 基于深度强化学习的新型电力系统调度优化方法综述[J]. 电力系统自动化, 2023, 47(17): 187-199. Feng Bin, Hu Yijie, Huang Gang, et al.Review on optimization methods for new power system dispatch based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2023, 47(17): 187-199. [33] Duan Jiajun, Shi Di, Diao Ruisheng, et al.Deep-reinforcement-learning-based autonomous voltage control for power grid operations[J]. IEEE Transactions on Power Systems, 2020, 35(1): 814-817. [34] 周毅斌, 肖浩, 裴玮, 等. 基于纵向联邦学习的微电网群协同优化运行与策略进化[J]. 电力系统自动化, 2023, 47(11): 121-132. Zhou Yibin, Xiao Hao, Pei Wei, et al.Collaborative optimization operation and strategy evolution of microgrid cluster based on vertical federated learning[J]. Automation of Electric Power Systems, 2023, 47(11): 121-132. [35] 董雷, 杨子民, 乔骥, 等. 基于分层约束强化学习的综合能源多微网系统优化调度[J]. 电工技术学报, 2024, 39(5): 1436-1453. Dong Lei, Yang Zimin, Qiao Ji, et al.Optimal scheduling of integrated energy multi-microgrid system based on hierarchical constraint reinforcement learning[J]. Transactions of China Electrotechnical Society, 2024, 39(5): 1436-1453. [36] 蔺伟山, 王小君, 孙庆凯, 等. 计及安全约束的综合能源系统深度强化学习优化调度策略研究[J]. 电网技术, 2023, 47(5): 1970-1983. Lin Weishan, Wang Xiaojun, Sun Qingkai, et al.Optimal dispatch strategy of integrated energy system based on deep reinforcement learning considering security constraints[J]. Power System Technology, 2023, 47(5): 1970-1983. [37] Zhou Suyang, Hu Zijian, Gu Wei, et al.Combined heat and power system intelligent economic dispatch: a deep reinforcement learning approach[J]. International Journal of Electrical Power & Energy Systems, 2020, 120: 106016.
2025, Vol. 40 
