The Multi-Time-Scale Management Optimization Method for Park Integrated Energy System Based on the Bi-Layer Deep Reinforcement Learning
Chen Minghao1, Sun Yi1, Xie Zhiyuan2
1. School of Electrical and Electronic Engineering North China Electric Power University Beijing 102206 China; 2. School of Electrical and Electronic Engineering North China Electric Power University Baoding 071000 China
Abstract：In park integrated energy system (PIES), various facilities including gas turbine, gas boiler, power-to-gas, electric boiler, and so on, are utilized to realize the electricity/heat/gas purchasing, conversion and delivery. However, the complex multi-energy-coupling effects between various facilities and the different time-scale for adjustment of facilities’ operating status pose a main challenge to PIES management optimization for economic or environment benefits. Recent years, most of the papers were mainly devoted to investigate multi-time-scale optimization model or singular-level deep reinforcement learning (DRL) algorithm for PIES management, but most of them suffered from the obstacle for system modelling or the “dimension disaster” of DRL. To circumvent this problem, this paper proposes a multi-time-scale management-optimized method for PIES based on bi-layer deep deterministic policy gradient (DDPG). By respectively controlling energy conversion and storage facilities with long-term (half-hourly) and short-term (5 minutes) time scales with upper and lower DDPG agents, it improves the management performance of PIES. Firstly, a typical PIES model considering electricity, heat, and gas, is proposed to illustrate the energy coupling relations between different kinds of facilities, where their energy conversion process and operation constrains are cleanly defined. Secondly, utilizing DDPG algorithm, the optimization problem of PIES management is formulated mathematically, and the philosophy of DDPG are introduced to explain how agent’s policy could adaptively control PIES faced to different environment states. Thirdly, the multi-time-scale management-optimized method for PIES based on bi-layer DDPG is designed, which respectively control the heat and gas facilities under the long-time-scale and electricity facilities under the short-time-scale by two DDPG agents. Finally, the state space, action space, and reward function of upper and lower DDPG agents are defined, in which the adjustment of lower DDPG agent is considered as a kind of penalty cost for upper DDPG agent in each training round. Numerical results show that there is the “reward transferring effect” between bi-level DDPG agents during the training progress, i.e. the iterations number of upper DDPG agent’s reward converging is smaller than the lower DDPG agent. The reason is that the adjustment of P2G and EB are feedback to the upper DDPG agent as one penalty item inside its reward function and affect its converging. Relying on trained bi-level DDPG management model, the scheduling results of various energy conversion facilities are obtained facing floating demands and prices of multi-energy, in which the operating power of facilities consuming typical energy would decrease when the energy price is on-peak, and its outputs shortage would be made up by other facilities consuming cheaper energy, and vice versa. Finally, to verify the performance of the proposed model with singular level DDPG model and traditional optimization, same dataset and PIES model are combined to examinate the training time, economic benefits, facility number, and the time-scale. The results show that our proposed method is superior for each index. The following conclusions can be drawn from the simulation analysis: (1) A PIES model with multiple kinds of energy conversion and storage units are constructed, accompanying the uncertainty of renewable generation, demands, and energy purchasing prices. In this sense, it is closer to reality than existing PIES models. (2) The proposed PIES management model could learn and obtain the scheduling knowledge of different energy conversion facilities through the interactions between DDPG agent and environment with time-varing demands and prices of multi-energy. The propose model is appropriate to realize the adaptively controlling for PIES. (3) The proposed bi-level DDPG management method overcomes the difference of time-scale within the PIES’s different energy systems and the contradiction between the time-scale and state transition of DRL algorithm. Compared with traditional DRL algorithms, it owns the advantages of training efficiency and the economic benefits of PIES management.
陈明昊, 孙毅, 谢志远. 基于双层深度强化学习的园区综合能源系统多时间尺度优化管理[J]. 电工技术学报, 2023, 38(7): 1864-1881.
Chen Minghao, Sun Yi, Xie Zhiyuan. The Multi-Time-Scale Management Optimization Method for Park Integrated Energy System Based on the Bi-Layer Deep Reinforcement Learning. Transactions of China Electrotechnical Society, 2023, 38(7): 1864-1881.
[1] 孙伟卿, 刘唯, 张婕. 高比例可再生能源背景下配电网动态重构与移动储能协同优化[J]. 电力系统自动化, 2021, 45(19): 80-90. Sun Weiqing, Liu Wei, Zhang Jie.Collaborative optimization for dynamic reconfiguration of distribution network and mobile energy storage in background of high proportion of renewable energy[J]. Automation of Electric Power Systems, 2021, 45(19): 80-90. [2] 施泉生, 丁建勇, 刘坤, 等. 含电、气、热3种储能的微网综合能源系统经济优化运行[J]. 电力自动化设备, 2019, 39(8): 269-276, 293. Shi Quansheng, Ding Jianyong, Liu Kun, et al.Economic optimal operation of microgrid integrated energy system with electricity, gas and heat storage[J]. Electric Power Automation Equipment, 2019, 39(8): 269-276, 293. [3] Demirhan C D, Tso W W, Powell J B, et al.A multi-scale energy systems engineering approach towards integrated multi-product network optimization[J]. Applied Energy, 2021, 281: 116020. [4] 王雪纯, 陈红坤, 陈磊. 提升区域综合能源系统运行灵活性的多主体互动决策模型[J]. 电工技术学报, 2021, 36(11): 2207-2219. Wang Xuechun, Chen Hongkun, Chen Lei.Multi-player interactive decision-making model for operational flexibility improvement of regional integrated energy system[J]. Transactions of China Electrotechnical Society, 2021, 36(11): 2207-2219. [5] 张雨曼, 刘学智, 严正, 等. 光伏-储能-热电联产综合能源系统分解协调优化运行研究[J]. 电工技术学报, 2020, 35(11): 2372-2386. Zhang Yuman, Liu Xuezhi, Yan Zheng, et al.Decomposition-Coordination based optimization for PV-BESS-CHP integrated energy system[J]. Transactions of China Electrotechnical Society, 2020, 35(11): 2372-2386. [6] 陈昌铭, 张群, 黄亦昕, 等. 考虑最优建设时序和云储能的园区综合能源系统优化配置方法[J]. 电力系统自动化, 2022, 46(2): 24-32. Chen Changming, Zhang Qun, Huang Yixin, et al.Optimal configuration method of park-level integrated energy system considering optimal construction time sequence and cloud energy storage[J]. Automation of Electric Power Systems, 2022, 46(2): 24-32. [7] 熊宇峰, 司杨, 郑天文, 等. 基于主从博弈的工业园区综合能源系统氢储能优化配置[J]. 电工技术学报, 2021, 36(3): 507-516. Xiong Yufeng, Si Yang, Zheng Tianwen, et al.Optimal configuration of hydrogen storage in industrial park integrated energy system based on stackelberg game[J]. Transactions of China Electrotechnical Society, 2021, 36(3): 507-516. [8] 邓杰, 姜飞, 王文烨, 等. 考虑动态能效模型的园区综合能源系统梯级优化运行[J]. 电网技术, 2022, 46(3): 1027-1038. Deng Jie, Jiang Fei, Wang Wenye, et al.Study on cascade optimization operation of park-level integrated energy system considering dynamic energy efficiency model[J]. Power System Technology, 2022, 46(3): 1027-1038. [9] Chen Changming, Wu Xueyan, Li Yan, et al.Distributionally robust day-ahead scheduling of park-level integrated energy system considering generalized energy storages[J]. Applied Energy, 2021, 302: 117493. [10] 赵海彭, 苗世洪, 李超, 等. 考虑冷热电需求耦合响应特性的园区综合能源系统优化运行策略研究[J]. 中国电机工程学报, 2022, 42(2): 573-588. Zhao Haipeng, Miao Shihong, Li Chao, et al.Research on optimal operation strategy for park-level integrated energy system considering cold-heat-electric demand coupling response characteristics[J]. Proceedings of the CSEE, 2022, 42(2): 573-588. [11] 边晓燕, 史越奇, 裴传逊, 等. 计及经济性和可靠性因素的区域综合能源系统双层协同优化配置[J]. 电工技术学报, 2021, 36(21): 4529-4543. Bian Xiaoyan, Shi Yueqi, Pei Chuanxun, et al.Bi-level collaborative configuration optimization of integrated community energy system considering economy and reliability[J]. Transactions of China Electrotechnical Society, 2021, 36(21): 4529-4543. [12] 王磊, 周建平, 朱刘柱, 等. 基于分布式模型预测控制的综合能源系统多时间尺度优化调度[J]. 电力系统自动化, 2021, 45(13): 57-65. Wang Lei, Zhou Jianping, Zhu Liuzhu, et al.Multi-time-scale optimization scheduling of integrated energy system based on distributed model predictive control[J]. Automation of Electric Power Systems, 2021, 45(13): 57-65. [13] 何畅, 程杉, 徐建宇, 等. 基于多时间尺度和多源储能的综合能源系统能量协调优化调度[J]. 电力系统及其自动化学报, 2020, 32(2): 77-84, 97. He Chang, Cheng Shan, Xu Jianyu, et al.Coordinated optimal scheduling of integrated energy system considering multi-time scale and hybrid energy storage system[J]. Proceedings of the CSU-EPSA, 2020, 32(2): 77-84, 97. [14] 汤翔鹰, 胡炎, 耿琪, 等. 考虑多能灵活性的综合能源系统多时间尺度优化调度[J]. 电力系统自动化, 2021, 45(4): 81-90. Tang Xiangying, Hu Yan, Geng Qi, et al.Multi-time-scale optimal scheduling of integrated energy system considering multi-energy flexibility[J]. Automation of Electric Power Systems, 2021, 45(4): 81-90. [15] Li Xiaozhu, Wang Weiqing, Wang Haiyun.Hybrid time-scale energy optimal scheduling strategy for integrated energy system with bilateral interaction with supply and demand[J]. Applied Energy, 2021, 285: 116458. [16] 谢鹏, 蔡泽祥, 刘平, 等. 考虑多时间尺度不确定性耦合影响的风光储微电网系统储能容量协同优化[J]. 中国电机工程学报, 2019, 39(24): 7126-7136, 7486. Xie Peng, Cai Zexiang, Liu Ping, et al.Cooperative optimization of energy storage capacity for renewable and storage involved microgrids considering multi time scale uncertainty coupling influence[J]. Proceedings of the CSEE, 2019, 39(24): 7126-7136, 7486. [17] 张大海, 贠韫韵, 王小君, 等. 考虑广义储能及光热电站的电热气互联综合能源系统经济调度[J]. 电力系统自动化, 2021, 45(19): 33-42. Zhang Dahai, Yun Yunyun, Wang Xiaojun, et al.Economic dispatch of integrated electricity-heat-gas energy system considering generalized energy storage and concentrating solar power plant[J]. Automation of Electric Power Systems, 2021, 45(19): 33-42. [18] 程杉, 黄天力, 魏荣宗. 含冰蓄冷空调的冷热电联供型微网多时间尺度优化调度[J]. 电力系统自动化, 2019, 43(5): 30-40. Cheng Shan, Huang Tianli, Wei Rongzong.Multi-time-scale optimal scheduling of CCHP microgrid with ice-storage air-conditioning[J]. Automation of Electric Power Systems, 2019, 43(5): 30-40. [19] Wang L X, Zheng J H, Li M S, et al.Multi-time scale dynamic analysis of integrated energy systems: an individual-based model[J]. Applied Energy, 2019, 237: 848-861. [20] Watari D, Taniguchi I, Goverde H, et al.Multi-time scale energy management framework for smart PV systems mixing fast and slow dynamics[J]. Applied Energy, 2021, 289: 116671. [21] 杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47. Yang Ting, Zhao Liyuan, Liu Yachuang, et al.Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47. [22] Yang Ting, Zhao Liyuan, Li Wei, et al.Dynamic energy dispatch strategy for integrated energy system based on improved deep reinforcement learning[J]. Energy, 2021, 235: 121377. [23] 聂欢欢, 张家琦, 陈颖, 等. 基于双层强化学习方法的多能园区实时经济调度[J]. 电网技术, 2021, 45(4): 1330-1336. Nie Huanhuan, Zhang Jiaqi, Chen Ying, et al.Real-time economic dispatch of community integrated energy system based on a double-layer reinforcement learning method[J]. Power System Technology, 2021, 45(4): 1330-1336. [24] Xu Zhengwei, Han Guangjie, Liu Li, et al.Multi-energy scheduling of an industrial integrated energy system by reinforcement learning-based differential evolution[J]. IEEE Transactions on Green Communications and Networking, 2021, 5(3): 1077-1090. [25] 乔骥, 王新迎, 张擎, 等. 基于柔性行动器-评判器深度强化学习的电-气综合能源系统优化调度[J]. 中国电机工程学报, 2021, 41(3): 819-832. Qiao Ji, Wang Xinying, Zhang Qing, et al.Optimal dispatch of integrated electricity-gas system with soft actor-critic deep reinforcement learning[J]. Proceedings of the CSEE, 2021, 41(3): 819-832. [26] Zhang Bin, Hu Weihao, Li Jinghua, et al.Dynamic energy conversion and management strategy for an integrated electricity and natural gas system with renewable energy: deep reinforcement learning approach[J]. Energy Conversion and Management, 2020, 220: 113063. [27] Zhang Xiangyu, Biagioni D, Cai Mengmeng, et al.An edge-cloud integrated solution for buildings demand response using reinforcement learning[J]. IEEE Transactions on Smart Grid, 2021, 12(1): 420-431. [28] 倪爽, 崔承刚, 杨宁, 等. 基于深度强化学习的配电网多时间尺度在线无功优化[J]. 电力系统自动化, 2021, 45(10): 77-85. Ni Shuang, Cui Chenggang, Yang Ning, et al.Multi-time-scale online optimization for reactive power of distribution network based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(10): 77-85. [29] Zhang Xi, Liu Youbo, Duan Jiajun, et al.DDPG-based multi-agent framework for SVC tuning in urban power grid with renewable energy resources[J]. IEEE Transactions on Power Systems, 2021, 36(6): 5465-5475. [30] 刁涵彬, 李培强, 王继飞, 等. 考虑电/热储能互补协调的综合能源系统优化调度[J]. 电工技术学报, 2020, 35(21): 4532-4543. Diao Hanbin, Li Peiqiang, Wang Jifei, et al.Optimal dispatch of integrated energy system considering complementary coordination of electric/thermal energy storage[J]. Transactions of China Electrote-chnical Society, 2020, 35(21): 4532-4543. [31] 李山山, 李华强, 金智博, 等. 基于共享经济理念的园区分布式能源共享服务机制[J]. 中国电机工程学报, 2022, 42(1): 56-70. Li Shanshan, Li Huaqiang, Jin Zhibo, et al.Distributed energy sharing service mechanism for park based on the concept of sharing economy[J]. Proceedings of the CSEE, 2022, 42(1): 56-70. [32] 陈剑龙. 基于深度强化学习的微能源网能量管理策略研究[D]. 广州: 华南理工大学, 2020. [33] 邱高, 刘友波, 许立雄, 等. 基于深度确定性策略梯度的电网断面极限传输能力动态趋优控制[J]. 中国电机工程学报, 2021, 41(15): 5128-5138. Qiu Gao, Liu Youbo, Xu Lixiong, et al.A deep deterministic policy gradient based-dynamic optimizing control for power system total transfer capability[J]. Proceedings of the CSEE, 2021, 41(15): 5128-5138. [34] 刘俊峰, 陈剑龙, 王晓生, 等. 基于深度强化学习的微能源网能量管理与优化策略研究[J]. 电网技术, 2020, 44(10): 3794-3803. Liu Junfeng, Chen Jianlong, Wang Xiaosheng, et al.Energy management and optimization of multi-energy grid based on deep reinforcement learning[J]. Power System Technology, 2020, 44(10): 3794-3803. [35] Zhang Bin, Hu Weihao, Cao Di, et al.Soft actor-critic-based multi-objective optimized energy conversion and management strategy for integrated energy systems with renewable energy[J]. Energy Conversion and Management, 2021, 243: 114381. [36] 郭明萱, 穆云飞, 肖迁, 等. 考虑电池寿命损耗的园区综合能源电/热混合储能优化配置[J]. 电力系统自动化, 2021, 45(13): 66-75. Guo Mingxuan, Mu Yunfei, Xiao Qian, et al.Optimal configuration of electric/thermal hybrid energy storage for park-level integrated energy system considering battery life loss[J]. Automation of Electric Power Systems, 2021, 45(13): 66-75.
2023, Vol. 38 
