计及高渗透率光伏消纳与深度强化学习的综合能源系统预测调控方法

doi:10.19595/j.cnki.1000-6753.tces.231320

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (1079 KB)
输出: BibTeX | EndNote (RIS)

摘要

深度强化学习(deep reinforcement learning, DRL)是支撑园区综合能源系统(park integrated energy system, PIES)自适应调控其多能转换与存储设备,以消纳光伏发电及满足用户多能需求的重要技术。然而,DRL智能体通常利用其与PIES的实时环境-动作交互来调控其设备运行状态,难以在高渗透率光伏场景下考虑尖峰光伏发电并预留充足的储能资源。基于模型预测控制理论,该文提出了一种基于DRL与光伏发电区间预测的PIES优化调控方法。该方法面向电-气-热园区综合能源系统,利用时序卷积网络与核密度估计得到光伏发电区间预测结果,并采用柔性Actor-Critic(soft Actor-Critic, SAC)算法构建PIES预测优化调控模型。该模型将光伏发电预测区间构建为SAC智能体状态空间,通过迭代试错训练来获得PIES多能存储与转换的动态调节策略,从而优化光伏消纳率和运行成本。仿真实验表明,所提方法通过动态调节电、气、热三种能源转换设备的运行功率和预留三种储能设备的储能量,有效提升PIES在高渗透率光伏场景下的消纳率并优化其运行经济效益。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	陈明昊
	朱月瑶
	孙毅
	谢志远
	吴鹏

关键词 ：综合能源系统, 深度强化学习, SAC, TCN, 模型预测控制

Abstract：

As the interface between different energy infrastructures and users, park integrated energy system (PIES) has gained universal recognition for improving the reliability, resiliency, and profitability of multi-carrier energy systems by adaptively scheduling fast energy conversion units (e.g., combined heat and power (CHP), gas boiler (GB), and electric boiler (EB)) and participating in the various energy markets (e.g., electricity, heat, and natural gas). As a promising technology for replacing the rule-based decision-making in PIES, deep reinforcement learning (DRL) is a practical solution to identify the optimal control for energy conversion equipment. However, as PIES's customers perform more casual energy-consumption behaviors, the intermittency and volatility of demands make managing multi-energy supply and storage much harder for DRL agents. To tackle this task, focusing on the utilization of high penetration photovoltaic and the optimization of PIES's benefits, this article proposes an optimization scheduling method for PIES that combines the deep reinforcement learning and the interval prediction of photovoltaic power generation, considering the uncertainty of photovoltaic power generation.
Firstly, taking the equipment of energy conversion and storage as the scheduling objects, we design the predictive-control optimization structure, which can be divided into the facility level and information level, of PIES with electricity, gas, and heat, introducing the coordination between different sub-models. Secondly, the continuous and discrete feature data are respectively normalized and encoded for deterministic and probabilistic predicting the photovoltaic power generation based on temporal convolutional networks and kernel density estimation. Thirdly, based on the theory of model predictive control, the iteratively obtained intervals of photovoltaic power generation are used to construct the operating environment state of the control agent of soft actor critic (SAC) and to obtain the scheduling actions for PIES's equipment of energy conversion and storage.
Numerical results show that the proposed PFP-SAC method is able to identify the generation of photovoltaic power, improve the utilization of PV generation, and optimize the benefit of PIES by dynamic scheduling these conversion and storage equipment and increasing their operation efficiency. Meanwhile, these results prove that the gaps of energy purchasing price is the motivation of multi-energy conversion for PIES and its cost-saving. On the contrast, in the scenario of high penetration of photovoltaic power, the multi-energy conversion and storage of PIES need to simultaneously consider the consumption demand for photovoltaic power and the price-gaps of multi energy, and improve its utilization of photovoltaic power generation as much as possible by reserving energy storage resources. Finally, taking the traditional SAC and deep deterministic policy gradient (DDPG) as the benchmarks, the same datasets are utilized to verify the performance of proposed method and benchmarks, including the scheduling benefit and SOC of storage. The results show that our proposed method is superior for each index.
The following conclusions can be drawn from the simulation analysis: 1) A PIES model with multiple kinds of energy conversion and storage units are constructed, accompanying the uncertainty of renewable generation, demands, and energy purchasing prices. In this sense, it is closer to reality than existing PIES models. 2) Model predictive control theory and deep reinforcement learning algorithm are employed to cope with the intermittent nature of multi-energy demands. This paper constructs the state space of DRL models with prediction intervals of multi-energy demands of PIES, which is obtained by TCN and KDE. 3) Taking the operating cost saving as the prioritize objective and the generation utilization of photovoltaic power as secondary goal of PIES scheduling, soft actor critic, which is a promising DRL algorithm, is applied to reduce the operational expenditures and improve the usage of multi-energy storage capacity as much as possible. Compared with traditional DRL algorithms, it owns the advantages of predicting accuracy and the economic benefits of PIES management.

Key words： Integrated energy system deep reinforcement learning soft actor-critic temporal convolutional network model predictive control

收稿日期: 2023-08-16

PACS:

TM73

基金资助:

国家电网有限公司科技项目资助（52130X230008）

通讯作者: 孙毅, 男,1972年生,教授,博士生导师,研究方向为需求侧管理、电力系统自动化与信息化。E-mail：sy@ncepu.edu.cn

作者简介: 陈明昊, 男,1997年生,博士研究生,研究方向为电力信息与通信系统、综合能源系统管理。E-mail：120212101090@ncepu.edu.cn

引用本文:

陈明昊, 朱月瑶, 孙毅, 谢志远, 吴鹏. 计及高渗透率光伏消纳与深度强化学习的综合能源系统预测调控方法[J]. 电工技术学报, 0, (): 9039-39. Chen Minghao, Zhu Yueyao, Sun Yi, Xie Zhiyuan, Wu Peng. The Predictive-control Optimization Method for Park Integrated Energy System Considering the High Penetration of Photovoltaics and Deep Reinforcement Learning. Transactions of China Electrotechnical Society, 0, (): 9039-39.

链接本文:

https://dgjsxb.ces-transaction.com/CN/10.19595/j.cnki.1000-6753.tces.231320 https://dgjsxb.ces-transaction.com/CN/Y0/V/I/9039

[1] 国家能源局.我国可再生能源发电总装机突破13亿千瓦[EB/OL].国家能源局:[2023-11-18].http://www.nea.gov.cn/2023-07/19/c_1310733273.htm.
[2] 吴孟雪, 房方. 计及风光不确定性的电-热-氢综合能源系统分布鲁棒优化[J]. 电工技术学报, 2023, 38(13): 3473-3485.
Wu Mengxue, Fang Fang.Distributionally robust optimization of electricity-heat-hydrogen integrated energy system with wind and solar uncertainties[J]. Transactions of China Electrotechnical Society, 2023, 38(13): 3473-3485.
[3] 罗潇,任洲洋,温紫豪,等.考虑氢能系统热回收的电氢区域综合能源系统日前优化运行[J/OL].电工技术学报:1-14[2023-11-13].
Luo Xiao,Ren Zhouyang,Wen Zihao,et al.A day-ahead dispatching method of regional integrated electric-hydrogen energy systems considering the heat recycle of hydrogen systems[J].Transactions of China Electrotechnical Society:1-14[2023-11-13].
[4] Chen Ruijun, Tsay Y S, Zhang Ting.A multi-objective optimization strategy for building carbon emission from the whole life cycle perspective[J]. Energy, 2023, 262: 125373.
[5] 曾艾东, 王佳伟, 邹宇航, 等. 考虑供热管网储热的综合能源系统多时间尺度优化调度[J]. 高电压技术, 2023, 49(10): 4192-4202.
Zeng Aidong, Wang Jiawei, Zou Yuhang, et al.Multi-time-scale optimal scheduling of integrated energy system considering heat storage characteristics of heating network[J]. High Voltage Engineering, 2023, 49(10): 4192-4202.
[6] Robert L R, Ravi Singh L R S. Economic emission dispatch of hydro-thermal-wind using CMQLSPSN technique[J]. IET Renewable Power Generation, 2020, 14(14): 2680-2692.
[7] 黄文杰, 崔雪, 陈君, 等. 基于多智能体Q学习算法的能源互联园区协调调度[J]. 武汉大学学报(工学版), 2022, 55(11): 1141-1148.
Huang Wenjie, Cui Xue, Chen Jun, et al.Coordinated scheduling of energy interconnected parks based on multi-agent Q-learning algorithm[J]. Engineering Journal of Wuhan University, 2022, 55(11): 1141-1148.
[8] Correa-Jullian C, López Droguett E, Cardemil J M.Operation scheduling in a solar thermal system: a reinforcement learning-based framework[J]. Applied Energy, 2020, 268: 114943.
[9] 张帆, 武东昊, 陈玉萍, 等. 多智能体深度强化学习的分布式园区综合能源系统经济调度策略[J]. 电力系统及其自动化学报, 2022, 34(12): 18-26.
Zhang Fan, Wu Donghao, Chen Yuping, et al.Economic scheduling strategy for integrated energy system in distributed parks based on multi-agent deep reinforcement learning[J]. Proceedings of the CSU-EPSA, 2022, 34(12): 18-26.
[10] 杨挺, 赵黎媛, 刘亚闯, 等. 基于深度强化学习的综合能源系统动态经济调度[J]. 电力系统自动化, 2021, 45(5): 39-47.
Yang Ting, Zhao Liyuan, Liu Yachuang, et al.Dynamic economic dispatch for integrated energy system based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(5): 39-47.
[11] 程义, 李更丰. 基于双层模仿学习的多园区综合能源系统分布式协同优化调度[J]. 电力系统自动化, 2022, 46(24): 16-25.
Cheng Yi, Li Gengfeng.Distributed collaborative optimal dispatch of multi-park integrated energy system based on bilayer imitation learning[J]. Automation of Electric Power Systems, 2022, 46(24): 16-25.
[12] 陈明昊, 孙毅, 胡亚杰, 等. 基于纵向联邦强化学习的居民社区综合能源系统协同训练与优化管理方法[J]. 中国电机工程学报, 2022, 42(15): 5535-5549.
Chen Minghao, Sun Yi, Hu Yajie, et al. The collaborative training and management-optimized method for residential integrated energy system based on vertical federated reinforcement learning[J]. Proceedings of the CSEE, 2022, 42(15): 5535-5549, 中插13.
[13] Bag A, Subudhi B, Ray P K.A combined reinforcement learning and sliding mode control scheme for grid integration of a PV system[J]. CSEE Journal of Power and Energy Systems, 2019, 5(4): 498-506.
[14] 刘自发, 张婷, 王岩. 基于模型预测控制的主动配电网多场景变时间尺度优化调度[J]. 电力自动化设备, 2022, 42(4): 121-128.
Liu Zifa, Zhang Ting, Wang Yan.Multi-scenario variable time scale optimal scheduling of active distribution network based on model predictive control[J]. Electric Power Automation Equipment, 2022, 42(4): 121-128.
[15] 余洋, 贾浩, 陈启维, 等. 基于改进模型预测控制的电-气系统新能源功率波动平滑策略[J]. 电力建设, 2021, 42(9): 65-73.
Yu Yang, Jia Hao, Chen Qiwei, et al.Fluctuation smoothing strategy of new energy power for electricity-gas interconnected system based on improved model predictive control[J]. Electric Power Construction, 2021, 42(9): 65-73.
[16] 舒晓欣, 林其友, 张健, 等. 基于双层模型预测控制的微电网经济调度[J]. 浙江工业大学学报, 2023, 51(3): 324-329.
Shu Xiaoxin, Lin Qiyou, Zhang Jian, et al.A Two-stage model predictive control strategy for economical operation of microgrid[J]. Journal of Zhejiang University of Technology, 2023, 51(3): 324-329.
[17] 吴梦丹, 张俊礼, 吴嘉峰, 等. 基于经济模型预测控制的光伏光热一体化热泵系统动态能效优化[J]. 中国电机工程学报, 2023, 43(6): 2119-2130.
Wu Mengdan, Zhang Junli, Wu Jiafeng, et al.Dynamic energy efficiency optimization of photovoltaic/thermal integrated heat pump system based on economic model predictive control[J]. Proceedings of the CSEE, 2023, 43(6): 2119-2130.
[18] 王阳, 刘希喆. 基于GRU-MPC的光储充电站日前-日内两阶段优化控制[J]. 电力自动化设备, 2022, 42(10): 177-183.
Wang Yang, Liu Xizhe.Day-ahead and intra-day two-stage optimal control of photovoltaic-energy storage charging station based on GRU-MPC[J]. Electric Power Automation Equipment, 2022, 42(10): 177-183.
[19] 王晓霞, 俞敏, 霍泽健, 等. 基于近邻传播聚类与LSTNet的分布式光伏电站群短期功率预测[J]. 电力系统自动化, 2023, 47(6): 133-141.
Wang Xiaoxia, Yu Min, Huo Zejian, et al.Short-term power forecasting of distributed photovoltaic station clusters based on affinity propagation clustering and long short-term time-series network[J]. Automation of Electric Power Systems, 2023, 47(6): 133-141.
[20] Zhang Qingyong, Chen Jiahua, Xiao Gang, et al.TransformGraph: a novel short-term electricity net load forecasting model[J]. Energy Reports, 2023, 9: 2705-2717.
[21] Blad C, Bøgh S, Kallesøe C S.Data-driven offline reinforcement learning for HVAC-systems[J]. Energy, 2022, 261: 125290.
[22] 陈柘, 刘嘉华, 赵斌, 等. 基于GCN和TCN的多因素城市路网出租车需求预测[J]. 控制与决策, 2023, 38(4): 1031-1038.
Chen Zhe, Liu Jiahua, Zhao Bin, et al.Multi-factor taxi demand forecasting for urban road network based on GCN and TCN[J]. Control and Decision, 2023, 38(4): 1031-1038.
[23] 邢晨, 张照贝. 基于改进时间卷积网络的短期光伏出力概率预测方法[J]. 太阳能学报, 2023, 44(2): 373-380.
Xing Chen, Zhang Zhaobei.Short-term probabilistic forecasting method of photovoltaic output power based on improved temporal convolutional network[J]. Acta Energiae Solaris Sinica, 2023, 44(2): 373-380.
[24] 宋绍剑, 姜屹远, 刘斌. 一种TCN的改进模型及其在短期光伏功率区间预测的应用[J]. 计算机应用研究, 2023, 40(10): 3064-3069.
Song Shaojian, Jiang Yiyuan, Liu Bin.Improved TCN model and its application in short-term photovoltaic power interval prediction[J]. Application Research of Computers, 2023, 40(10): 3064-3069.
[25] Wang Kejun, Qi Xiaoxia, Liu Hongda.Photovoltaic power forecasting based LSTM-Convolutional Network[J]. Energy, 2019, 189: 116225.
[26] 万灿, 崔文康, 宋永华. 新能源电力系统概率预测:基本概念与数学原理[J]. 中国电机工程学报, 2021, 41(19): 6493-6509.
Wan Can, Cui Wenkang, Song Yonghua.Probabilistic forecasting for power systems with renewable energy sources: basic concepts and mathematical principles[J]. Proceedings of the CSEE, 2021, 41(19): 6493-6509.
[27] Cheng Lilin, Zang Haixiang, Xu Yan, et al.Probabilistic residential load forecasting based on micrometeorological data and customer consumption pattern[J]. IEEE Transactions on Power Systems, 2021, 36(4): 3762-3775.
[28] 陈明昊, 孙毅, 谢志远. 基于双层深度强化学习的园区综合能源系统多时间尺度优化管理[J]. 电工技术学报, 2023, 38(7): 1864-1881.
Chen Minghao, Sun Yi, Xie Zhiyuan.The multi-time-scale management optimization method for park integrated energy system based on the Bi-layer deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2023, 38(7): 1864-1881.
[29] 朱振山, 陈哲盛, 盛明鼎. 基于柔性行动器-评判器的园区综合能源系统运行优化[J]. 高电压技术, 2022, 48(12): 4949-4958.
Zhu Zhenshan, Chen Zhesheng, Sheng Mingding.Operation optimization of park-level integrated energy system based on soft actor-critic[J]. High Voltage Engineering, 2022, 48(12): 4949-4958.