Efficient Deep Deterministic Policy Gradient Algorithm for Real-Time Self-Dispatch of Wind-Storage Power Plant
Song Yuhao1, Wei Wei1, Huang Shaowei1, Wu Qiren2, Mei Shengwei1
1. Department of Electrical Engineering Tsinghua University Beijing 100084 China; 2. China Three Gorges Renewables (Group) Co. Ltd Beijing 101100 China
Abstract：The development of wind power and other renewable energy is of great significance to achieve the dual carbon goal, and the wind-storage power plant is the main form of wind power connected to the power grid in the future. This paper studies the real-time self-dispatch problem of the wind-storage power plant commercialized on the generating side, with the goal of maximizing its expected income. Due to the large prediction error of the field-level wind power and the difficulty in accurately predicting the electricity price of the grid due to the limited information of independent power producers, the real-time self-dispatch of the wind-storage power plant is faced with multiple uncertainties, which is extremely challenging. In this paper, an efficient DDPG algorithm was proposed to solve the real-time self-dispatch strategy of the wind-storage power plant, and realize the field-level online decision-making independent of prediction. Firstly, Lyapunov optimization was used to construct the basic strategy to obtain a good but not necessarily local optimal strategy. Then, samples were pre-generated by the basic strategy to initialize the experience base and improve the search efficiency. Further, DDPG algorithm with expert mechanism was applied to train the locally optimal self-scheduling strategy. Case study shows that compared with the basic dispatch strategy and the classical DDPG, the proposed method can effectively improve the average revenue of the wind-storage power plant.
[1] 国家能源局. 2021年全国电力工业统计数据[EB/OL]. [2022-01-26]. http://www.nea.gov.cn/2022-01/26/c_1310441589.htm. [2] 国家能源局. 截至3月底全国发电装机容量约24亿千瓦,3月份可再生能源发电量较快增长.[EB/OL]. [2022-04-22]. http://www.nea.gov.cn/2022-04/22/c_1310569074.htm. [3] 姜书鹏, 乔颖, 徐飞, 等. 风储联合发电系统容量优化配置模型及敏感性分析[J]. 电力系统自动化, 2013, 37(20): 16-21. Jiang Shupeng, Qiao Ying, Xu Fei, et al.Capacity optimization and sensitivity analysis of cogeneration system of wind power and energy storage[J]. Automation of Electric Power Systems, 2013, 37(20): 16-21. [4] 陆秋瑜, 罗澍忻, 胡伟, 等. 集群风储联合系统广域协调控制及利益分配策略[J]. 电力系统自动化, 2019, 43(20): 183-191. Lu Qiuyu, Luo Shuxin, Hu Wei, et al.Wide-area coordinated control and benefit assignment strategy of clustering wind-energy storage integrated system[J]. Automation of Electric Power Systems, 2019, 43(20): 183-191. [5] 孙辉, 刘鑫, 贲驰, 等. 含风储一体化电站的电力系统多目标风险调度模型[J]. 电力系统自动化, 2018, 42(5): 94-101. Sun Hui, Liu Xin, Ben Chi, et al.Multi-objective risk scheduling model of power system containing power station with integrated wind power and energy storage[J]. Automation of Electric Power Systems, 2018, 42(5): 94-101. [6] 王佳丽. 探路风电预报[J]. 能源, 2014(3): 74-76. Wang Jiali.Pathfinder wind power forecast[J]. Energy, 2014(3): 74-76. [7] 姚子麟, 张亮, 邹斌, 等. 含高比例风电的电力市场电价预测[J]. 电力系统自动化, 2020, 44(12): 49-55. Yao Zilin, Zhang Liang, Zou Bin, et al.Electricity price prediction for electricity market with high proportion of wind power[J]. Automation of Electric Power Systems, 2020, 44(12): 49-55. [8] García C E, Prett D M, Morari M.Model predictive control: theory and practice—a survey[J]. Automatica, 1989, 25(3): 335-348. [9] Xie Le, Gu Yingzhong, Eskandari A, et al.Fast MPC-based coordination of wind power and battery energy storage systems[J]. Journal of Energy Engineering, 2012, 138(2): 43-53. [10] Lin Minghong, Liu Zhenhua, Wierman A, et al.Online algorithms for geographical load balancing[C]//2012 International Green Computing Conference (IGCC), San Jose, 2012: 1-10. [11] Wu Hongyu, Shahidehpour M, Li Zuyi, et al.Chance-constrained day-ahead scheduling in stochastic power system operation[J]. IEEE Transactions on Power Systems, 2014, 29(4): 1583-1591. [12] Ben-Tal A, Nemirovski A.Robust solutions of uncertain linear programs[J]. Operations Research Letters, 1999, 25(1): 1-13. [13] Garcia-Gonzalez J, de la Muela R M R, Santos L M, et al. Stochastic joint optimization of wind generation and pumped-storage units in an electricity market[J]. IEEE Transactions on Power Systems, 2008, 23(2): 460-468. [14] Lara J D, Olivares D E, Cañizares C A.Robust energy management of isolated microgrids[J]. IEEE Systems Journal, 2019, 13(1): 680-691. [15] Neely M J.Stochastic network optimization with application to communication and queueing systems[M]. Berlin: Springer, 2010. [16] Guo Zhongjie, Wei Wei, Chen Laijun, et al.Real-time self-dispatch of a remote wind-storage integrated power plant without predictions: explicit policy and performance guarantee[J]. IEEE Open Access Journal of Power and Energy, 2021, 8: 484-496. [17] 赵冬梅, 陶然, 马泰屹, 等. 基于多智能体深度确定策略梯度算法的有功-无功协调调度模型[J]. 电工技术学报, 2021, 36(9): 1914-1925. Zhao Dongmei, Tao Ran, Ma Taiyi, et al.Active and reactive power coordinated dispatching based on multi-agent deep deterministic policy gradient algorithm[J]. Transactions of China Electrotechnical Society, 2021, 36(9): 1914-1925. [18] 李涛, 胡维昊, 李坚, 等. 基于深度强化学习算法的光伏-抽蓄互补系统智能调度[J]. 电工技术学报, 2020, 35(13): 2757-2768. Li Tao, Hu Weihao, Li Jian, et al.Intelligent economic dispatch for PV-PHS integrated system: a deep reinforcement learning-based approach[J]. Transactions of China Electrotechnical Society, 2020, 35(13): 2757-2768. [19] 刁浩然, 杨明, 陈芳, 等. 基于强化学习理论的地区电网无功电压优化控制方法[J]. 电工技术学报, 2015, 30(12): 408-414. Diao Haoran, Yang Ming, Chen Fang, et al.Reactive power and voltage optimization control approach of the regional power grid based on reinforcement learning theory[J]. Transactions of China Electrotechnical Society, 2015, 30(12): 408-414. [20] 梁煜东, 陈峦, 张国洲, 等. 基于深度强化学习的多能互补发电系统负荷频率控制策略[J]. 电工技术学报, 2022, 37(7): 1768-1779. Liang Yudong, Chen Luan, Zhang Guozhou, et al.Load frequency control strategy of hybrid power generation system: a deep reinforcement learning—based approach[J]. Transactions of China Electrotechnical Society, 2022, 37(7): 1768-1779. [21] 于一潇, 杨佳峻, 杨明, 等. 基于深度强化学习的风电场储能系统预测决策一体化调度[J]. 电力系统自动化, 2021, 45(1): 132-140. Yu Yixiao, Yang Jiajun, Yang Ming, et al.Prediction and decision integrated scheduling of energy storage system in wind farm based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(1): 132-140. [22] Lillicrap T P, Hunt J J, Pritzel A, et al. Continuous control with deep reinforcement learning[EB/OL].2019: arXiv: 1509.02971. https://arxiv.org/abs/1509.02971 [23] 蔡新雷, 崔艳林, 董锴, 等. 基于改进K-means和MADDPG算法的风储联合系统日前优化调度方法[J]. 储能科学与技术, 2021, 10(6): 2200-2208. Cai Xinlei, Cui Yanlin, Dong Kai, et al.Day-ahead optimal scheduling approach of wind-storage joint system based on improved K-means and MADDPG algorithm[J]. Energy Storage Science and Technology, 2021, 10(6): 2200-2208. [24] 张淑兴, 马驰, 杨志学, 等. 基于深度确定性策略梯度算法的风光储系统联合调度策略[J/OL]. 中国电力, 2022: 1-9. [2022-07-03]. http://kns.cnki.net/kcms/detail/11.3265.TM.20211115.1426.002.html. Zhang Shuxing, Ma Chi, Yang Zhixue, et al. Joint dispatch of wind-photovoltaic-storage hybrid system based on deep deterministic policy gradient algorithm [J/OL]. Electric Power, 2022: 1-9. [2022-07-03]. http://kns.cnki.net/kcms/detail/11.3265.TM.20211115.1426.002.html. [25] Shen Ziqi, Wei Wei, Wu Danman, et al.Modeling arbitrage of an energy storage unit without binary variables[J]. CSEE Journal of Power and Energy Systems, 2021, 7(1): 156-161. [26] 束金龙, 闻人凯. 线性规划理论与模型应用[M]. 北京: 科学出版社, 2003. [27] Sigaud O, Buffet O.Markov decision processes in artificial intelligence[M]. New York: John Wiley & Sons, 2013.
2022, Vol. 37 
