Load Frequency Control Strategy of Hybrid Power Generation System: a Deep Reinforcement Learning—Based Approach
Liang Yudong1, Chen Luan1, Zhang Guozhou1, Ren Manman2, Hu Weihao1
1. Key Laboratory of Wide-area Mearsurement and Control on Power System of Sichuan Province University of Electronic Science and Technology of China Chengdu 611731 China; 2. State grid Anhui Electric Power Co. Ltd Electric Power Research Institute Hefei 230000 China
Abstract：To solve the problem of frequency modulation performance degradation caused by large-scale renewable energy access to the power grid, this paper proposes a data-driven load frequency coordinated optimization control method for hybrid energy system consisted of wind, thermal power and energy storage. Firstly, this paper establishes a mathematical model of the multi-area hybrid energy system through mechanism analysis. Secondly, a reward function with control performance standard (CPS), wind power casting and dynamic performance index is established. The load frequency control problem is transformed into a maximum reward function problem, and the deep deterministic policy gradient (DDPG) algorithm is introduced to solve this problem. Through pre-learning and online application, the optimal adaptive coordinated control strategy can be obtained under acturl output of wind turbine. Finally, the performance of the proposed method in improving the performance of load frequency control (LFC) is verified by adding continuous stepped disturbance and actual wind speed disturbance. Simulation results show that when the power system is disturbed, the introduction of energy storage equipment and the proposed method can not only suppress fluctuations effectively, but also shorten the adjustment time required by LFC and increase the proportion of wind power consumption.
梁煜东, 陈峦, 张国洲, 任曼曼, 胡维昊. 基于深度强化学习的多能互补发电系统负荷频率控制策略[J]. 电工技术学报, 2022, 37(7): 1768-1779.
Liang Yudong, Chen Luan, Zhang Guozhou, Ren Manman, Hu Weihao. Load Frequency Control Strategy of Hybrid Power Generation System: a Deep Reinforcement Learning—Based Approach. Transactions of China Electrotechnical Society, 2022, 37(7): 1768-1779.
[1] 彭思敏, 窦真兰, 凌志斌, 等. 并联型储能系统孤网运行协调控制策略[J]. 电工技术学报, 2013, 28(5): 128-134. Peng Simin, Dou Zhenlan, Ling Zhibin, et al.Cooperative control for parallel-connected battery energy storage system of islanded power system[J]. Transactions of China Electrotechnical Society, 2013, 28(5): 128-134. [2] 赵晶晶, 李敏, 何欣芹, 等. 基于限转矩控制的风储联合调频控制策略[J]. 电工技术学报, 2019, 34(23): 4982-4950. Zhao Jingjing, Li Min, He Xinqin, et al.Coordinated control strategy of wind power and energy storage in frequency regulation based on torque limit control[J]. Transactions of China Electrotechnical Society, 2019, 34(23): 4982-4950. [3] 陈文倩, 辛小南, 程志平. 基于虚拟同步发电机的光储并网发电控制技术[J]. 电工技术学报, 2018, 33(2): 538-545. Chen Wenqian, Xin Xiaonan, Cheng Zhiping.Control of grid-connected of photovoltaic system with storage based on virtual synchronous generator[J]. Transactions of China Electrotechnical Society, 2018, 33(2): 538-545. [4] 常烨骙, 李卫东, 巴宇, 等. 基于运行安全的频率控制性能评价新方法[J]. 电工技术学报, 2019, 34(6): 1218-1229. Chang Yekui, Li Weidong, Ba Yu, et al.A new method for frequency control performance assessment on operation security[J]. Transactions of China Electrotechnical Society, 2019, 34(6): 1218-1229. [5] 左剑, 谢平平, 李银红, 等. 基于智能优化算法的互联电网负荷频率控制器设计及其控制性能分析[J]. 电工技术学报, 2018, 33(3): 478-489. Zuo Jian, Xie Pingping, Li Yinhong, et al.Intelligent optimization algorithm based load frequency controller design and its control performance assessment in interconnected power grids[J]. Transactions of China Electrotechnical Society, 2018, 33(3): 478-489. [6] 单华, 和婧, 范立新, 等. 面向抽水蓄能电站区域负荷频率的分数阶PID控制研究[J]. 电网技术, 2020, 44(4): 1410-1418. Shan Hua, He Jing, Fan Lixin, et al.Research on fractional order PID control of regional load frequency of pumped storage power station[J]. Power System Technology, 2020, 44(4): 1410-1418. [7] Wang Haixin, Yang Junyou, Chen Zhe, et al.Model predictive control of PMSG-based wind turbines for frequency regulation in an isolated grid[J]. IEEE Trans actions on Industry Applications, 2018, 54(4): 3077-3089. [8] 程乐峰, 余涛, 张孝顺, 等. 机器学习在能源与电力系统领域的应用和展望[J]. 电力系统自动化, 2019, 43(1): 15-31. Cheng Lefeng, Yu Tao, Zhang Xiaoshun, et al.Application and prospects of machine learning in the field of energy and power systems[J]. Automation of Electirc Power Systems, 2019, 43(1): 15-31. [9] 余涛, 周斌, 陈家荣. 基于Q学习的互联电网动态最优CPS控制[J]. 中国电机工程学报, 2009, 29(19): 13-19. Yu Tao, Zhou Bin, Chen Jiarong.Q-learning-based dynamic optimal CPS control methodology for interconnected power systems[J]. Proceedings of the CSEE, 2009, 29(19): 13-19. [10] 余涛, 甄卫国, 叶文加, 等. 基于多步回溯Q学习的自动发电控制指令动态优化分配算法[J]. 控制理论与应用, 2011, 28(1): 58-64. Yu Tao, Zhen Weiguo, Ye Wenjia, et al.Multi-step backtrack Q-learning based dynamic optimal algorithm for auto generation control order dispatch[J]. Control Theory&Applications, 2011, 28(1): 58-64. [11] 张孝顺, 余涛, 唐捷. 基于CEQ(λ)多智能体协同学习的互联电网性能标准控制指令动态分配优化算法[J]. 电工技术学报, 2016, 31(8): 125-133. Zhang Xiaoshun, Yu Tao, Tang Jie.Dynamic optimal allocation algorithm for control performance standard order of interconnected power grids using synergetic learning of multi-agent CEQ(λ)[J]. Transactions of China Electrotechnical Society, 2016, 31(8):125-133. [12] 李涛, 胡维昊, 李坚, 等. 基于深度强化学习算法的光伏-抽蓄互补系统智能调度[J]. 电工技术学报, 2020, 35(13): 2757-2768. Li Tao, Hu Weihao, Li Jian, et al.Intelligent economic dispatch for PV-PHS integrated system: a deep reinforcement learning -based approach[J]. Transactions of China Electrotechnical Society, 2020, 35(13): 2757-2768. [13] Mocanu E, Mocanu D C, Nguyen P H, et al.On-line building energy optimization using deep reinfocement learning[J]. IEEE Transactions on Smart Grid, 2019, 10(4): 3698-3708. [14] 汪波, 郑文迪. 基于改进Q学习算法的储能系统实时优化决策研究[J]. 电气技术, 2018, 19(2): 54-60, 65. Wang Bo, Zheng Wendi.Research on real-time optimization decision of energy storage system based on improved Q-learning algorithm[J]. Electrical Engineering, 2018, 19(2): 54-60, 65. [15] 邹晓敏, 肖曦, 何琪, 等. 基于在线附加Q学习的伺服电机速度最优跟踪控制方法[J]. 电工技术学报, 2019, 34(5): 917-923. Zou Xiaomin, Xiao Xi, He Qi, et al.Optimal tracking control of servo motor speed based on online supplementary Q-learning[J]. Transactions of China Electrotechnical Society, 2019, 34(5): 917-923. [16] Yan Ziming, Xu Yan.Data-driven load frequency control for stochastic power systems: a deep reinforcement learning method with continuous action search[J]. IEEE Transactions on Power Systems, 2019, 34(2): 1653-1656. [17] Yan Ziming, Xu Yan.A multi-agent deep reinforcement learning method for cooperative load frequency control of a multi-area power system[J]. IEEE Transactions on Power Systems, 2020, 35(6): 4599-4608. [18] 吴云亮, 孙元章, 徐箭, 等. 基于多变量广义预测理论的互联电力系统负荷-频率协调控制体系[J]. 电工技术学报, 2012, 27(9): 101-107. Wu Yunliang, Sun Yuanzhang, Xu Jian, et al.Coordinated load-frequency control system in interconnected power system based on multivariable generalized predictive control theory[J]. Transactions of China Electrotechnical Society, 2012, 27(9): 101-107. [19] Wei Xu, Dong Hu, Gang Lei, et al.System-level efficiency optimization of a linear induction motor drive system[J]. CES Transactions on Electrical Machines and Systems, 2019, 3(3): 285-291. [20] 张冠锋, 杨俊友, 孙峰, 等. 基于虚拟惯量和频率下垂控制的双馈风电机组一次调频策略[J]. 电工技术学报, 2017, 32(22): 225-232. Zhang Guanfeng, Yang Junyou, Sun Feng, et al.Primary frequency regulation strategy of DFIG based on virtual inertia and frequency droop control[J]. Transactions of China Electrotechnical Society, 2017, 32(22): 225-232. [21] 章艳, 高晗, 张萌. 不同虚拟同步机控制下双馈风机系统频率响应差异研究[J]. 电工技术学报, 2020, 35(13): 2889-2900. Zhang Yan, Gao Han, Zhang Meng.Research on frequency response difference of doubly-fed induction generator system controlled by different virtual synchronous generator controls[J]. Transactions of China Electrotechnical Society, 2020, 35(13): 2889-2900. [22] 颜湘武, 崔森, 常文斐. 考虑储能自适应调节的双馈感应发电机一次调频控制策略[J]. 电工技术学报, 2021, 36(5): 1027-1039. Yan Xiangwu, Cui Sen, Chang Wenfei.Primary frequency regulation control strategy of doubly-fed induction generator considering supercapacitor SOC feedback adaptive adjustment[J]. Transactions of China Electrotechnical Society, 2021, 36(5): 1027-1039. [23] 余涛, 王宇名, 刘前进, 等. 互联电网CPS调节指令动态最有分配Q-学习算法[J]. 中国电机工程学报, 2010, 30(7): 62-69. Yu Tao, Wang Yuming, Liu Qianjin, et al.Q-learning-based dynamic optimal allocation algorithm for CPS order of interconnected power grids[J]. Proceedings of the CSEE, 2010, 30(7): 62-69. [24] 席磊, 余涛, 张孝顺, 等. 基于狼爬山快速多智能体学习策略的电力系统智能发电控制方法[J]. 电工技术学报, 2015, 30(23): 93-101. Xi Lei, Yu Tao, Zhang Xiaoshun, et al.A fast multi-agent learning strategy based on DWoLF-PHC(λ) for smart generation control of power systems[J]. Transactions of China Electrotechnical Society, 2015, 30(23): 93-101. [25] Elgerd O I, Fosha C E. Optimum megawatt-frequency control of multi-area electric energy systems[J]. IEEE Transactions on Power Apparatus and Systems, 1970, PAS-89(4): 556-563. [26] Zhang Guozhou, Hu Weihao, Cao Di, et al.Data-driven optimal energy management for a wind-solar-diesel-battery-reverse osmosis hybrid energy system using a deep reinforcement learning approach[J]. Energy Conversion and Management, 2021, 227: 113608.
2022, Vol. 37 
