Active and Reactive Power Coordinated Dispatching Based on Multi-Agent Deep Deterministic Policy Gradient Algorithm
Zhao Dongmei1, Tao Ran1, Ma Taiyi1, Xia Xuan2, Wang Haoxiang1
1. School of Electrical and Electronic Engineering North China Electric Power University Beijing 102206 China; 2. State Grid Shaoxing Power Supply Company Shaoxing 312000 China
Abstract：Achieving active and reactive power coordination dispatching is a key link in promoting construction of "future integrated large scale power grid control system". In order to solve the problems of repeated regulation and difficult to coordinate conflicts in dispatching, multi-agent technology is adopted to intelligently organize various active and reactive power control resources, and establish a power grid active and reactive power coordination dispatching model. In order to solve the instability of the power system environment in the process of multi-agent exploration, adopt multi-agent deep deterministic policy gradient algorithm, and design a multi-agent environment which is suitable for active and reactive power coordination dispatching model, and constructs the agent's state, action and reward function. The effectiveness of the proposed model and algorithm is verified by case study and comparative analysis.
赵冬梅, 陶然, 马泰屹, 夏轩, 王浩翔. 基于多智能体深度确定策略梯度算法的有功-无功协调调度模型[J]. 电工技术学报, 2021, 36(9): 1914-1925.
Zhao Dongmei, Tao Ran, Ma Taiyi, Xia Xuan, Wang Haoxiang. Active and Reactive Power Coordinated Dispatching Based on Multi-Agent Deep Deterministic Policy Gradient Algorithm. Transactions of China Electrotechnical Society, 2021, 36(9): 1914-1925.
[1] 许洪强, 姚建国, 南贵林, 等. 未来电网调度控制系统应用功能的新特征[J]. 电力系统自动化, 2018, 42(1): 1-7. Xu Hongqiang, Yao Jianguo, Nan Guilin, et al.New features of application function for future dispatching and control systems[J]. Automation of Electric Power Systems, 2018, 42(1): 1-7. [2] 郭建成, 南贵林, 许丹, 等. 大电网全局监控内涵与关键技术[J]. 电力系统自动化, 2018, 42(18): 1-8. Guo Jiancheng, Nan Guilin, Xu Dan, et al.Connotation and key technology of global monitoring for large power grid[J]. Automation of Electric Power Systems, 2018, 42(18): 1-8. [3] 许洪强, 姚建国, 於益军, 等. 支撑一体化大电网的调度控制系统架构及关键技术[J]. 电力系统自动化, 2018, 42(6): 1-8. Xu Hongqiang, Yao Jianguo, Yu Yijun, et al.Architecture and key technologies of dispatch and control system supporting integrated bulk power grids[J]. Automation of Electric Power Systems, 2018, 42(6): 1-8. [4] 刘一兵, 吴文传, 张伯明, 等. 基于混合整数二阶锥规划的主动配电网有功-无功协调多时段优化运行[J]. 中国电机工程学报, 2014, 34(16): 2575-2583. Liu Yibing, Wu Wenchuan, Zhang Boming, et al.A mixed integer second order cone programming based active and reactive power coordinated multi-period optimization for active distribution network[J]. Proceedings of the CSEE, 2014, 34(16): 2575-2583. [5] 任佳依. 有源配电系统有功无功协调优化研究[D]. 南京: 东南大学, 2017. [6] 何婷. 主动配电网有功-无功电源的综合优化配置研究[D]. 广州: 华南理工大学, 2018. [7] 陆文甜. 含连续/离散控制的多区域电力系统分布式优化调度方法研究[D]. 广州: 华南理工大学, 2018. [8] 颜湘武, 徐韵. 考虑网络动态重构含多异质可再生分布式电源参与调控的配电网多时空尺度无功优化[J]. 电工技术学报, 2019, 34(20): 4358-4372. Yan Xiangwu, Xu Yun.Multiple time and space scale reactive power optimization for distribution network with multi-heterogeneous RDG participating in regulation and considering network dynamic reconfiguration[J]. Transactions of China Electrotechnical Society, 2019, 34(20): 4358-4372. [9] 颜湘武, 徐韵, 李若瑾, 等. 基于模型预测控制含可再生分布式电源参与调控的配电网多时间尺度无功动态优化[J]. 电工技术学报, 2019, 34(10): 2022-2037. Yan Xiangwu, Xu Yun, Li Ruojin, et al.Multi-time scale reactive power optimization of distribution grid based on model predictive control and including RDG regulation[J]. Transactions of China Electrotechnical Society, 2019, 34(10): 2022-2037. [10] 乐健, 王曹, 李星锐, 等. 中压配电网多目标分布式优化控制策略[J]. 电工技术学报, 2019, 34(23): 4972-4981. Le Jian, Wang Cao, Li Xingrui, et al.The multi-object distributed optimization control strategy of medium voltage distribution networks[J]. Transactions of China Electrotechnical Society, 2019, 34(23): 4972-4981. [11] 颜湘武, 徐韵. 考虑网络动态重构含多异质可再生分布式电源参与调控的配电网多时空尺度无功优化[J]. 电工技术学报, 2019, 34(20): 4358-4372. Yan Xiangwu, Xu Yun.Multiple time and space scale reactive power optimization for distribution network with multi-heterogeneous RDG participating in regulation and considering network dynamic reconfiguration[J]. Transactions of China Electrotechnical Society, 2019, 34(20): 4358-4372. [12] 石宪, 薛毓强, 曾静岚. 基于有功-无功控制的光伏并网点电压调节方案[J]. 电气技术, 2019, 20(3): 50-56. Shi Xian, Xue Yuqiang, Zeng Jinglan.Voltage regulation strategies based on power control for grid-connected photovoltaic at point of common coupling[J]. Electrical Engineering, 2019, 20(3): 50-56. [13] International Energy Agency.Empowering variable renewables‐options for flexible electricity systems:(complete edition)[J]. International Energy, 2009(23): 31-36. [14] 张孝顺. 电力系统的迁移强化学习优化算法研究[D]. 广州: 华南理工大学, 2017. [15] 张孝顺, 余涛. 互联电网AGC功率动态分配的虚拟发电部落协同一致性算法[J]. 中国电机工程学报, 2015, 35(15): 3750-3759. Zhang Xiaoshun, Yu Tao.Virtual generation tribe based collaborative consensus algorithm for dynamic generation dispatch of AGC in interconnected power grids[J]. Proceedings of the CSEE, 2015, 35(15): 3750-3759. [16] 刁浩然, 杨明, 陈芳, 等. 基于强化学习理论的地区电网无功电压优化控制方法[J]. 电工技术学报, 2015, 30(12): 408-414. Diao Haoran, Yang Ming, Chen Fang, et al.Reactive power and voltage optimization control approach of the regional power grid based on reinforcement learning theory[J]. Transactions of China Electrotechnical Society, 2015, 30(12): 408-414. [17] 余涛, 刘靖, 胡细兵. 基于分布式多步回溯Q(λ)学习的复杂电网最优潮流算法[J]. 电工技术学报, 2012, 27(4): 185-192. Yu Tao, Liu Jing, Hu Xibing.Optimal power flow for complex power grid using distributed multi-step backtrack Q(λ) learning[J]. Transactions of China Electrotechnical Society, 2012, 27(4): 185-192. [18] 张孝顺, 余涛, 唐捷. 基于CEQ(λ)多智能体协同学习的互联电网性能标准控制指令动态分配优化算法[J]. 电工技术学报, 2016, 31(8): 125-133. Zhang Xiaoshun, Yu Tao, Tang Jie.Dynamic optimal allocation algorithm for control performance standard order of interconnected power grids using synergetic learning of multi-agent CEQ(λ)[J]. Transactions of China Electrotechnical Society, 2016, 31(8): 125-133. [19] 李宏仲, 王磊, 林冬, 等. 多主体参与可再生能源消纳的Nash博弈模型及其迁移强化学习求解[J]. 中国电机工程学报, 2019, 39(14): 4135-4150. Li Hongzhong, Wang Lei, Lin Dong, et al.A nash game model of multi-agent participation in renewable energy consumption and the solving method via transfer reinforcement learning[J]. Proceedings of the CSEE, 2019, 39(14): 4135-4150. [20] 席磊, 余璐, 付一木, 等. 基于探索感知思维深度强化学习的自动发电控制[J]. 中国电机工程学报, 2019, 39(14): 4150-4162. Xi Lei, Yu Lu, Fu Yimu, et al.Automatic generation control based on deep reinforcement learning with exploration awareness[J]. Proceedings of the CSEE, 2019, 39(14): 4150-4162. [21] 王怀智, 余涛, 唐捷. 基于多智能体相关均衡算法的自动发电控制[J]. 中国电机工程学报, 2014, 34(4): 620-627. Wang Huaizhi, Yu Tao, Tang Jie.Automatic generation control for interconnected power grids based on multi-agent correlated equilibrium learning system[J]. Proceedings of the CSEE, 2014, 34(4): 620-627. [22] Kouveliotis-lysikatos I N, Koukoula D I, Hatziargyriou N D. A double-layered fully distributed voltage control method for active distribution networks[J]. IEEE Transactions on Smart Grid, 2019, 10(2): 1465-1476. [23] 梁琳. 电力市场环境下火电机组有偿与无偿调峰划分方法研究[D]. 北京: 华北电力大学, 2009. [24] 薛晨, 任景, 张小东, 等. 含虚拟储能的新能源高渗透电网深度调峰备用决策模型[J]. 中国电力, 2019, 52(11): 35-43. Xue Chen, Ren Jing, Zhang Xiaodong, et al.A reserve decision model forhigh-proportional renew energy integrated power grid based on deep peak-shaving and virtual storage[J]. Electric Power, 2019, 52(11): 35-43. [25] 邢振中, 冷杰, 张永兴, 等. 火力发电机组深度调峰研究[J]. 东北电力技术, 2014, 35(4): 18-23. Xing Zhenzhong, Leng Jie, Zhang Yongxing, et al.Research on depth peak load cycling of thermal power generator units[J]. Northeast Electric Power Technology, 2014, 35(4): 18-23. [26] 郭庆来, 孙宏斌, 张伯明, 等. 基于无功源控制空间聚类分析的无功电压分区[J]. 电力系统自动化, 2005, 29(10): 36-40. Guo Qinglai, Sun Hongbin, Zhang Boming, et al.Power network partitioning based on clustering analysis in Mvar control space[J]. Automation of Electric Power Systems, 2005, 29(10): 36-40. [27] Peters J, Bagnell J A.Policy gradient methods[J]. Encyclopedia of Machine Learning, 2010, 5(11): 774-776. [28] Kim B, Park J, Park S, et al.Impedance learning for robotic contact tasks using natural actor-critic algorithm[J]. IEEE Transactions on Systems, Man, and Cybernetics, 2010, 40(2): 433-443. [29] 陈启鑫, 康重庆, 夏清, 等. 低碳电力调度方式及其决策模型[J]. 电力系统自动化, 2010, 34(12): 18-23. Chen Qixin, Kang Chongqing, Xia Qing, et al.Mechanism and modelling approach to low-carbon power dispatch[J]. Automation of Electric Power Systems, 2010, 34(12): 18-23. [30] Lillicrap T P, Hunt J J, Pritzel A, et al.Continuous control with deep reinforcement learning[J]. Computer Science, 2015, 8(6): A187. [31] Duryea E, Ganger M, Hu W.Exploring deep reinforcement learning with multi Q-learning[J]. Intelligent Control and Automation, 2016, 7(4): 129-144.
2021, Vol. 36 
