Dual Timescales Coordinated and Optimal Voltages Control in Distribution Systems using Data-Driven and Physical Optimization
Zhang Jian1, Cui Mingjian2, He Yigang3
1. School of Electrical and Automation Engineering Hefei University of Technology Hefei 230009 China; 2. School of Electrical and Information Engineering Tianjin University Tianjin 300072 China; 3. School of Electrical and Automation Wuhan University Wuhan 430072 China
Abstract：A large number of electric vehicles (EVs), distributed solar and/or wind turbine generators (WTGs) connected to distribution systems lead to frequent and sharp voltages fluctuations. The action rates of conventional adjustable devices and smart inverters are very different. In this context, a novel dual-timescale voltage control scheme is proposed by organically combining data-driven with physics-based optimization. On fast timescale, a quadratic programming (QP) for balanced and unbalanced distribution systems is developed based on branch flow equations. The optimal reactive power of renewable distributed generators (DGs) and static VAR compensators (SVCs) is configured on several minutes. Whereas, on slow timescale, a data-driven Markovian decision process (MDP) is developed, in which the charge/discharge power of energy storage systems (ESSs), statuses/ratios of switchable capacitors reactors (SCRs), and voltage regulators (VRs) are configured hourly to minimize long-term discounted squared voltages magnitudes deviations using an adapted deep deterministic policy gradient (DDPG) deep reinforcement learning (DRL) algorithm. The capabilities of the proposed method are validated with IEEE 33-bus balanced and 123-bus unbalanced distribution systems. The contributions of this paper are summarized as follows: (1)Combining data-driven with physics-based methods, a strategy for coordinated control of five different types of adjustable equipment, namely VRs, SCRs, ESSs, SVCs and DGs inverters on fast and slow timescales is proposed. (2) A slow timescale (say 1 hour) MDP for active and reactive power coordination is constructed. The (near) optimal settings of ratios/statuses of VRs, SCRs and charge/discharge power of ESSs are found using DRL algorithm. As a result, the deficiency of low computing rate for conventional physics -based large scale mixed integer non-convex nonlinear stochastic programming is completely overcome. (3) The charge/discharge power of ESS is continuous variable while ratios/statues of VRs and SCRs are discrete decisions. DDPG algorithm cannot be directly applicable to discrete action while DQN algorithm cannot be applicable to continuous action. Further, when there are a large number of VRs and SCRs in distribution network, DQN algorithm leads to dimensionality curses. The existing DRL algorithm in literatures cannot deal with joint continuous-discrete action (efficiently). To eliminate dimensionality curses in joint continuous- discrete action space, ratios/statuses of VRs and SCRs are firstly relaxed to continuous variables. Then, for the proto action given by actor of DDPG agent, Knn nearest neighbors are found out in the joint continuous-discrete action space. Finally, each of the Knn actions is transferred to the critic of DDPG agent one by one to evaluate its value. The action with the greatest value is chosen to interact with the distribution network. (4) Given the (near) optimal solution of the slow timescale MDP, a QP for VVO with DGs, SVCs inverters settings on fast timescale (say several minutes or seconds) to minimize squared voltages magnitudes deviations are developed for balanced and unbalanced distribution systems. As a result, voltage violations on fast timescale caused by sizable, rapid and frequent power fluctuations from renewable DGs and fast charged EVs can be mitigated in real time. (5) One of the outstanding advantages of the proposed method is very easy to perform in practice with near optimal solution. Further, when Knn =20 or Knn =40, the proposed method has much more stable training process than multi-agent DQN algorithm and much higher computing rate than conventional multi-slot single fast timescale mixed integer QP by about 18.0~36.7 times.
张剑, 崔明建, 何怡刚. 结合数据驱动与物理模型的主动配电网双时间尺度电压协调优化控制[J]. 电工技术学报, 2024, 39(5): 1327-1339.
Zhang Jian, Cui Mingjian, He Yigang. Dual Timescales Coordinated and Optimal Voltages Control in Distribution Systems using Data-Driven and Physical Optimization. Transactions of China Electrotechnical Society, 2024, 39(5): 1327-1339.
[1] Yong Jiaying, Ramachandaramurthy V K, Tan Kangmiao, et al.Bi-directional electric vehicle fast charging station with novel reactive power compensation for voltage regulation[J]. International Journal of Electrical Power & Energy Systems, 2015, 64: 300-310. [2] Wang Pengfei, Liang D H, Yi Jialiang, et al.Integrating electrical energy storage into coordinated voltage control schemes for distribution networks[J]. IEEE Transactions on Smart Grid, 2014, 5(2): 1018-1032. [3] Ranamuka D, Agalgaonkar A P, Muttaqi K M.Examining the interactions between DG units and voltage regulating devices for effective voltage control in distribution systems[J]. IEEE Transactions on Industry Applications, 2017, 53(2): 1485-1496. [4] Agalgaonkar Y P, Pal B C, Jabr R A.Distribution voltage control considering the impact of PV generation on tap changers and autonomous regulators[J]. IEEE Transactions on Power Systems, 2014, 29(1): 182-192. [5] Kawabe K, Tanaka K.Impact of dynamic behavior of photovoltaic power generation systems on short-term voltage stability[J]. IEEE Transactions on Power Systems, 2015, 30(6): 3416-3424. [6] Taranto G N, Assis T M L, Falcao D M, et al. Highlighting the importance of chronology on voltage protection and control in active distribution networks[J]. IEEE Transactions on Power Delivery, 2017, 32(1): 361-369. [7] Bose S, Low S H, Teeraratkul T, et al.Equivalent relaxations of optimal power flow[J]. IEEE Transactions on Automatic Control, 2015, 60(3): 729-742. [8] Valverde G, Van Cutsem T.Model predictive control of voltages in active distribution networks[J]. IEEE Transactions on Smart Grid, 2013, 4(4): 2152-2161. [9] 黄大为, 王孝泉, 于娜, 等. 计及光伏出力不确定性的配电网混合时间尺度无功/电压控制策略[J]. 电工技术学报, 2022, 37(17): 4377-4389. Huang Dawei, Wang Xiaoquan, Yu Na, et al.Hybrid timescale voltage/var control in distribution network considering PV power uncertainty[J]. Transactions of China Electrotechnical Society, 2022, 37(17): 4377-4389. [10] Magnússon S, Qu Guannan, Fischione C, et al.Voltage control using limited communication[J]. IEEE Transactions on Control of Network Systems, 2019, 6(3): 993-1003. [11] Bolognani S, Carli R, Cavraro G, et al.Distributed reactive power feedback control for voltage regulation and loss minimization[J]. IEEE Transactions on Automatic Control, 2015, 60(4): 966-981. [12] 杨珺, 侯俊浩, 刘亚威, 等. 分布式协同控制方法及在电力系统中的应用综述[J]. 电工技术学报, 2021, 36(19): 4035-4049. Yang Jun, Hou Junhao, Liu Yawei, et al.Distributed cooperative control method and application in power system[J]. Transactions of China Electrotechnical Society, 2021, 36(19): 4035-4049. [13] 李忠文, 程志平, 张书源, 等. 考虑经济调度及电压恢复的直流微电网分布式二次控制[J]. 电工技术学报, 2021, 36(21): 4482-4492. Li Zhongwen, Cheng Zhiping, Zhang Shuyuan, et al.Distributed secondary control for economic dispatch and voltage restoration of DC microgrid[J]. Transactions of China Electrotechnical Society, 2021, 36(21): 4482-4492. [14] Zhang Liang, Wang Gang, Giannakis G B.Real-time power system state estimation and forecasting via deep unrolled neural networks[J]. IEEE Transactions on Signal Processing, 2019, 67(15): 4069-4077. [15] 巨云涛, 陈希, 李嘉伟, 等. 基于分布式深度强化学习的微网群有功无功协调优化调度[J]. 电力系统自动化, 2023, 47(1): 115-125. Ju Yuntao, Chen Xi, Li Jiawei, et al.Active and reactive power coordinated optimal dispatch of networked microgrids based on distributed deep reinforcement learning[J]. Automation of Electric Power Systems, 2023, 47(1): 115-125. [16] 顾雪平, 刘彤, 李少岩, 等. 基于改进双延迟深度确定性策略梯度算法的电网有功安全校正控制[J]. 电工技术学报, 2023, 38(8): 2162-2177. Gu Xueping, Liu Tong, Li Shaoyan, et al.Active power correction control of power grid based on improved twin delayed deep deterministic policy gradient algorithm[J]. Transactions of China Electrotechnical Society, 2023, 38(8): 2162-2177. [17] 陈泽宇, 方志远, 杨瑞鑫, 等. 基于深度强化学习的混合动力汽车能量管理策略[J]. 电工技术学报, 2022, 37(23): 6157-6168. Chen Zeyu, Fang Zhiyuan, Yang Ruixin, et al.Energy management strategy for hybrid electric vehicle based on the deep reinforcement learning method[J]. Transactions of China Electrotechnical Society, 2022, 37(23): 6157-6168. [18] 陈明昊, 孙毅, 谢志远. 基于双层深度强化学习的园区综合能源系统多时间尺度优化管理[J]. 电工技术学报, 2023, 38(7): 1864-1881. Chen Minghao, Sun Yi, Xie Zhiyuan.The multi-time-scale management optimization method for park integrated energy system based on the Bi-layer deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2023, 38(7): 1864-1881. [19] 宋煜浩, 魏韡, 黄少伟, 等. 风储联合电站实时自调度的高效深度确定性策略梯度算法[J]. 电工技术学报, 2022, 37(23): 5987-5999. Song Yuhao, Wei Wei, Huang Shaowei, et al.Efficient deep deterministic policy gradient algorithm for real-time self-dispatch of wind-storage power plant[J]. Transactions of China Electrotechnical Society, 2022, 37(23): 5987-5999. [20] Xu Hanchen, Domínguez-García A D, Sauer P W. Optimal tap setting of voltage regulation transformers using batch reinforcement learning[J]. IEEE Transactions on Power Systems, 2020, 35(3): 1990-2001. [21] Al-Saffar M, Musilek P.Reinforcement learning-based distributed BESS management for mitigating overvoltage issues in systems with high PV penetration[J]. IEEE Transactions on Smart Grid, 2020, 11(4): 2980-2994. [22] Wang Wei, Yu Nanpeng, Gao Yuanqi, et al.Safe off-policy deep reinforcement learning algorithm for volt-VAR control in power distribution systems[J]. IEEE Transactions on Smart Grid, 2020, 11(4): 3008-3018. [23] Cao Di, Hu Weihao, Zhao Junbo, et al.A multi-agent deep reinforcement learning based voltage regulation using coordinated PV inverters[J]. IEEE Transactions on Power Systems, 2020, 35(5): 4120-4123. [24] Yang Qiuling, Wang Gang, Sadeghi A, et al.Two-timescale voltage control in distribution grids using deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(3): 2313-2323. [25] Zhang Ying, Wang Xinan, Wang Jianhui, et al.Deep reinforcement learning based volt-VAR optimization in smart distribution systems[J]. IEEE Transactions on Smart Grid, 2021, 12(1): 361-371. [26] Dulac-Arnold G, Evans R, Van Hasselt H, et al.Deep reinforcement learning in large discrete action spaces[J]. Computer Science, 2015. DOI:10.48550/ ARXiv.1512.07679. [27] Zhang Jian, He Yigang.Fast solving method for two-stage multi-period robust optimization of active and reactive power coordination in active distribution networks[J]. IEEE Access, 2022, 11: 30208-30222. [28] Ding Tao, Liu Shiyu, Yuan Wei, et al.A two-stage robust reactive power optimization considering uncertain wind power integration in active distribution networks[J]. IEEE Transactions on Sustainable Energy, 2016, 7(1): 301-311.
2024, Vol. 39 
