基于深度强化学习的有源配电网多时间尺度源荷储协同优化调控

doi:10.19595/j.cnki.1000-6753.tces.240316

摘要
图/表
参考文献
相关文章 (14)

全文: PDF (1037 KB)
输出: BibTeX | EndNote (RIS)

摘要

构建以清洁能源为主体的新型电力系统是实现“双碳”目标的重要举措,配电网源荷储协同是促进高比例风光能源消纳的有力措施。基于数据驱动的人工智能方法具有无模型、自适应等特点,可以自主学习风光能源及负荷的复杂不确定性,对有源配电网优化调控具有良好的支撑作用。该文考虑源荷功率预测精度特点和设备运行调控特性,提出基于深度强化学习算法的有源配电网多时间尺度智能优化调控方法。其中,日前阶段制定储能系统和柔性负荷的调控计划,以实现配电网的经济运行,减小对上级电网造成的调峰压力,并针对多节点多时段状态空间设计了相应的特征提取方法;日内阶段将优化调度问题转换为马尔科夫决策过程,设计了表征联络线功率波动平抑和灵活性资源日前计划跟踪效果的奖励函数,实现了对全调控时段内平抑功率波动及跟踪日前计划效果的统筹优化。最后通过修改后的IEEE-33算例系统验证了所提方法的有效性与优越性。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	李鹏
	钟瀚明
	马红伟
	李建锋
	刘洋
	王加浩

关键词 ：有源配电网, 优化调控, 源荷储协同, 深度强化学习

Abstract：

The stochastic, volatile, and uncontrollable nature of renewable energy generation such as wind and solar leads to increased uncertainty and difficulty in regulation with the expansion of installed capacity. Consequently, the safe, economical, and stable operation of distribution networks faces formidable challenges. The data-driven artificial intelligence method has the characteristics of model-free and self-adaptation. It can independently learn the complex uncertainty of wind and solar energy and load, and has a good supporting role for the optimal regulation of active distribution network. Consider the problem of high dimensional variable information redundancy in day-ahead scheduling and the dilemma of stabilizing fluctuating power and tracking day ahead scheduling in intraday scheduling, this paper proposes a multi-timescale intelligent optimal regulation method for active distribution network based on deep reinforcement learning algorithm.
Firstly, a multi time scale framework for active distribution networks (AND) was constructed based on the operational characteristics and regulatory requirements of the equipment, and the corresponding state inputs and action outputs of deep reinforcement learning (DRL) agents at different time scales were described. In the day-ahead stage, this paper takes the lowest daily operation cost of ADN and the smoothest tie line power as the optimization objective. In order to reduce the influence of high-dimensional variable information redundancy in the day ahead state space on the optimization effect, an improved feature extraction method considering the spatio-temporal coupling characteristics of the day-ahead state information is proposed. In the intraday stage, considering the full absorption of uncertain power will cause deviations from the pre-scheduled operating status of controllable equipment as energy storage system (ESS), this paper designs a reward function that reasonably reflects the tie line power deviation and state of charge (SOC) deviation of ESS, and realizes the optimization in the whole scheduling period through the deep reinforcement learning method.
Simulation results show that agents on both time scales can converge effectively. In the day ahead stage, while controlling the daily comprehensive operation cost of ADN, the peak value on the tie line is smoothed. The active power loss and the mean square deviation of tie line power is reduced. Compared with using fully connected network and n×n convolutional network as feature extraction network, the proposed feature extraction method has improved the convergence and convergence speed of DRL agent. In the intraday stage, the proposed method realizes the adaptive stabilization of fluctuating power while tracking the day ahead plan of scheduling equipment. The optimization effect of the proposed method in the whole scheduling period is better than that of the full consumption and single-step optimization.
The following conclusions can be drawn from the simulation analysis: (1) Using the data-driven deep reinforcement learning method, the probability information of uncertainty in the data set can be learned autonomously. The feature extraction network designed according to the spatio-temporal coupling characteristics of multi period state space variables can effectively improve the optimization effect and convergence of the deep learning method. (2) Without assuming the probability distribution of source and load uncertainty, the day-ahead dispatch realizes the multi-objective optimization of the ADN with the lowest daily operation cost and the smoothest tie line power. (3) The intraday dispatch takes into account the offset between the actual power and the day ahead forecast power of the source and load, and realizes the real-time stabilization of the fluctuation of uncertain power while tracking the day-ahead plan.

Key words： Active distribution network optimal dispatch source-load-storage coordination deep reinforcement learning

收稿日期: 2024-02-29

PACS:

TM73

基金资助:

国家电网公司科技项目资助（4000-202316071A-1-1-ZN）

通讯作者: 钟瀚明男,1999年生,硕士研究生,研究方向为人工智能在电网优化运行方面的应用。E-mail：ncepuzhm@ncepu.edu.cn

作者简介: 李鹏男,1965年生,教授,博士生导师,研究方向为新能源电力系统、综合能源系统与微网、储能技术应用、人工智能在智慧能源中的应用等。E-mail：ncepulp@ncepu.edu.cn

引用本文:

李鹏, 钟瀚明, 马红伟, 李建锋, 刘洋, 王加浩. 基于深度强化学习的有源配电网多时间尺度源荷储协同优化调控[J]. 电工技术学报, 0, (): 2492921-2492921. Li Peng, Zhong Hanming, Ma Hongwei, Li Jianfeng, Liu Yang, Wang Jiahao. Multi-Timescale Optimal Dispatch of Source-Load-Storage Coordination in Active Distribution Network Based on Deep Reinforcement Learning. Transactions of China Electrotechnical Society, 0, (): 2492921-2492921.

链接本文:

https://dgjsxb.ces-transaction.com/CN/10.19595/j.cnki.1000-6753.tces.240316 https://dgjsxb.ces-transaction.com/CN/Y0/V/I/2492921

[1] 张智刚, 康重庆. 碳中和目标下构建新型电力系统的挑战与展望[J]. 中国电机工程学报, 2022, 42(8): 2806-2819.
Zhang Zhigang, Kang Chongqing.Challenges and prospects for constructing the new-type power system towards a carbon neutrality future[J]. Proceedings of the CSEE, 2022, 42(8): 2806-2819.
[2] 辛保安, 单葆国, 李琼慧, 等. “双碳” 目标下 “能源三要素” 再思考[J]. 中国电机工程学报, 2022, 42(9): 3117-3126.
Xin Baoan, Shan Baoguo, Li Qionghui, et al.Rethinking of the \ “three elements of energy\” toward carbon peak and carbon neutrality[J]. Proceedings of the CSEE, 2022, 42(9): 3117-3126.
[3] Boicea V A.Energy storage technologies: the past and the present[J]. Proceedings of the IEEE, 2014, 102(11): 1777-1794.
[4] 王珂, 姚建国, 姚良忠, 等. 电力柔性负荷调度研究综述[J]. 电力系统自动化, 2014, 38(20): 127-135.
Wang Ke, Yao Jianguo, Yao Liangzhong, et al.Survey of research on flexible loads scheduling technologies[J]. Automation of Electric Power Systems, 2014, 38(20): 127-135.
[5] Gill S, Kockar I, Ault G W.Dynamic optimal power flow for active distribution networks[J]. IEEE Transactions on Power Systems, 2014, 29(1): 121-131.
[6] 尤毅, 刘东, 钟清, 等. 主动配电网优化调度策略研究[J]. 电力系统自动化, 2014, 38(9): 177-183.
You Yi, Liu Dong, Zhong Qing, et al.Research on optimal schedule strategy for active distribution network[J]. Automation of Electric Power Systems, 2014, 38(9): 177-183.
[7] Chen Xiaoshuang, Lin Jin, Liu Feng, et al.Optimal control of DERs in ADN under spatial and temporal correlated uncertainties[J]. IEEE Transactions on Smart Grid, 2020, 11(2): 1216-1228.
[8] 金国彬, 潘狄, 陈庆, 等. 考虑源荷不确定性的直流配电网模糊随机日前优化调度[J]. 电工技术学报, 2021, 36(21): 4517-4528.
Jin Guobin, Pan Di, Chen Qing, et al.Fuzzy random day-ahead optimal dispatch of DC distribution network considering the uncertainty of source-load[J]. Transactions of China Electrotechnical Society, 2021, 36(21): 4517-4528.
[9] 徐崇博, 杨晓东, 张有兵, 等. 考虑风险管控的含智能软开关主动配电网随机运行优化方法[J]. 电力系统自动化, 2021, 45(11): 68-76.
Xu Chongbo, Yang Xiaodong, Zhang Youbing, et al.Stochastic operation optimization method for active distribution networks with soft open point considering risk management and control[J]. Automation of Electric Power Systems, 2021, 45(11): 68-76.
[10] 李鹏, 王加浩, 黎灿兵, 等. 计及源荷不确定性与设备变工况特性的园区综合能源系统协同优化运行方法[J]. 中国电机工程学报, 2023, 43(20): 7802-7812.
Li Peng, Wang Jiahao, Li Canbing, et al.Collaborative optimal scheduling of the community integrated energy system considering source-load uncertainty and equipment off-design performance[J]. Proceedings of the CSEE, 2023, 43(20): 7802-7812.
[11] 梁俊文, 林舜江, 刘明波, 等. 主动配电网分布式鲁棒优化调度方法[J]. 电网技术, 2019, 43(4): 1336-1344.
Liang Junwen, Lin Shunjiang, Liu Mingbo, et al.Distributed robust optimal dispatch in active distribution networks[J]. Power System Technology, 2019, 43(4): 1336-1344.
[12] 李鹏, 姜磊, 王加浩, 等. 基于深度强化学习的新能源配电网双时间尺度无功电压优化[J]. 中国电机工程学报, 2023, 43(16): 6255-6266.
Li Peng, Jiang Lei, Wang Jiahao, et al.Optimization of dual-time scale reactive voltage for distribution network with renewable energy based on deep reinforcement learning[J]. Proceedings of the CSEE, 2023, 43(16): 6255-6266.
[13] Li Yunyi, Wan Can, Chen Dawei, et al.Nonparametric probabilistic optimal power flow[J]. IEEE Transactions on Power Systems, 2022, 37(4): 2758-2770.
[14] 王骞, 张学广, 徐殿国. 考虑内-外生双重不确定性的风储系统联合规划方法[J]. 中国电机工程学报, 2023, 43(1): 169-181.
Wang Qian, Zhang Xueguang, Xu Dianguo.A joint planning method of wind-storage system considering endogenous and exogenous uncertainty[J]. Proceedings of the CSEE, 2023, 43(1): 169-181.
[15] Lopez-Ramos L M, Kekatos V, Marques A G, et al. Two-timescale stochastic dispatch of smart distribution grids[J]. IEEE Transactions on Smart Grid, 2018, 9(5): 4282-4292.
[16] Li Peng, Wang Zixuan, Wang Jiahao, et al.A multi-time-space scale optimal operation strategy for a distributed integrated energy system[J]. Applied Energy, 2021, 289: 116698.
[17] 陈明昊, 孙毅, 谢志远. 基于双层深度强化学习的园区综合能源系统多时间尺度优化管理[J]. 电工技术学报, 2023, 38(7): 1864-1881.
Chen Minghao, Sun Yi, Xie Zhiyuan.The multi-time-scale management optimization method for park integrated energy system based on the Bi-layer deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2023, 38(7): 1864-1881.
[18] 李勇, 凌锋, 乔学博, 等. 基于网侧资源协调的自储能柔性互联配电系统日前-日内优化[J]. 电工技术学报, 2024, 39(3): 758-773, 923.
Li Yong, Ling Feng, Qiao Xuebo, et al.Day-ahead and intra-day optimization of flexible interconnected distribution system with self-energy storage based on the grid-side resource coordination[J]. Transactions of China Electrotechnical Society, 2024, 39(3): 758-773, 923.
[19] Kalantar-Neyestanaki M, Cherkaoui R.Coordinating distributed energy resources and utility-scale battery energy storage system for power flexibility provision under uncertainty[J]. IEEE Transactions on Sustainable Energy, 2021, 12(4): 1853-1863.
[20] Zhu Jianquan, Chen Jiajun, Zhuo Yelin, et al.Stochastic energy management of active distribution network based on improved approximate dynamic programming[J]. IEEE Transactions on Smart Grid, 2022, 13(1): 406-416.
[21] Wang Kang, Wang Chengfu, Zhang Zhenwei, et al.Multi-timescale active distribution network optimal dispatching based on SMPC[J]. IEEE Transactions on Industry Applications, 2022, 58(2): 1644-1653.
[22] Kamruzzaman M, Duan Jiajun, Shi Di, et al.A deep reinforcement learning-based multi-agent framework to enhance power system resilience using shunt resources[J]. IEEE Transactions on Power Systems, 2021, 36(6): 5525-5536.
[23] 顾雪平, 刘彤, 李少岩, 等. 基于改进双延迟深度确定性策略梯度算法的电网有功安全校正控制[J]. 电工技术学报, 2023, 38(8): 2162-2177.
Gu Xueping, Liu Tong, Li Shaoyan, et al.Active power correction control of power grid based on improved twin delayed deep deterministic policy gradient algorithm[J]. Transactions of China Electrotechnical Society, 2023, 38(8): 2162-2177.
[24] 董雷, 杨子民, 乔骥, 等. 基于分层约束强化学习的综合能源多微网系统优化调度[J]. 电工技术学报, 2024, 39(5): 1436-1453.
Dong Lei, Yang Zimin, Qiao Ji, et al.Optimal scheduling of integrated energy multi-microgrid system based on hierarchical constraint reinforcement learning[J]. Transactions of China Electrotechnical Society, 2024, 39(5): 1436-1453.
[25] 胡维昊, 曹迪, 黄琦, 等. 深度强化学习在配电网优化运行中的应用[J]. 电力系统自动化, 2023, 47(14): 174-191.
Hu Weihao, Cao Di, Huang Qi, et al.Application of deep reinforcement learning in optimal operation of distribution network[J]. Automation of Electric Power Systems, 2023, 47(14): 174-191.
[26] 王继业. 人工智能赋能源网荷储协同互动的应用及展望[J]. 中国电机工程学报, 2022, 42(21): 7667-7682.
Wang Jiye.Application and prospect of Source-grid-load-storage coordination enabled by artificial intelligence[J]. Proceedings of the CSEE, 2022, 42(21): 7667-7682.
[27] 汤涌, 姚伟, 王宏志, 等. 电网仿真分析与决策的人工智能方法[J]. 中国电机工程学报, 2022, 42(15): 5384-5406.
Tang Yong, Yao Wei, Wang Hongzhi, et al.Artificial intelligence techniques for power grid simulation analysis and decision making[J]. Proceedings of the CSEE, 2022, 42(15): 5384-5406.
[28] 李涛, 胡维昊, 李坚, 等. 基于深度强化学习算法的光伏-抽蓄互补系统智能调度[J]. 电工技术学报, 2020, 35(13): 2757-2768.
Li Tao, Hu Weihao, Li Jian, et al.Intelligent economic dispatch for PV-PHS integrated system: a deep reinforcement learning-based approach[J]. Transactions of China Electrotechnical Society, 2020, 35(13): 2757-2768.
[29] 张剑, 崔明建, 何怡刚. 结合数据驱动与物理模型的主动配电网双时间尺度电压协调优化控制[J]. 电工技术学报, 2024, 39(5): 1327-1339.
Zhang Jian, Cui Mingjian, He Yigang.Dual timescales coordinated and optimal voltages control in distribution systems using data-driven and physical optimization[J]. Transactions of China Electrotechnical Society, 2024, 39(5): 1327-1339.
[30] 张自东, 邱才明, 张东霞, 等. 基于深度强化学习的微电网复合储能协调控制方法[J]. 电网技术, 2019, 43(6): 1914-1921.
Zhang Zidong, Qiu Caiming, Zhang Dongxia, et al.A coordinated control method for hybrid energy storage system in microgrid based on deep reinforcement learning[J]. Power System Technology, 2019, 43(6): 1914-1921.
[31] Huang Shiying, Li Peng, Yang Ming, et al.A control strategy based on deep reinforcement learning under the combined wind-solar storage system[J]. IEEE Transactions on Industry Applications, 2021, 57(6): 6547-6558.
[32] Yang Qiuling, Wang Gang, Sadeghi A, et al.Two-timescale voltage control in distribution grids using deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(3): 2313-2323.
[33] Wu Hongyu, Shahidehpour M, Alabdulwahab A, et al.Demand response exchange in the stochastic day-ahead scheduling with variable renewable generation[J]. IEEE Transactions on Sustainable Energy, 2015, 6(2): 516-525.
[34] 刘全, 翟建伟, 章宗长, 等. 深度强化学习综述[J]. 计算机学报, 2018, 41(1): 1-27.
Liu Quan, Zhai Jianwei, Zhang Zongchang, et al.A survey on deep reinforcement learning[J]. Chinese Journal of Computers, 2018, 41(1): 1-27.
[35] 董雷, 陈卉, 蒲天骄, 等. 基于模型预测控制的主动配电网多时间尺度动态优化调度[J]. 中国电机工程学报, 2016, 36(17): 4609-4617.
Dong Lei, Chen Hui, Pu Tianjiao, et al.Multi-time scale dynamic optimal dispatch in active distribution network based on model predictive control[J]. Proceedings of the CSEE, 2016, 36(17): 4609-4617.
[36] 姜云鹏, 任洲洋, 李秋燕, 等. 考虑多灵活性资源协调调度的配电网新能源消纳策略[J]. 电工技术学报, 2022, 37(7): 1820-1835.
Jiang Yunpeng, Ren Zhouyang, Li Qiuyan, et al.An accommodation strategy for renewable energy in distribution network considering coordinated dispatching of multi-flexible resources[J]. Transactions of China Electrotechnical Society, 2022, 37(7): 1820-1835.
[37] Li Peng, Wang Zixuan, Liu Haitao, et al.Bi-level optimal configuration strategy of community integrated energy system with coordinated planning and operation[J]. Energy, 2021, 236: 121539.
[38] 朱兰, 周雪莹, 唐陇军, 等. 计及可中断负荷的微电网多目标优化运行[J]. 电网技术, 2017, 41(6): 1847-1854.
Zhu Lan, Zhou Xueying, Tang Longjun, et al.Multi-objective optimal operation for microgrid considering interruptible loads[J]. Power System Technology, 2017, 41(6): 1847-1854.
[39] Sheng Hongzhang, Wang Chengfu, Li Bowen, et al.Multi-timescale active distribution network scheduling considering demand response and user comprehensive satisfaction[J]. IEEE Transactions on Industry Applications, 2021, 57(3): 1995-2005.
[40] 刘俊峰, 罗燕, 侯媛媛, 等. 考虑广义储能的微电网主动能量管理优化算法研究[J]. 电网技术, 2023, 47(1): 245-255.
Liu Junfeng, Luo Yan, Hou Yuanyuan, et al.Research on optimization algorithm of active microgrid energy management considering generalized energy storage[J]. Power System Technology, 2023, 47(1): 245-255.
[41] 王珂, 姚建国, 余佩遥, 等. 基于深度强化学习的电网前瞻调度智能决策架构及关键技术初探[J]. 中国电机工程学报, 2022, 42(15): 5430-5439.
Wang Ke, Yao Jianguo, Yu Peiyao, et al.Architecture and key technologies of intelligent decision-making of power grid look-ahead dispatch based on deep reinforcement learning[J]. Proceedings of the CSEE, 2022, 42(15): 5430-5439.
[42] 李相俊, 马锐. 考虑电池组健康状态的储能系统能量管理方法[J]. 电网技术, 2020, 44(11): 4210-4217.
Li Xiangjun, Ma Rui.Energy management method of energy storage system considering the SOH of battery pack[J]. Power System Technology, 2020, 44(11): 4210-4217.