A Safe Scheduling Strategy for Distribution Network Considering 5G Base Station Energy Storage Based on Time and Graph Embedding Reinforcement Learning
Pei Qingqi1, Chen Yishi2, He Zhiyong2, Liu Qinghua2, Zhang Senlin1
1. College of Electrical Engineering Zhejiang University Hangzhou 310027 China; 2. China Three Gorges Corporation Zhejiang Energy Investment Co. Ltd Hangzhou 310020 China
Abstract：With the large-scale deployment of 5G base stations, a number of base station energy storage systems have been integrated into the power grid. These storage devices possess a certain degree of dispatch flexibility and have the potential to facilitate the consumption of renewable energy. How to make full use of these energy storage resources in distribution network scheduling has become an important issue. Existing scheduling algorithms, however, suffer from long calculation times, resulting in poor real-time performance. In addition, traditional algorithms rely on precise mathematical models and are easily affected by environmental uncertainties, which leads to inaccurate solutions that deviate from the optimal solution. To address these challenges, this paper proposes a real-time scheduling algorithm based on reinforcement learning (RL), tailored to the characteristics of 5G base station energy storage, leveraging the fast decision capability of RL to handle the scheduling challenges. Firstly, based on the economic operation requirements of the distribution network, multiple cost components are considered, including the cost of purchasing electricity from the main grid, the cost of abandoning renewable energy, and the operating cost of the 5G base station energy storages. Taking into account the safety capacity constraints of 5G base station energy storages and physical constraints of distribution network, a mathematical optimization model for real-time scheduling is established. This model is then transformed into a Markov decision process (MDP), providing the foundation for an RL-based solution framework. Secondly, considering the characteristics of state information during the scheduling process, different types of information are processed separately. For temporal information, a novel Time2Vec (T2V) architecture is introduced to extract time-related features. For the graph-structured information, a graph convolutional network (GCN) is employed for feature extraction. By integrating these modules, the traditional deep deterministic policy gradient (DDPG) algorithm is improved, resulting in the proposed T2V-GCN-DDPG algorithm, which enhances the ability of agent to extract hidden features from the state, thereby improving decision-making performance. In addition, to ensure the safety of actions from agent, a safety constraint layer based on quadratic programming (QP) is designed to fine-tune the charging and discharging power of the base station energy storage. The QP model aims to minimize the magnitude of action adjustment while keeping the actions within the safe range. Finally, to validate the effectiveness of the proposed algorithm, experiments were conducted in modified IEEE 33-bus and 141-bus distribution networks incorporating 5G base station energy storage systems. By comparing the results with those obtained from intraday rolling optimization, proximal policy optimization (PPO), and behavior cloning, it was demonstrated that the proposed algorithm achieves superior decision-making performance. It effectively reduces the deviation in actions caused by environmental uncertainties and generates actions closer to the optimal solution. Moreover, comparisons of the algorithm before and after the proposed improvements further confirm the effectiveness of the enhancement method. Simulation results also indicate that the proposed algorithm, while ensuring the safety of decision actions, can reduce power grid operating costs and improve overall economic performance. Therefore, the proposed algorithm provides a reliable and efficient solution for the safe and economical operation of the power grid.
裴青琦, 陈逸诗, 何智勇, 刘清华, 张森林. 基于时间与图嵌入强化学习的含5G基站储能配电网安全调度策略[J]. 电工技术学报, 2026, 41(13): 4386-4402.
Pei Qingqi, Chen Yishi, He Zhiyong, Liu Qinghua, Zhang Senlin. A Safe Scheduling Strategy for Distribution Network Considering 5G Base Station Energy Storage Based on Time and Graph Embedding Reinforcement Learning. Transactions of China Electrotechnical Society, 2026, 41(13): 4386-4402.
[1] 雍培, 张宁, 慈松, 等. 5G通信基站参与需求响应: 关键技术与前景展望[J]. 中国电机工程学报, 2021, 41(16): 5540-5552. Yong Pei, Zhang Ning, Ci Song, et al.5G communication base stations participating in demand response: key technologies and prospects[J]. Proceedings of the CSEE, 2021, 41(16): 5540-5552. [2] 麻秀范, 张乐萱, 于琨澎, 等. 考虑5G基站备用储能优化调控的配电网重构双层优化方法[J]. 电工技术学报, 2024, 39(16): 5028-5041. Ma Xiufan, Zhang Lexuan, Yu Kunpeng, et al.A two-layer optimization approach for distribution network reconfiguration considering optimal regulation of 5G base station backup energy storage[J]. Transactions of China Electrotechnical Society, 2024, 39(16): 5028-5041. [3] 孙名轶, 赵霞, 武桢寓, 等. 5G基站与区域综合能源系统的水-能耦合与协同[J]. 中国电机工程学报, 2023, 43(15): 5903-5916. Sun Mingyi, Zhao Xia, Wu Zhenyu, et al.Water- energy nexus and coordination between 5G base stations and regional integrated energy systems[J]. Proceedings of the CSEE, 2023, 43(15): 5903-5916. [4] 张巍, 祝童童, 苏瑾. 考虑电动汽车和5G基站的电力-信息-交通耦合网络需求响应策略[J]. 电力系统自动化, 2024, 48(7): 116-126. Zhang Wei, Zhu Tongtong, Su Jin.Demand response strategy for power-cyber-transportation coupling network considering electric vehicles and 5G base stations[J]. Automation of Electric Power Systems, 2024, 48(7): 116-126. [5] 刘洪, 王志杰, 徐正阳, 等. 基于通信负载站间迁移的配电网与5G基站日前互动运行方法[J]. 电力系统自动化, 2024, 48(21): 82-91. Liu Hong, Wang Zhijie, Xu Zhengyang, et al.Day- ahead interactive operation method for distribution network and 5G base stations based on inter-station migration of communication load[J]. Automation of Electric Power Systems, 2024, 48(21): 82-91. [6] 毕皓淳, 祁琪, 刘向军, 等. 计及5G基站运行协同的分布式储能优化配置方法[J]. 电工技术学报, 2024, 39(21): 6819-6833. Bi Haochun, Qi Qi, Liu Xiangjun, et al.An optimal configuration method for distributed energy storage considering coordinated operation of 5G base station[J]. Transactions of China Electrotechnical Society, 2024, 39(21): 6819-6833. [7] 高冠中, 杨胜春, 郭晓蕊, 等. 深度强化学习在含分布式柔性资源的电网优化调度中的应用研究综述[J]. 中国电机工程学报, 2024, 44(16): 6385-6404. Gao Guanzhong, Yang Shengchun, Guo Xiaorui, et al.A review of research on the application of deep reinforcement learning in optimization dispatch of power grids with distributed flexible resources[J]. Proceedings of the CSEE, 2024, 44(16): 6385-6404. [8] 李鹏, 钟瀚明, 马红伟, 等. 基于深度强化学习的有源配电网多时间尺度源荷储协同优化调控[J]. 电工技术学报, 2025, 40(5): 1487-1502. Li Peng, Zhong Hanming, Ma Hongwei, et al.Multi- timescale optimal dispatch of source-load-storage coordination in active distribution network based on deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2025, 40(5): 1487-1502. [9] 李东颖, 朱建全, 陈一熙. 基于双缓冲区生成对抗模仿学习的电力系统实时安全约束经济调度[J]. 电网技术, 2025, 49(3): 1121-1129. Li Dongying, Zhu Jianquan, Chen Yixi.Security- constrained economic dispatch of power systems based on dual buffer generative adversarial imitation learning[J]. Power System Technology, 2025, 49(3): 1121-1129. [10] 成梁成, 严嘉豪, 姚建国, 等. 面向电网前瞻调度嵌入领域知识的深度强化学习方法[J]. 电网技术, 2024, 48(8): 3133-3142. Cheng Liangcheng, Yan Jiahao, Yao Jianguo, et al.Look-ahead dispatch method via deep reinforcement learning embedded with domain knowledge[J]. Power System Technology, 2024, 48(8): 3133-3142. [11] 杨志学, 任洲洋, 孙志媛, 等. 基于近端策略优化算法的新能源电力系统安全约束经济调度方法[J]. 电网技术, 2023, 47(3): 988-998. Yang Zhixue, Ren Zhouyang, Sun Zhiyuan, et al.Security-constrained economic dispatch of renewable energy integrated power systems based on proximal policy optimization algorithm[J]. Power System Technology, 2023, 47(3): 988-998. [12] 季颖, 王建辉. 基于深度强化学习的微电网在线优化调度[J]. 控制与决策, 2022, 37(7): 1675-1684. Ji Ying, Wang Jianhui.Online optimal scheduling of a microgrid based on deep reinforcement learning[J]. Control and Decision, 2022, 37(7): 1675-1684. [13] 仪忠凯, 梁寿愚, 王伟, 等. 电力系统调度决策: 一种示教学习辅助加速的安全强化学习方法[J]. 中国电机工程学报, 2024, 44(13): 5084-5097. Yi Zhongkai, Liang Shouyu, Wang Wei, et al.Power system dispatch: an accelerated safe reinforcement learning approach by incorporating learning from demonstration[J]. Proceedings of the CSEE, 2024, 44(13): 5084-5097. [14] 蒋廷耀, 谢龙恩, 杜雨, 等. 基于深度强化学习的5G基站储能调度策略[J]. 电力系统自动化, 2023, 47(9): 147-157. Jiang Tingyao, Xie Longen, Du Yu, et al.Dispatching strategy of energy storage for 5G base stations based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2023, 47(9): 147-157. [15] 陈实, 郭正伟, 周步祥, 等. 考虑5G基站储能可调度容量的有源配电网协同优化调度方法[J]. 电网技术, 2023, 47(12): 5225-5241. Chen Shi, Guo Zhengwei, Zhou Buxiang, et al.Coordinated optimization scheduling for active distribution networks considering schedulable capacity of energy storage for 5G base stations[J]. Power System Technology, 2023, 47(12): 5225-5241. [16] 曾博, 穆宏伟, 孟自帅, 等. 考虑通信灵活性的5G基站与含光伏电源配电网多目标区间规划[J]. 太阳能学报, 2024, 45(4): 423-433. Zeng Bo, Mu Hongwei, Meng Zishuai, et al.Multi- objective interval planning for 5g base stations and distribution networks with photovoltaic power sources considering communication flexibility[J]. Acta Energiae Solaris Sinica, 2024, 45(4): 423-433. [17] 王鹏程, 王岩庆. 考虑电动汽车充放电切换次数的微电网两阶段优化调度研究[J]. 山东工业技术, 2024(4): 78-84. Wang Pengcheng, Wang Yanqing.Research on two-stage optimization scheduling of microgrids considering the switching frequency of electric vehicle charging and discharging[J]. Journal of Shandong Industrial Technology, 2024(4): 78-84. [18] 高洪超, 李楚一, 王冠雄, 等. 聚合规模化5G基站的虚拟电厂动态构建与响应实证[J]. 电力系统自动化, 2024, 48(18): 47-55. Gao Hongchao, Li Chuyi, Wang Guanxiong, et al.Dynamic construction and response demonstration of virtual power plant with aggregated large-scale 5G base stations[J]. Automation of Electric Power Systems, 2024, 48(18): 47-55. [19] 李俊双, 胡炎, 邰能灵. 计及通信负载与供电可靠性的5G基站储能与配电网协同优化调度[J]. 上海交通大学学报, 2023, 57(7): 791-802. Li Junshuang, Hu Yan, Tai Nengling.Collaborative optimization scheduling of 5G base station energy storage and distribution network considering communication load and power supply reliability[J]. Journal of Shanghai Jiao Tong University, 2023, 57(7): 791-802. [20] 麻秀范, 孟祥玉, 朱秋萍, 等. 计及通信负载的5G基站储能调控策略[J]. 电工技术学报, 2022, 37(11): 2878-2887. Ma Xiufan, Meng Xiangyu, Zhu Qiuping, et al.Control strategy of 5G base station energy storage considering communication load[J]. Transactions of China Electrotechnical Society, 2022, 37(11): 2878-2887. [21] Peng Yangfan, Shi Yong, Li Junshuang, et al.Optimal scheduling of 5G base station energy storage considering wind and solar complementation[C]// 2022 4th Asia Energy and Electrical Engineering Symposium (AEEES), Chengdu, China, 2022: 384-389. [22] 李昆, 方家琨, 艾小猛, 等. 考虑通信与配套设备协调优化的大规模5G宏基站网络能量管理模型[J]. 中国电机工程学报, 2023, 43(14): 5391-5404. Li Kun, Fang Jiakun, Ai Xiaomeng, et al.Energy management model of large-scale 5G macro base station network considering the coordinated optimization of communication equipment and standard equipment[J]. Proceedings of the CSEE, 2023, 43(14): 5391-5404. [23] Yong Pei, Zhang Ning, Liu Yuxiao, et al.Exploring the cellular base station dispatch potential towards power system frequency regulation[J]. IEEE Transactions on Power Systems, 2022, 37(1): 820-823. [24] Yong Pei, Zhang Ning, Hou Qingchun, et al.Evaluating the dispatchable capacity of base station backup batteries in distribution networks[J]. IEEE Transactions on Smart Grid, 2021, 12(5): 3966-3979. [25] 张津源, 蒲天骄, 李烨, 等. 基于多智能体深度强化学习的分布式电源优化调度策略[J]. 电网技术, 2022, 46(9): 3496-3504. Zhang Jinyuan, Pu Tianjiao, Li Ye, et al.Multi-agent deep reinforcement learning based optimal dispatch of distributed generators[J]. Power System Technology, 2022, 46(9): 3496-3504. [26] Kazemi S M, Goel R, Eghbali S, et al.Time2Vec: learning a vector representation of time[J/OL]. ArXiv, 2019: 1907.05321v1. https://doi.org/10.48550/arXiv.1907.05321. [27] 王新, 张良, 任晓龙, 等. 融合图神经网络模型与强化学习的综合能源系统优化调度[J]. 电力系统保护与控制, 2023, 51(24): 102-110. Wang Xin, Zhang Liang, Ren Xiaolong, et al.Optimal scheduling of integrated energy systems by fusing a graph neural network model and reinforcement learning[J]. Power System Protection and Control, 2023, 51(24): 102-110. [28] 廖文龙, 于贇, 王煜森, 等. 基于图卷积网络的配电网无功优化[J]. 电网技术, 2021, 45(6): 2150-2160. Liao Wenlong, Yu Yun, Wang Yusen, et al.Reactive power optimization of distribution network based on graph convolutional network[J]. Power System Technology, 2021, 45(6): 2150-2160. [29] Yi Zhongkai, Wang Xue, Yang Cheng, et al.Real- time sequential security-constrained optimal power flow: a hybrid knowledge-data-driven reinforcement learning approach[J]. IEEE Transactions on Power Systems, 2024, 39(1): 1664-1680. [30] 程小华, 王泽夫, 曾君, 等. 基于EA-RL算法的分布式能源集群调度方法[J]. 华南理工大学学报(自然科学版), 2025, 53(1): 1-9. Cheng Xiaohua, Wang Zefu, Zeng Jun, et al.Distributed energy cluster scheduling method based on EA-RL algorithm[J]. Journal of South China University of Technology (Natural Science Edition), 2025, 53(1): 1-9. [31] 中华人民共和国住房和城乡建设部. 通信电源设备安装工程设计规范: GB 51194—2016[S]. 北京: 中国计划出版社, 2017. [32] 高崇, 段瑶, 程苒, 等. 考虑5G基站可调度备用储能和智能软开关的主动配电网协同优化调度方法[J]. 上海交通大学学报, 2025, 59(11): 1603-1617. Gao Chong, Duan Yao, Cheng Ran, et al.Collaborative optimization scheduling method for active distribution networks considering dispatchable backup batteries of 5G base station and soft open point[J]. Journal of Shanghai Jiao Tong University, 2025, 59(11): 1603-1617. [33] 葛磊蛟, 范延赫, 李小平, 等. 计及日前随机-日内滚动分布式的高比例光伏配电网无功优化方法[J]. 电网技术, 2026, 50(3): 1071-1084. Ge Leijiao, Fan Yanhe, Li Xiaoping, et al.A reactive power optimization method for high percentage of PV distribution networks taking into account day-ahead random and intraday rolling distributions[J]. Power System Technology, 2026, 50(3): 1071-1084.
2026, Vol. 41 
