Energy Management Strategy for Hybrid Electric Vehicle Based on the Deep Reinforcement Learning Method
Chen Zeyu1,2, Fang Zhiyuan1, Yang Ruixin2, Yu Quanqing2,3, Kang Mingxin1
1. School of Mechanical Engineering and Automation Northeastern University Shenyang 110819 China; 2. School of Mechanical Engineering Beijing Institute of Technology Beijing 100081 China; 3. School of Automotive Engineering Harbin Institute of Technology at Weihai Weihai 264209 China
Abstract To resolve the problem of poor adaptability to varying driving cycles when energy management strategy for hybrid electric vehicles is running online, a design method of energy management strategy (EMS) with deep reinforcement learning ability is proposed. The presented method determines the optimal change rate of engine power based on the deep deterministic policy gradient algorithm and then establishes the power management strategy of the onboard energy system. The established control strategy includes a two-layer logical framework of offline interactive learning and online update learning. The control parameters are dynamically updated according to the vehicle operation characteristics to improve the vehicle energy-saving effect in online applications. To verify the proposed control strategy, the effectiveness of the algorithm is analyzed with the practical vehicle test data in Shenyang, and compared with the control effect of the particle swarm optimization algorithm. The results show that the proposed deep reinforcement learning EMS can achieve energy-saving effects better than particle swarm optimization-based strategy. Especially when the driving characteristics of vehicles change suddenly, deep reinforcement learning control strategy can achieve better adaptability.
Chen Zeyu,Fang Zhiyuan,Yang Ruixin等. Energy Management Strategy for Hybrid Electric Vehicle Based on the Deep Reinforcement Learning Method[J]. Transactions of China Electrotechnical Society, 2022, 37(23): 6157-6168.
[1] 安小宇, 李元丰, 孙建彬, 等. 基于模糊逻辑的电动汽车双源混合储能系统能量管理策略[J]. 电力系统保护与控制, 2021, 49(16): 135-142. An Xiaoyu, Li Yuanfeng, Sun Jianbin, et al.Energy management strategy of a dual-source hybrid energy storage system for electric vehicles based on fuzzy logic[J]. Power System Protection and Control, 2021, 49(16): 135-142. [2] 姚颖蓓, 陆建忠, 傅业盛, 等. 华东地区电动汽车发展趋势及用电需求预测[J]. 电力系统保护与控制, 2021, 49(4): 141-145. Yao Yingbei, Lu Jianzhong, Fu Yesheng, et al.Electric vehicle development trends and electricity demand forecast in East China[J]. Power System Protection and Control, 2021, 49(4): 141-145. [3] 程杉, 杨堃, 魏昭彬, 等. 计及电价优化和放电节制的电动汽车充电站有序充放电调度[J]. 电力系统保护与控制, 2021, 49(11): 1-8. Cheng Shan, Yang Kun, Wei Zhaobin, et al.Orderly charging and discharging scheduling of an electric vehicle charging station considering price optimization and discharge behavior control[J]. Power System Protection and Control, 2021, 49(11): 1-8. [4] 陈嘉鹏, 汤乃云, 王雪松. 基于电动汽车入网特性的电网经济调度研究[J]. 电气技术, 2019, 20(3): 24-30, 36. Chen Jiapeng, Tang Naiyun, Wang Xuesong.Research on economic dispatch of power grid based on vehicle to grid characteristics of electric vehicle[J]. Electrical Engineering, 2019, 20(3): 24-30, 36. [5] 郭国太. 电动汽车充电站负荷计算及影响因素[J]. 电气技术, 2019, 20(3): 93-97. Guo Guotai.Load calculation and influence factors of electric vehicle charging station[J]. Electrical Engineering, 2019, 20(3): 93-97. [6] 宋健, 李梦佳, 刘囡, 等. 基于聚类算法的电动汽车充放电分时电价优化[J]. 电气技术, 2018, 19(8): 168-173. Song Jian, Li Mengjia, Liu Nan, et al.The time-of-use price optimization of electric vehicle charging and discharging based on clustering algorithm[J]. Electrical Engineering, 2018, 19(8): 168-173. [7] 周美兰, 冯继峰, 张宇, 等. 纯电动客车复合储能系统功率分配控制策略研究[J]. 电工技术学报, 2019, 34(23): 5001-5013. Zhou Meilan, Feng Jifeng, Zhang Yu, et al.Research on power allocation control strategy for compound electric energy storage system of pure electric bus[J]. Transactions of China Electrotechnical Society, 2019, 34(23): 5001-5013. [8] 石庆升, 张承慧, 崔纳新. 新型双能量源纯电动汽车能量管理问题的优化控制[J]. 电工技术学报, 2008, 23(8): 137-142. Shi Qingsheng, Zhang Chenghui, Cui Naxin.Optimal control of energy management in novel electric vehicles with dual-source energy storage system[J]. Transactions of China Electrotechnical Society, 2008, 23(8): 137-142. [9] 王琪, 孙玉坤, 罗印升. 混合动力电动汽车的复合电源功率分配控制策略[J]. 电工技术学报, 2017, 32(18): 143-151. Wang Qi, Sun Yukun, Luo Yinsheng.A power distribution control strategy of hybrid energy storage system in hybrid electric vehicles[J]. Transactions of China Electrotechnical Society, 2017, 32(18): 143-151. [10] 李家祥, 汪凤翔, 柯栋梁, 等. 基于粒子群算法的永磁同步电机模型预测控制权重系数设计[J]. 电工技术学报, 2021, 36(1): 50-59, 76. Li Jiaxiang, Wang Fengxiang, Ke Dongliang, et al.Weighting factors design of model predictive control for permanent magnet synchronous machine using particle swarm optimization[J]. Transactions of China Electrotechnical Society, 2021, 36(1): 50-59, 76. [11] Wu Jiangling, Sun Xiaodong, Zhu Jianguo.Accurate torque modeling with PSO-based recursive robust LSSVR for a segmented-rotor switched reluctance motor[J]. CES Transactions on Electrical Machines and Systems, 2020, 4(2): 96-104. [12] Liu Jichao, Chen Yangzhou, Li Wei, et al.Hybrid-trip-model-based energy management of a PHEV with computation-optimized dynamic programming[J]. IEEE Transactions on Vehicular Technology, 2018, 67(1): 338-353. [13] Zhou Wei, Yang Lin, Cai Yishan, et al.Dynamic programming for new energy vehicles based on their work modes part Ⅰ: electric vehicles and hybrid electric vehicles[J]. Journal of Power Sources, 2018, 406: 151-166. [14] Chen Zeyu, Xiong Rui, Cao Jiayi.Particle swarm optimization-based optimal power management of plug-in hybrid electric vehicles considering uncertain driving conditions[J]. Energy, 2016, 96: 197-208. [15] Lü Xueqin, Wu Yinbo, Lian Jie, et al.Energy management of hybrid electric vehicles: a review of energy optimization of fuel cell hybrid power system based on genetic algorithm[J]. Energy Conversion and Management, 2020, 205: 112474. [16] Xie Shaobo, Hu Xiaosong, Xin Zongke, et al.Pontryagin's minimum principle based model predictive control of energy management for a plug-in hybrid electric bus[J]. Applied Energy, 2019, 236: 893-905. [17] Yang Ye, Zhang Youtong, Tian Jingyi, et al.Adaptive real-time optimal energy management strategy for extender range electric vehicle[J]. Energy, 2020, 197: 117237. [18] 陈剑, 杜文娟, 王海风. 采用深度迁移学习定位含直驱风机次同步振荡源机组的方法[J]. 电工技术学报, 2021, 36(1): 179-190. Chen Jian, Du Wenjuan, Wang Haifeng.A method of locating the power system subsynchronous oscillation source unit with grid-connected PMSG using deep transfer learning[J]. Transactions of China Electrotechnical Society, 2021, 36(1): 179-190. [19] 李超然, 肖飞, 樊亚翔, 等. 基于卷积神经网络的锂离子电池SOH估算[J]. 电工技术学报, 2020, 35(19): 4106-4119. Li Chaoran, Xiao Fei, Fan Yaxiang, et al.An approach to lithium-ion battery SOH estimation based on convolutional neural network[J]. Transactions of China Electrotechnical Society, 2020, 35(19): 4106-4119. [20] 徐建军, 黄立达, 闫丽梅, 等. 基于层次多任务深度学习的绝缘子自爆缺陷检测[J]. 电工技术学报, 2021, 36(7): 1407-1415. Xu Jianjun, Huang Lida, Yan Limei, et al.Insulator self-explosion defect detection based on hierarchical multi-task deep learning[J]. Transactions of China Electrotechnical Society, 2021, 36(7): 1407-1415. [21] 张甜, 赵奇, 陈中, 等. 基于深度强化学习的家庭能量管理分层优化策略[J]. 电力系统自动化, 2021, 45(21): 149-158. Zhang Tian, Zhao Qi, Chen Zhong, et al.Hierarchical optimization strategy for home energy management based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(21): 149-158. [22] 叶宇剑, 王卉宇, 汤奕, 等. 基于深度强化学习的居民实时自治最优能量管理策略[J]. 电力系统自动化, 2022, 46(1): 110-119. Ye Yujian, Wang Huiyu, Tang Yi, et al.Real-time autonomous optimal energy management strategy for residents based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2022, 46(1): 110-119. [23] 赵冬梅, 陶然, 马泰屹, 等. 基于多智能体深度确定策略梯度算法的有功-无功协调调度模型[J]. 电工技术学报, 2021, 36(9): 1914-1925. Zhao Dongmei, Tao Ran, Ma Taiyi, et al.Active and reactive power coordinated dispatching based on multi-agent deep deterministic policy gradient algorithm[J]. Transactions of China Electrotechnical Society, 2021, 36(9): 1914-1925. [24] Zhao Pu, Wang Yanzhi, Chang N, et al.A deep reinforcement learning framework for optimizing fuel economy of hybrid electric vehicles[C]//2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), Jeju, Korea (South), 2018: 196-202. [25] He Dingbo, Zou Yuan, Wu Jinlong, et al.Deep Q-learning based energy management strategy for a series hybrid electric tracked vehicle and its adaptability validation[C]//2019 IEEE Transportation Electrification Conference and Expo (ITEC), Detroit, MI, USA, 2019: 1-6. [26] Han Xuefeng, He Hongwen, Wu Jingda, et al.Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Applied Energy, 2019, 254: 113708. [27] Zhu Zhaoxuan, Liu Yuxing, Canova M.Energy management of hybrid electric vehicles via deep Q-networks[C]//2020 American Control Conference (ACC), Denver, CO, USA, 2020: 3077-3082. [28] Du Guodong, Zou Yuan, Zhang Xudong, et al.Deep reinforcement learning based energy management for a hybrid electric vehicle[J]. Energy, 2020, 201: 117591. [29] Tan Huachun, Zhang Hailong, Peng Jiankun, et al.Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space[J]. Energy Conversion and Management, 2019, 195: 548-560. [30] Li Yuecheng, He Hongwen, Khajepour A, et al.Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information[J]. Applied Energy, 2019, 255: 113762. [31] Li Yuecheng, He Hongwen, Peng Jiankun, et al.Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information[J]. IEEE Transactions on Vehicular Technology, 2019, 68(8): 7416-7430. [32] Geng Wenran, Lou Diming, Wang Chen, et al.A cascaded energy management optimization method of multimode power-split hybrid electric vehicles[J]. Energy, 2020, 199: 117224. [33] Fei Zhigen, Wu Zhiying, Xiao Yanqiu, et al.A new short-arc fitting method with high precision using Adam optimization algorithm[J]. Optik, 2020, 212: 164788. [34] Chen Zeyu, Zhang Qing, Lu Jiahuan, et al.Optimization-based method to develop practical driving cycle for application in electric vehicle power management: a case study in Shenyang, China[J]. Energy, 2019, 186: 115766.
2022, Vol. 37 
