基于深度强化学习的混合动力汽车能量管理策略

doi:10.19595/j.cnki.1000-6753.tces.211342

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (2752 KB) HTML
输出: BibTeX | EndNote (RIS)

摘要针对混合动力汽车能量管理策略在线运行时对工况适应性差的问题,提出一种具有深度强化学习能力的能量管理策略设计方法。该方法基于深度确定性策略梯度算法来确定发动机功率最佳变化率,进而建立车载能源功率分配策略。所建立的控制策略包含离线交互学习与在线更新学习双层逻辑框架,根据车辆运行特性对控制参数进行动态更新,以提高在线运行时车辆的节能效果。为了验证所提出的控制策略,以沈阳实车测试数据对算法有效性进行分析,并与粒子群优化算法的控制效果进行对比。结果表明,所提出的深度强化学习能量管理策略可以实现优于粒子群优化策略的节能效果,尤其当车辆行驶特性发生突然变化时,深度强化学习控制策略具备更强的适应性。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	陈泽宇
	方志远
	杨瑞鑫
	于全庆
	康铭鑫

关键词 ：混合动力汽车, 电源系统, 能量管理策略, 深度强化学习, 机器学习

Abstract：To resolve the problem of poor adaptability to varying driving cycles when energy management strategy for hybrid electric vehicles is running online, a design method of energy management strategy (EMS) with deep reinforcement learning ability is proposed. The presented method determines the optimal change rate of engine power based on the deep deterministic policy gradient algorithm and then establishes the power management strategy of the onboard energy system. The established control strategy includes a two-layer logical framework of offline interactive learning and online update learning. The control parameters are dynamically updated according to the vehicle operation characteristics to improve the vehicle energy-saving effect in online applications. To verify the proposed control strategy, the effectiveness of the algorithm is analyzed with the practical vehicle test data in Shenyang, and compared with the control effect of the particle swarm optimization algorithm. The results show that the proposed deep reinforcement learning EMS can achieve energy-saving effects better than particle swarm optimization-based strategy. Especially when the driving characteristics of vehicles change suddenly, deep reinforcement learning control strategy can achieve better adaptability.

Key words： Hybrid electric vehicle power system energy management strategy deep reinforcement learning machine learning

收稿日期: 2021-08-25

PACS:

U469

基金资助:国家自然科学基金项目（51977029）、中央高校基本科研业务专项资金资助项目（N2003002）、辽宁省科技计划项目（2021JH6/10500135）和中国博士后科学基金资助项目（2021M690395）资助

通讯作者: 杨瑞鑫男,1988年生,博士,研究方向为新能源汽车动力电池管理。E-mail：yangruixin@bit.edu.cn

作者简介: 陈泽宇男,1982年生,副教授,博士生导师,研究方向为新能源汽车与电驱动技术、车用电池安全与故障诊断技术和电池智能化管理系统等。E-mail：chenzy@mail.neu.edu.cn

引用本文:

陈泽宇, 方志远, 杨瑞鑫, 于全庆, 康铭鑫. 基于深度强化学习的混合动力汽车能量管理策略[J]. 电工技术学报, 2022, 37(23): 6157-6168. Chen Zeyu, Fang Zhiyuan, Yang Ruixin, Yu Quanqing, Kang Mingxin. Energy Management Strategy for Hybrid Electric Vehicle Based on the Deep Reinforcement Learning Method. Transactions of China Electrotechnical Society, 2022, 37(23): 6157-6168.

链接本文:

https://dgjsxb.ces-transaction.com/CN/10.19595/j.cnki.1000-6753.tces.211342 https://dgjsxb.ces-transaction.com/CN/Y2022/V37/I23/6157

[1] 安小宇, 李元丰, 孙建彬, 等. 基于模糊逻辑的电动汽车双源混合储能系统能量管理策略[J]. 电力系统保护与控制, 2021, 49(16): 135-142.
An Xiaoyu, Li Yuanfeng, Sun Jianbin, et al.Energy management strategy of a dual-source hybrid energy storage system for electric vehicles based on fuzzy logic[J]. Power System Protection and Control, 2021, 49(16): 135-142.
[2] 姚颖蓓, 陆建忠, 傅业盛, 等. 华东地区电动汽车发展趋势及用电需求预测[J]. 电力系统保护与控制, 2021, 49(4): 141-145.
Yao Yingbei, Lu Jianzhong, Fu Yesheng, et al.Electric vehicle development trends and electricity demand forecast in East China[J]. Power System Protection and Control, 2021, 49(4): 141-145.
[3] 程杉, 杨堃, 魏昭彬, 等. 计及电价优化和放电节制的电动汽车充电站有序充放电调度[J]. 电力系统保护与控制, 2021, 49(11): 1-8.
Cheng Shan, Yang Kun, Wei Zhaobin, et al.Orderly charging and discharging scheduling of an electric vehicle charging station considering price optimization and discharge behavior control[J]. Power System Protection and Control, 2021, 49(11): 1-8.
[4] 陈嘉鹏, 汤乃云, 王雪松. 基于电动汽车入网特性的电网经济调度研究[J]. 电气技术, 2019, 20(3): 24-30, 36.
Chen Jiapeng, Tang Naiyun, Wang Xuesong.Research on economic dispatch of power grid based on vehicle to grid characteristics of electric vehicle[J]. Electrical Engineering, 2019, 20(3): 24-30, 36.
[5] 郭国太. 电动汽车充电站负荷计算及影响因素[J]. 电气技术, 2019, 20(3): 93-97.
Guo Guotai.Load calculation and influence factors of electric vehicle charging station[J]. Electrical Engineering, 2019, 20(3): 93-97.
[6] 宋健, 李梦佳, 刘囡, 等. 基于聚类算法的电动汽车充放电分时电价优化[J]. 电气技术, 2018, 19(8): 168-173.
Song Jian, Li Mengjia, Liu Nan, et al.The time-of-use price optimization of electric vehicle charging and discharging based on clustering algorithm[J]. Electrical Engineering, 2018, 19(8): 168-173.
[7] 周美兰, 冯继峰, 张宇, 等. 纯电动客车复合储能系统功率分配控制策略研究[J]. 电工技术学报, 2019, 34(23): 5001-5013.
Zhou Meilan, Feng Jifeng, Zhang Yu, et al.Research on power allocation control strategy for compound electric energy storage system of pure electric bus[J]. Transactions of China Electrotechnical Society, 2019, 34(23): 5001-5013.
[8] 石庆升, 张承慧, 崔纳新. 新型双能量源纯电动汽车能量管理问题的优化控制[J]. 电工技术学报, 2008, 23(8): 137-142.
Shi Qingsheng, Zhang Chenghui, Cui Naxin.Optimal control of energy management in novel electric vehicles with dual-source energy storage system[J]. Transactions of China Electrotechnical Society, 2008, 23(8): 137-142.
[9] 王琪, 孙玉坤, 罗印升. 混合动力电动汽车的复合电源功率分配控制策略[J]. 电工技术学报, 2017, 32(18): 143-151.
Wang Qi, Sun Yukun, Luo Yinsheng.A power distribution control strategy of hybrid energy storage system in hybrid electric vehicles[J]. Transactions of China Electrotechnical Society, 2017, 32(18): 143-151.
[10] 李家祥, 汪凤翔, 柯栋梁, 等. 基于粒子群算法的永磁同步电机模型预测控制权重系数设计[J]. 电工技术学报, 2021, 36(1): 50-59, 76.
Li Jiaxiang, Wang Fengxiang, Ke Dongliang, et al.Weighting factors design of model predictive control for permanent magnet synchronous machine using particle swarm optimization[J]. Transactions of China Electrotechnical Society, 2021, 36(1): 50-59, 76.
[11] Wu Jiangling, Sun Xiaodong, Zhu Jianguo.Accurate torque modeling with PSO-based recursive robust LSSVR for a segmented-rotor switched reluctance motor[J]. CES Transactions on Electrical Machines and Systems, 2020, 4(2): 96-104.
[12] Liu Jichao, Chen Yangzhou, Li Wei, et al.Hybrid-trip-model-based energy management of a PHEV with computation-optimized dynamic programming[J]. IEEE Transactions on Vehicular Technology, 2018, 67(1): 338-353.
[13] Zhou Wei, Yang Lin, Cai Yishan, et al.Dynamic programming for new energy vehicles based on their work modes part Ⅰ: electric vehicles and hybrid electric vehicles[J]. Journal of Power Sources, 2018, 406: 151-166.
[14] Chen Zeyu, Xiong Rui, Cao Jiayi.Particle swarm optimization-based optimal power management of plug-in hybrid electric vehicles considering uncertain driving conditions[J]. Energy, 2016, 96: 197-208.
[15] Lü Xueqin, Wu Yinbo, Lian Jie, et al.Energy management of hybrid electric vehicles: a review of energy optimization of fuel cell hybrid power system based on genetic algorithm[J]. Energy Conversion and Management, 2020, 205: 112474.
[16] Xie Shaobo, Hu Xiaosong, Xin Zongke, et al.Pontryagin's minimum principle based model predictive control of energy management for a plug-in hybrid electric bus[J]. Applied Energy, 2019, 236: 893-905.
[17] Yang Ye, Zhang Youtong, Tian Jingyi, et al.Adaptive real-time optimal energy management strategy for extender range electric vehicle[J]. Energy, 2020, 197: 117237.
[18] 陈剑, 杜文娟, 王海风. 采用深度迁移学习定位含直驱风机次同步振荡源机组的方法[J]. 电工技术学报, 2021, 36(1): 179-190.
Chen Jian, Du Wenjuan, Wang Haifeng.A method of locating the power system subsynchronous oscillation source unit with grid-connected PMSG using deep transfer learning[J]. Transactions of China Electrotechnical Society, 2021, 36(1): 179-190.
[19] 李超然, 肖飞, 樊亚翔, 等. 基于卷积神经网络的锂离子电池SOH估算[J]. 电工技术学报, 2020, 35(19): 4106-4119.
Li Chaoran, Xiao Fei, Fan Yaxiang, et al.An approach to lithium-ion battery SOH estimation based on convolutional neural network[J]. Transactions of China Electrotechnical Society, 2020, 35(19): 4106-4119.
[20] 徐建军, 黄立达, 闫丽梅, 等. 基于层次多任务深度学习的绝缘子自爆缺陷检测[J]. 电工技术学报, 2021, 36(7): 1407-1415.
Xu Jianjun, Huang Lida, Yan Limei, et al.Insulator self-explosion defect detection based on hierarchical multi-task deep learning[J]. Transactions of China Electrotechnical Society, 2021, 36(7): 1407-1415.
[21] 张甜, 赵奇, 陈中, 等. 基于深度强化学习的家庭能量管理分层优化策略[J]. 电力系统自动化, 2021, 45(21): 149-158.
Zhang Tian, Zhao Qi, Chen Zhong, et al.Hierarchical optimization strategy for home energy management based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(21): 149-158.
[22] 叶宇剑, 王卉宇, 汤奕, 等. 基于深度强化学习的居民实时自治最优能量管理策略[J]. 电力系统自动化, 2022, 46(1): 110-119.
Ye Yujian, Wang Huiyu, Tang Yi, et al.Real-time autonomous optimal energy management strategy for residents based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2022, 46(1): 110-119.
[23] 赵冬梅, 陶然, 马泰屹, 等. 基于多智能体深度确定策略梯度算法的有功-无功协调调度模型[J]. 电工技术学报, 2021, 36(9): 1914-1925.
Zhao Dongmei, Tao Ran, Ma Taiyi, et al.Active and reactive power coordinated dispatching based on multi-agent deep deterministic policy gradient algorithm[J]. Transactions of China Electrotechnical Society, 2021, 36(9): 1914-1925.
[24] Zhao Pu, Wang Yanzhi, Chang N, et al.A deep reinforcement learning framework for optimizing fuel economy of hybrid electric vehicles[C]//2018 23rd Asia and South Pacific Design Automation Conference (ASP-DAC), Jeju, Korea (South), 2018: 196-202.
[25] He Dingbo, Zou Yuan, Wu Jinlong, et al.Deep Q-learning based energy management strategy for a series hybrid electric tracked vehicle and its adaptability validation[C]//2019 IEEE Transportation Electrification Conference and Expo (ITEC), Detroit, MI, USA, 2019: 1-6.
[26] Han Xuefeng, He Hongwen, Wu Jingda, et al.Energy management based on reinforcement learning with double deep Q-learning for a hybrid electric tracked vehicle[J]. Applied Energy, 2019, 254: 113708.
[27] Zhu Zhaoxuan, Liu Yuxing, Canova M.Energy management of hybrid electric vehicles via deep Q-networks[C]//2020 American Control Conference (ACC), Denver, CO, USA, 2020: 3077-3082.
[28] Du Guodong, Zou Yuan, Zhang Xudong, et al.Deep reinforcement learning based energy management for a hybrid electric vehicle[J]. Energy, 2020, 201: 117591.
[29] Tan Huachun, Zhang Hailong, Peng Jiankun, et al.Energy management of hybrid electric bus based on deep reinforcement learning in continuous state and action space[J]. Energy Conversion and Management, 2019, 195: 548-560.
[30] Li Yuecheng, He Hongwen, Khajepour A, et al.Energy management for a power-split hybrid electric bus via deep reinforcement learning with terrain information[J]. Applied Energy, 2019, 255: 113762.
[31] Li Yuecheng, He Hongwen, Peng Jiankun, et al.Deep reinforcement learning-based energy management for a series hybrid electric vehicle enabled by history cumulative trip information[J]. IEEE Transactions on Vehicular Technology, 2019, 68(8): 7416-7430.
[32] Geng Wenran, Lou Diming, Wang Chen, et al.A cascaded energy management optimization method of multimode power-split hybrid electric vehicles[J]. Energy, 2020, 199: 117224.
[33] Fei Zhigen, Wu Zhiying, Xiao Yanqiu, et al.A new short-arc fitting method with high precision using Adam optimization algorithm[J]. Optik, 2020, 212: 164788.
[34] Chen Zeyu, Zhang Qing, Lu Jiahuan, et al.Optimization-based method to develop practical driving cycle for application in electric vehicle power management: a case study in Shenyang, China[J]. Energy, 2019, 186: 115766.