Optimal Tracking Control of Servo Motor Speed Based on Online Supplementary Q-Learning
Zou Xiaomin1, Xiao Xi1, He Qi2, Shkodyrev Vyacheslav3
1. Department of Electrical Engineering Tsinghua University Beijing 100084 China; 2. AVIC Shaanxi Aero Electric Co. Ltd Xi’an 710077 China;; 3. Peter the Great St. Petersburg Polytechnic University St. Petersburg 195251 Russia
Abstract：This paper combined online Q-learning with supplementary control and discussed its application to the optimal tracking control problem of servo motor speed. Firstly, the problem to be solved was defined in the framework of linear quadratic tracking. Then, the iterative algorithm of policy evaluation and policy improvement for online supplementary Q-learning was given. In the simulation test, for the motor speed tracking problem in servo system, the traditional PI controller was firstly designed, and then the supplementary controller proposed in this paper was connected to it in parallel to form a new speed controller. The simulation results showed that the additional controller significantly improves the dynamic response characteristics of motor speed tracking, and has the adaptive ability to automatically adjust when parameters of the controlled system changes. When the nonlinear system can be locally linearized under certain conditions, the proposed method can also be applied to obtain better control performance.
邹晓敏,肖曦,何琪,ShkodyrevVyacheslav. 基于在线附加Q学习的伺服电机速度最优跟踪控制方法[J]. 电工技术学报, 2019, 34(5): 917-923.
Zou Xiaomin, Xiao Xi, He Qi, Shkodyrev Vyacheslav. Optimal Tracking Control of Servo Motor Speed Based on Online Supplementary Q-Learning. Transactions of China Electrotechnical Society, 2019, 34(5): 917-923.
[1] Lewis F L, Vrabie D L, Syrmos V L.Optimal control[M]. 3rd ed. New York: John Wiley & Sons, Inc., 2012. [2] Lewis F L, Vrabie D.Reinforcement learning and adaptive dynamic programming for feedback control[J]. IEEE Circuits & Systems Magazine, 2009, 9(3): 32-50. [3] 张化光, 张欣, 罗艳红, 等. 自适应动态规划综述[J]. 自动化学报, 2013, 39(4): 303-311. Zhang Huaguang, Zhang Xin, Luo Yanhong, et al.An overview of research on adaptive dynamic programming[J]. Acta Automatica Sinica, 2013, 39(4): 303-311. [4] Kiumarsi B, Lewis F L, Modares H, et al.Reinforcement Q-learning for optimal tracking control of linear discrete-time systems with unknown dynamics[J]. Automatica, 2014, 50(4): 1167-1175. [5] Wang Ding, Liu Derong, Wei Qinglai.Finite-horizon neuro-optimal tracking control for a class of discrete-time nonlinear systems using adaptive dynamic programming approach[J]. Neurocomputing, 2012, 78(1): 14-22. [6] Guo Wentao, Liu Feng, Si Jennie, et al.Online supplementary ADP learning controller design and application to power system frequency control with large-scale wind energy integration[J]. IEEE Transactions on Neural Networks and Learning Systems, 2016, 27(8): 1748-1761. [7] Guo Wentao, Liu Feng, Si Jenjie, et al.Approximate dynamic programming based supplementary reactive power control for DFIG wind farm to enhance power system stability[J]. Neurocomputing, 2015, 170: 417-427. [8] Malla N, Tamrakar U, Shrestha D, et al.Online learning control for harmonics reduction based on current controlled voltage source power inverters[J]. IEEE/CAA Journal of Automatica Sinica, 2017, 4(3): 447-457. [9] Malla N, Shrestha D, Ni Zhen, et al.Supplementary control for virtual synchronous machine based on adaptive dynamic programming[C]//2016 IEEE Congress on Evolutionary Computation (CEC), Vancouver, BC, Canada, 2016: 1998-2005. [10] Shrestha D, Tamrakar U, Malla N, et al.Reduction of energy consumption of virtual synchronous machine using supplementary adaptive dynamic programming[C]// IEEE International Conference on Electro Information Technology, Grand Forks, ND, USA, 2016: 0690-0694. [11] 刘向昕, 曹晓冬, 谭国俊, 等. 电励磁同步电机全速域自适应模型预测控制[J]. 电工技术学报, 2017, 32(4): 112-122. Liu Xiangxin, Cao Xiaodong, Tan Guojun, et al.Full speed adaptive model predictive control for electrical excitation synchronous motor[J]. Transactions of China Electrotechnical Society, 2017, 32(4): 112-122. [12] 崔淑梅, 匡志, 杜博超, 等. 基于自抗扰控制原理的全电飞机用永磁同步电机转速闭环控制[J]. 电工技术学报, 2017, 32(增刊1): 107-115. Cui Shumei, Kuang Zhi, Du Bochao, et al.Speed closed-loop control of permanent magnet synchronous motor for all-electric aircraft applications based on active disturbance rejection controller[J]. Transactions of China Electrotechnical Society, 2017, 32(S1): 107-115. [13] 张洪帅, 王平, 韩邦成. 基于模糊PI模型参考自适应的高速永磁同步电机转子位置检测[J]. 中国电机工程学报, 2014, 34(12): 1889-1896. Zhang Hongshuai, Wang Ping, Han Bangcheng.Rotor position measurement for high-speed permanent magnet synchronous motors based on fuzzy PI MRAS[J]. Proceedings of the CSEE, 2014, 34(12): 1889-1896. [14] 樊英, 周晓飞, 张向阳, 等. 基于新型趋近律和混合速度控制器的IPMSM调速系统滑模变结构控制[J]. 电工技术学报, 2017, 32(5): 9-18. Fan Ying, Zhou Xiaofei, Zhang Xiangyang, et al.Sliding mode control of IPMSM system based on a new reaching law and a hybrid speed controller[J]. Transactions of China Electrotechnical Society, 2017, 32(5): 9-18. [15] 王爽, 冯坚栋, 丁雪, 等. 基于最优开环截止频率学习的永磁伺服系统PI控制器参数整定方法[J]. 电工技术学报, 2017, 32(21): 44-54. Wang Shuang, Feng Jiandong, Ding Xue, et al.PI controllers tuning method of permanent magnet servo system based on optimal open-loop cut-off frequency learning[J]. Transactions of China Electrotechnical Society, 2017, 32(21): 44-54. [16] 符慧, 左月飞, 刘闯, 等. 永磁同步电机转速环的一种变结构PI控制器[J]. 电工技术学报, 2015, 30(12): 237-242. Fu Hui, Zuo Yuefei, Liu Chuang, et al.A variable structure PI controller for permanent magnetic synchronous motor speed-regulation system[J]. Transactions of China Electrotechnical Society, 2015, 30(12): 237-242. [17] Xia Changliang, Xue Mei, Chen Ziran.Adaptive PID control and on-line identification for switched reluctance motors based on BP neural network[C]//IEEE International Conference Mechatronics and Automation, Niagara Falls, Ont., Canada, 2005, 4: 1918-1922. [18] Landelius T, Knutsson H.Greedy adaptive critics for LQR problems: convergence proofs[R]. Report LiTH-ISY-R-1896. Linkoping, Sweden: Computer Vision Laboratory, 1997. [19] Wang Weihua, Xiao Xi, Ding Youshuang.An adaptive incremental predictive current control method of PMSM[C]//2013 15th European Conference on Power Electronics and Applications, Lille, France, 2013: 1-8.
2019, Vol. 34 
