Abstract：The voltage source converter based high voltage direct current (VSC-HVDC) systems featuring grid-forming (GFM) control can provide critical frequency and voltage support to weak or passive grids, making them a key research focus in modern power systems. This paper proposes a model-free integrated frequency control method for grid-forming voltage source converters (GFM-VSCs) within a virtual synchronous generator (VSG) framework. The method integrates the integral reinforcement learning algorithm-based optimal frequency control with active load disturbance compensation to deliver reliable frequency support to the target AC grid. Firstly, the dynamic models of the target AC grid and the GFM-VSC were established, and the oscillatory modes present in their coupled operation were analyzed. Subsequently, an optimal frequency controller was designed based on a predefined quadratic performance index to improve system damping, independent of the VSG parameters. To address unknown model dynamics, a policy iteration algorithm utilizing impulse perturbation excitation was developed under an integral reinforcement learning framework, to solve the optimal frequency controller in a fully data-driven manner. Furthermore, a reduced order load disturbance observer was designed, along with a disturbance compensation controller. This ancillary disturbance compensation module was used to enhance the VSC’s active response to load disturbances, enable rapid secondary frequency regulation, and operate in coordination with the optimal frequency controller trained by the policy iteration algorithm. The performance of the proposed method was evaluated using an equivalent system test model and the IEEE standard 9-Bus test model, both implemented in Matlab/Simulink. Comparative analysis included three alternative approaches: the VSG with a Nichols-PI regulator, the VSG with a V2-P-w droop regulator, and the traditional VSG scheme. During online training, the Critic neural network parameters converged within 10 iterations. The proposed impulse perturbation excitation method reduced the error norm of the Critic parameter matrix to below 0.24, demonstrating higher accuracy than traditional noise based excitation method, which achieved only 1.64. Under a 50 MW step load disturbance, the integrated frequency control method limited frequency deviations to within 0.001(pu) in the equivalent system and 0.001 5(pu) in the IEEE 9-Bus system. The grid frequency was restored to the nominal value within 1.5 s. The proposed method exhibited significantly better frequency support performance than the three benchmark methods. It also provided faster rate of change of frequency (RoCoF) attenuation and notably suppressed frequency and power oscillations. Under a heavy load test scenario (exceeding 85% of the rated power), the method successfully restricted the active power reference command to the predefined safety limit. This ability protected the VSC from overload beyond safe durations. Based on the simulation analysis, the following conclusions can be drawn: (1) The policy iteration algorithm incorporating impulse perturbation excitation demonstrates accelerated convergence rate and enhanced precision compared to the traditional noise based excitation methods. (2) The proposed integrated frequency control method demonstrates superior damping performance under load disturbances, significantly attenuating RoCoF, as well as frequency and power oscillations. (3) The proposed integrated frequency control method effectively compensates load-side disturbances and ensures prompt restoration of the grid frequency to its nominal value.
李寅生, 王冰, 陈玉全, 王万成. 基于积分强化学习的构网型VSC综合频率控制[J]. 电工技术学报, 2026, 41(7): 2368-2382.
Li Yinsheng, Wang Bing, Chen Yuquan, Wang Wancheng. Integrated Frequency Control for Grid-Forming VSC Based on Integral Reinforcement Learning. Transactions of China Electrotechnical Society, 2026, 41(7): 2368-2382.
[1] 詹长江, 吴恒, 王雄飞, 等. 构网型变流器稳定性研究综述[J]. 中国电机工程学报, 2023, 43(6): 2339-2359. Zhan Changjiang, Wu Heng, Wang Xiongfei, et al.An overview of stability studies of grid-forming voltage source converters[J]. Proceedings of the CSEE, 2023, 43(6): 2339-2359. [2] 韩丽, 陈硕, 王施琪, 等. 考虑风光消纳与电动汽车灵活性的调度策略[J]. 电工技术学报, 2024, 39(21): 6793-6803. Han Li, Chen Shuo, Wang Shiqi, et al.Scheduling strategy considering wind and photovoltaic power consumption and the flexibility of electric vehicles[J]. Transactions of China Electrotechnical Society, 2024, 39(21): 6793-6803. [3] 姜崇学, 马秀达, 邹强, 等. 柔性直流输电系统的高频谐波保护方法与工程实践[J]. 电力系统自动化, 2024, 48(3): 150-158. Jiang Chongxue, Ma Xiuda, Zou Qiang, et al.High-frequency harmonic protection methods and engineering practice for flexible DC transmission systems[J]. Automation of Electric Power Systems, 2024, 48(3): 150-158. [4] 王炜宇, 张一平, 李帅虎, 等. 基于扰动观测的构网型柔直系统阻尼控制器设计[J]. 电网技术, 2024, 48(6): 2262-2271. Wang Weiyu, Zhang Yiping, Li Shuaihu, et al.Disturbance observer-based damping controller of grid-forming VSC-HVDC systems[J]. Power System Technology, 2024, 48(6): 2262-2271. [5] 于国星, 宋蕙慧, 马广富, 等. 含海上风电场的VSC-MTDC系统参与电网调频的顺序控制方法[J]. 电力系统自动化, 2021, 45(4): 123-132. Yu Guoxing, Song Huihui, Ma Guangfu, et al.Sequence control method for VSC-MTDC system with offshore wind farm participating in frequency regulation of power grid[J]. Automation of Electric Power Systems, 2021, 45(4): 123-132. [6] 姚伟, 熊永新, 姚雅涵, 等. 海上风电柔直并网系统调频控制综述[J]. 高电压技术, 2021, 47(10): 3397-3413. Yao Wei, Xiong Yongxin, Yao Yahan, et al.Review of voltage source converter-based high voltage direct current integrated offshore wind farm on providing frequency support control[J]. High Voltage Engi-neering, 2021, 47(10): 3397-3413. [7] 谭珺敉, 彭晓涛, 李旭涛, 等. 基于协同控制优化风电-柔直并网惯性响应策略研究[J]. 电工技术学报, 2025, 40(5): 1355-1367. Tan Junmi, Peng Xiaotao, Li Xutao, et al.Optimization of inertia response strategy based on synergetic control for wind power integrating to power grid via VSC-HVDC[J]. Transactions of China Elec-trotechnical Society, 2025, 40(5): 1355-1367. [8] 王炜宇, 李勇, 曹一家, 等. 基于虚拟调速器的多端直流虚拟同步机控制策略[J]. 中国电机工程学报, 2018, 38(12): 3461-3470, 5. Wang Weiyu, Li Yong, Cao Yijia, et al.The virtual synchronous generator technology based on virtual governor for multi-terminal direct current system[J]. Proceedings of the CSEE, 2018, 38(12): 3461-3470, 5. [9] 葛平娟, 肖凡, 涂春鸣, 等. 考虑故障限流的下垂控制型逆变器暂态控制策略[J]. 电工技术学报, 2022, 37(14): 3676-3687. Ge Pingjuan, Xiao Fan, Tu Chunming, et al.Transient control strategy of droop-controlled inverter consi-dering fault current limitation[J]. Transactions of China Electrotechnical Society, 2022, 37(14): 3676-3687. [10] 刘英培, 谢乾, 梁海平. 柔性直流输电系统自适应虚拟惯性调频控制策略[J]. 电力系统自动化, 2021, 45(5): 129-136. Liu Yingpei, Xie Qian, Liang Haiping.Frequency regulation control strategy for flexible DC transmission system based on adaptive virtual inertia[J]. Automation of Electric Power Systems, 2021, 45(5): 129-136. [11] 刘辉, 于思奇, 孙大卫, 等. 构网型变流器控制技术及原理综述[J]. 中国电机工程学报, 2025, 45(1): 277-297. Liu Hui, Yu Siqi, Sun Dawei, et al.An overview of control technologies and principles for grid-forming converters[J]. Proceedings of the CSEE, 2025, 45(1): 277-297. [12] 徐菘, 杨博, 刘浩, 等. 一种提高虚拟同步机电流质量的电压-电流级联闭环控制方案[J]. 电工技术学报, 2024, 39(6): 1871-1885. Xu Song, Yang Bo, Liu Hao, et al.A cascaded harmonic voltage and current closed-loop control method to improve the current quality of virtual synchronous generators[J]. Transactions of China Electrotechnical Society, 2024, 39(6): 1871-1885. [13] 刘思佳, 刘海涛, 张隽, 等. 基于等效阻抗的虚拟同步机电压支撑影响因素分析与改进控制策略研究[J]. 电工技术学报, 2025, 40(9): 2738-2751. Liu Sijia, Liu Haitao, Zhang Jun, et al.Research on the analysis of virtual synchronous generator voltage support influence factors and improvement control strategies based on equivalent impedance[J]. Transactions of China Electrotechnical Society, 2025, 40(9): 2738-2751. [14] 马秀达, 卢宇, 田杰, 等. 柔性直流输电系统的构网型控制关键技术与挑战[J]. 电力系统自动化, 2023, 47(3): 1-11. Ma Xiuda, Lu Yu, Tian Jie, et al.Key technologies and challenges of grid-forming control for flexible DC transmission system[J]. Automation of Electric Power Systems, 2023, 47(3): 1-11. [15] 高本锋, 沈雨思, 宋瑞华, 等. 虚拟同步机控制模块化多电平变流器阻抗建模及次/超同步振荡稳定性分析[J]. 电工技术学报, 2025, 40(2): 559-573. Gao Benfeng, Shen Yusi, Song Ruihua, et al.Impedance modeling and sub/super synchronous oscillation stability analysis of modular multilevel converter under virtual synchronous generator control[J]. Transactions of China Electrotechnical Society, 2025, 40(2): 559-573. [16] 姚为正, 杨美娟, 张海龙, 等. VSC-HVDC受端换流器参与电网调频的VSG控制及其改进算法[J]. 中国电机工程学报, 2017, 37(2): 525-534. Yao Weizheng, Yang Meijuan, Zhang Hailong, et al.VSG control and its modified algorithm for VSC-HVDC inverter participating grid’s frequency regulation[J]. Proceedings of the CSEE, 2017, 37(2): 525-534. [17] 王炜宇, 易念棋, 蔡晔, 等. 基于功率扰动观测的构网型柔直系统辅助频率控制策略[J]. 中国电机工程学报, 2025, 45(5): 1680-1691. Wang Weiyu, Yi Nianqi, Cai Ye, et al.A power-disturbance observer-based auxiliary frequency control of grid-forming VSC-HVDC systems[J]. Proceedings of the CSEE, 2025, 45(5): 1680-1691. [18] Leon A E, Mauricio J M.Virtual synchronous generator for VSC-HVDC stations with DC voltage control[J]. IEEE Transactions on Power Systems, 2023, 38(1): 728-738. [19] Saad H, Peralta J, Dennetière S, et al.Dynamic averaged and simplified models for MMC-based HVDC transmission systems[J]. IEEE Transactions on Power Delivery, 2013, 28(3): 1723-1730. [20] 李寅生, 王冰, 陈玉全, 等. 基于双人零和博弈的孤岛微电网有界L2增益负荷频率控制[J]. 电力系统自动化, 2024, 48(3): 93-102. Li Yinsheng, Wang Bing, Chen Yuquan, et al.Bounded L2-gain load frequency control for islanded microgrid based on two-player zero-sum game[J]. Automation of Electric Power Systems, 2024, 48(3): 93-102. [21] Lin Liquan, Huang Jie.Distributed adaptive cooperative optimal output regulation via integral reinforcement learning[J]. Automatica, 2024, 170: 111861. [22] Kleinman D.On an iterative technique for Riccati equation computations[J]. IEEE Transactions on Automatic Control, 1968, 13(1): 114-115. [23] Jiao Qiang, Modares H, Xu Shengyuan, et al.Multi-agent zero-sum differential graphical games for disturbance rejection in distributed control[J]. Automatica, 2016, 69: 24-34. [24] Khalil H K, Praly L.High-gain observers in nonlinear feedback control[J]. International Journal of Robust and Nonlinear Control, 2014, 24(6): 993-1015. [25] 孔繁镍, 李啸骢, 吴杰康, 等. 基于尼科尔斯PID设计方法的负荷频率控制[J]. 中国电机工程学报, 2012, 32(22): 79-85, 15. Kong Fannie, Li Xiaocong, Wu Jiekang, et al.Design of Nichols PID controller for load frequency control[J]. Proceedings of the CSEE, 2012, 32(22): 79-85, 15. [26] Cao Yijia, Wang Weiyu, Li Yong, et al.A virtual synchronous generator control strategy for VSC-MTDC systems[J]. IEEE Transactions on Energy Conversion, 2018, 33(2): 750-761. [27] Kundur P, Balu N, Lauby M.Power System Stability and Control[M]. New York: McGraw-Hill, 1994. [28] Wang W, Beddard A, Barnes M, et al.Analysis of active power control for VSC-HVDC[J]. IEEE Transactions on Power Delivery, 2014, 29(4): 1978-1988.
2026, Vol. 41 
