基于积分强化学习的构网型VSC综合频率控制

doi:10.19595/j.cnki.1000-6753.tces.250670

摘要
图/表
参考文献
相关文章 (15)

全文: PDF (2005 KB)
输出: BibTeX | EndNote (RIS)

摘要柔性直流系统电压源换流器（VSC）具有快速功率调节能力,可为受端电网提供频率支撑。该文在构网型虚拟同步机控制框架的基础上,针对系统模型未知的场景,提出了一种基于积分强化学习和主动负荷扰动补偿的VSC综合频率控制方法。首先,推导了虚拟同步机控制的VSC与目标电网的等效模型,并对系统的振荡模态进行分析。为了抑制反相频率振荡,引入了基于二次型性能指标的最优频率控制方法。进一步地,设计了基于脉冲摄动激励的策略迭代型积分强化学习算法,用于实现无模型最优频率控制问题的在线高精度求解。此外,为了提高VSC对负荷扰动的主动响应能力,实现快速二次调频,设计了降阶的负荷扰动观测器和相应的补偿控制。最后,搭建了两个仿真测试系统,对综合频率控制方法的性能进行验证。仿真结果表明所提方法能有效抑制负荷扰动引起的频率与功率振荡,提高系统的动、稳态性能。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	李寅生
	王冰
	陈玉全
	王万成

关键词 ：柔性直流输电, 构网型控制, 虚拟同步机, 积分强化学习, 负荷扰动补偿

Abstract：The voltage source converter based high voltage direct current (VSC-HVDC) systems featuring grid-forming (GFM) control can provide critical frequency and voltage support to weak or passive grids, making them a key research focus in modern power systems. This paper proposes a model-free integrated frequency control method for grid-forming voltage source converters (GFM-VSCs) within a virtual synchronous generator (VSG) framework. The method integrates the integral reinforcement learning algorithm-based optimal frequency control with active load disturbance compensation to deliver reliable frequency support to the target AC grid.
Firstly, the dynamic models of the target AC grid and the GFM-VSC were established, and the oscillatory modes present in their coupled operation were analyzed. Subsequently, an optimal frequency controller was designed based on a predefined quadratic performance index to improve system damping, independent of the VSG parameters. To address unknown model dynamics, a policy iteration algorithm utilizing impulse perturbation excitation was developed under an integral reinforcement learning framework, to solve the optimal frequency controller in a fully data-driven manner. Furthermore, a reduced order load disturbance observer was designed, along with a disturbance compensation controller. This ancillary disturbance compensation module was used to enhance the VSC's active response to load disturbances, enable rapid secondary frequency regulation, and operate in coordination with the optimal frequency controller trained by the policy iteration algorithm.
The performance of the proposed method was evaluated using an equivalent system test model and the IEEE standard 9-Bus test model, both implemented in MATLAB/Simulink. Comparative analysis included three alternative approaches: the VSG with a Nichols-PI regulator, the VSG with a V²-P-w droop regulator, and the traditional VSG scheme. During online training, the Critic neural network parameters converged within 10 iterations. The proposed impulse perturbation excitation method reduced the error norm of the Critic parameter matrix to below 0.24, demonstrating higher accuracy than traditional noise based excitation method, which achieved only 1.64. Under a 50 MW step load disturbance, the integrated frequency control method limited frequency deviations to within 0.001 p.u. in the equivalent system and 0.0015 p.u. in the IEEE 9-Bus system. The grid frequency was restored to the nominal value within 1.5 s. The proposed method exhibited significantly better frequency support performance than the three benchmark methods. It also provided faster rate of change of frequency (RoCoF) attenuation and notably suppressed frequency and power oscillations. Under a heavy load test scenario (exceeding 85% of the rated power), the method successfully restricted the active power reference command to the predefined safety limit. This ability protected the VSC from overload beyond safe durations.
Based on the simulation analysis, the following conclusions can be drawn: (1) The policy iteration algorithm incorporating impulse perturbation excitation demonstrates accelerated convergence rate and enhanced precision compared to the traditional noise based excitation methods. (2) The proposed integrated frequency control method demonstrates superior damping performance under load disturbances, significantly attenuating RoCoF, as well as frequency and power oscillations. (3) The proposed integrated frequency control method effectively compensates load-side disturbances and ensures prompt restoration of the grid frequency to its nominal value.

Key words： Voltage source converter based high voltage direct current (VSC-HVDC) grid-forming control virtual synchronous generator integral reinforcement learning load disturbance compensation

收稿日期: 2025-04-22

PACS:

TM732

基金资助:国家重点研发计划项目（2022YFB4201303）和国家自然科学基金项目（62303158）资助

通讯作者: 王冰男,1975年生,教授,博士研究生导师,研究方向为可再生能源发电控制、多智能体网络控制。E-mail: icekingking@hhu.edu.cn

作者简介: 李寅生男,1995年生,博士研究生,研究方向为新型电力系统分析与控制、强化学习与智能优化控制。E-mail: 15150693623@139.com

引用本文:

李寅生, 王冰, 陈玉全, 王万成. 基于积分强化学习的构网型VSC综合频率控制[J]. 电工技术学报, 0, (): 250670-. Li Yinsheng, Wang Bing, Chen Yuquan, Wang Wancheng. Integrated Frequency Control for Grid-forming VSC Based on Integral Reinforcement Learning. Transactions of China Electrotechnical Society, 0, (): 250670-.

链接本文:

https://dgjsxb.ces-transaction.com/CN/10.19595/j.cnki.1000-6753.tces.250670 https://dgjsxb.ces-transaction.com/CN/Y0/V/I/250670

[1] 詹长江, 吴恒, 王雄飞, 等. 构网型变流器稳定性研究综述[J]. 中国电机工程学报, 2023, 43(6): 2339-2359.
Zhan Changjiang, Wu Heng, Wang Xiongfei, et al.An overview of stability studies of grid-forming voltage source converters[J]. Proceedings of the CSEE, 2023, 43(6): 2339-2359.
[2] 韩丽, 陈硕, 王施琪, 等. 考虑风光消纳与电动汽车灵活性的调度策略[J]. 电工技术学报, 2024, 39(21): 6793-6803.
Han Li, Chen Shuo, Wang Shiqi, et al.Scheduling strategy considering wind and photovoltaic power consumption and the flexibility of electric vehicles[J]. Transactions of China Electrotechnical Society, 2024, 39(21): 6793-6803.
[3] 姜崇学, 马秀达, 邹强, 等. 柔性直流输电系统的高频谐波保护方法与工程实践[J]. 电力系统自动化, 2024, 48(3): 150-158.
Jiang Chongxue, Ma Xiuda, Zou Qiang, et al.High-frequency harmonic protection methods and engineering practice for flexible DC transmission systems[J]. Automation of Electric Power Systems, 2024, 48(3): 150-158.
[4] 王炜宇, 张一平, 李帅虎, 等. 基于扰动观测的构网型柔直系统阻尼控制器设计[J]. 电网技术, 2024, 48(6): 2262-2271.
Wang Weiyu, Zhang Yiping, Li Shuaihu, et al.Disturbance observer-based damping controller of grid-forming VSC-HVDC systems[J]. Power System Technology, 2024, 48(6): 2262-2271.
[5] 于国星, 宋蕙慧, 马广富, 等. 含海上风电场的VSC-MTDC系统参与电网调频的顺序控制方法[J]. 电力系统自动化, 2021, 45(4): 123-132.
Yu Guoxing, Song Huihui, Ma Guangfu, et al.Sequence control method for VSC-MTDC system with offshore wind farm participating in frequency regulation of power grid[J]. Automation of Electric Power Systems, 2021, 45(4): 123-132.
[6] 姚伟, 熊永新, 姚雅涵, 等. 海上风电柔直并网系统调频控制综述[J]. 高电压技术, 2021, 47(10): 3397-3413.
Yao Wei, Xiong Yongxin, Yao Yahan, et al.Review of voltage source converter-based high voltage direct current integrated offshore wind farm on providing frequency support control[J]. High Voltage Engineering, 2021, 47(10): 3397-3413.
[7] 谭珺敉, 彭晓涛, 李旭涛, 等. 基于协同控制优化风电-柔直并网惯性响应策略研究[J]. 电工技术学报, 2025, 40(5): 1355-1367.
Tan Junmi, Peng Xiaotao, Li Xutao, et al.Optimization of inertia response strategy based on synergetic control for wind power integrating to power grid via VSC-HVDC[J]. Transactions of China Electrotechnical Society, 2025, 40(5): 1355-1367.
[8] 王炜宇, 李勇, 曹一家, 等. 基于虚拟调速器的多端直流虚拟同步机控制策略[J]. 中国电机工程学报, 2018, 38(12): 3461-3470, 5.
Wang Weiyu, Li Yong, Cao Yijia, et al.The virtual synchronous generator technology based on virtual governor for multi-terminal direct current system[J]. Proceedings of the CSEE, 2018, 38(12): 3461-3470, 5.
[9] 葛平娟, 肖凡, 涂春鸣, 等. 考虑故障限流的下垂控制型逆变器暂态控制策略[J]. 电工技术学报, 2022, 37(14): 3676-3687.
Ge Pingjuan, Xiao Fan, Tu Chunming, et al.Transient control strategy of droop-controlled inverter considering fault current limitation[J]. Transactions of China Electrotechnical Society, 2022, 37(14): 3676-3687.
[10] 刘英培, 谢乾, 梁海平. 柔性直流输电系统自适应虚拟惯性调频控制策略[J]. 电力系统自动化, 2021, 45(5): 129-136.
Liu Yingpei, Xie Qian, Liang Haiping.Frequency regulation control strategy for flexible DC transmission system based on adaptive virtual inertia[J]. Automation of Electric Power Systems, 2021, 45(5): 129-136.
[11] 刘辉, 于思奇, 孙大卫, 等. 构网型变流器控制技术及原理综述[J]. 中国电机工程学报, 2025, 45(1): 277-297.
Liu Hui, Yu Siqi, Sun Dawei, et al.An overview of control technologies and principles for grid-forming converters[J]. Proceedings of the CSEE, 2025, 45(1): 277-297.
[12] 徐菘, 杨博, 刘浩, 等. 一种提高虚拟同步机电流质量的电压-电流级联闭环控制方案[J]. 电工技术学报, 2024, 39(6): 1871-1885.
Xu Song, Yang Bo, Liu Hao, et al.A cascaded harmonic voltage and current closed-loop control method to improve the current quality of virtual synchronous generators[J]. Transactions of China Electrotechnical Society, 2024, 39(6): 1871-1885.
[13] 刘思佳, 刘海涛, 张隽, 等. 基于等效阻抗的虚拟同步机电压支撑影响因素分析与改进控制策略研究[J]. 电工技术学报, 2025, 40(9): 2738-2751.
Liu Sijia, Liu Haitao, Zhang Jun, et al.Research on the analysis of virtual synchronous generator voltage support influence factors and improvement control strategies based on equivalent impedance[J]. Transactions of China Electrotechnical Society, 2025, 40(9): 2738-2751.
[14] 马秀达, 卢宇, 田杰, 等. 柔性直流输电系统的构网型控制关键技术与挑战[J]. 电力系统自动化, 2023, 47(3): 1-11.
Ma Xiuda, Lu Yu, Tian Jie, et al.Key technologies and challenges of grid-forming control for flexible DC transmission system[J]. Automation of Electric Power Systems, 2023, 47(3): 1-11.
[15] 高本锋, 沈雨思, 宋瑞华, 等. 虚拟同步机控制模块化多电平变流器阻抗建模及次/超同步振荡稳定性分析[J]. 电工技术学报, 2025, 40(2): 559-573.
Gao Benfeng, Shen Yusi, Song Ruihua, et al.Impedance modeling and sub/super synchronous oscillation stability analysis of modular multilevel converter under virtual synchronous generator control[J]. Transactions of China Electrotechnical Society, 2025, 40(2): 559-573.
[16] 姚为正, 杨美娟, 张海龙, 等. VSC-HVDC受端换流器参与电网调频的VSG控制及其改进算法[J]. 中国电机工程学报, 2017, 37(2): 525-534.
Yao Weizheng, Yang Meijuan, Zhang Hailong, et al.VSG control and its modified algorithm for VSC-HVDC inverter participating grid’s frequency regulation[J]. Proceedings of the CSEE, 2017, 37(2): 525-534.
[17] 王炜宇, 易念棋, 蔡晔, 等. 基于功率扰动观测的构网型柔直系统辅助频率控制策略[J]. 中国电机工程学报, 2025, 45(5): 1680-1691.
Wang Weiyu, Yi Nianqi, Cai Ye, et al.A power-disturbance observer-based auxiliary frequency control of grid-forming VSC-HVDC systems[J]. Proceedings of the CSEE, 2025, 45(5): 1680-1691.
[18] Leon A E, Mauricio J M.Virtual synchronous generator for VSC-HVDC stations with DC voltage control[J]. IEEE Transactions on Power Systems, 2023, 38(1): 728-738.
[19] Saad H, Peralta J, Dennetière S, et al.Dynamic averaged and simplified models for MMC-based HVDC transmission systems[J]. IEEE Transactions on Power Delivery, 2013, 28(3): 1723-1730.
[20] 李寅生, 王冰, 陈玉全, 等. 基于双人零和博弈的孤岛微电网有界L₂增益负荷频率控制[J]. 电力系统自动化, 2024, 48(3): 93-102.
Li Yinsheng, Wang Bing, Chen Yuquan, et al.Bounded L₂-gain load frequency control for islanded microgrid based on two-player zero-sum game[J]. Automation of Electric Power Systems, 2024, 48(3): 93-102.
[21] Lin Liquan, Huang Jie.Distributed adaptive cooperative optimal output regulation via integral reinforcement learning[J]. Automatica, 2024, 170: 111861.
[22] Kleinman D.On an iterative technique for Riccati equation computations[J]. IEEE Transactions on Automatic Control, 1968, 13(1): 114-115.
[23] Jiao Qiang, Modares H, Xu Shengyuan, et al.Multi-agent zero-sum differential graphical games for disturbance rejection in distributed control[J]. Automatica, 2016, 69: 24-34.
[24] Khalil H K, Praly L.High-gain observers in nonlinear feedback control[J]. International Journal of Robust and Nonlinear Control, 2014, 24(6): 993-1015.
[25] 孔繁镍, 李啸骢, 吴杰康, 等. 基于尼科尔斯PID设计方法的负荷频率控制[J]. 中国电机工程学报, 2012, 32(22): 79-85, 15.
Kong Fannie, Li Xiaocong, Wu Jiekang, et al.Design of nichols PID controller for load frequency control[J]. Proceedings of the CSEE, 2012, 32(22): 79-85, 15.
[26] Cao Yijia, Wang Weiyu, Li Yong, et al.A virtual synchronous generator control strategy for VSC-MTDC systems[J]. IEEE Transactions on Energy Conversion, 2018, 33(2): 750-761.
[27] Kundur P, Balu N, Lauby M.Power system stability and control[M]. New York: McGraw-hill, 1994.
[28] Wang W, Beddard A, Barnes M, et al.Analysis of active power control for VSC-HVDC[J]. IEEE Transactions on Power Delivery, 2014, 29(4): 1978-1988.