Optimal Energy Management of Residential Hybrid Energy System by Embedding Domain Knowledge into Deep Reinforcement Learning
Zhao Liyuan1,2, Li Jinze1,2, Zhang Xian1,2, Chen Ting1,2
1. State Key Laboratory of Smart Power Distribution Equipment and System Hebei University of Technology Tianjin 300401 China; 2. Key Laboratory of Electromagnetic Field and Electrical Apparatus Reliability of Hebei Province Hebei University of Technology Tianjin 300401 China
Abstract：To cope with the operational differences of equipment and complex uncertainties in residential hybrid energy system, this paper proposes a residential energy optimization management method by embedding domain knowledge into deep reinforcement learning (DRL). The method can achieve joint optimization management of different types of equipment in residential hybrid energy system, and embed domain knowledge rules into the deep reinforcement learning framework to improve training efficiency. Firstly, an optimal operation model of residential hybrid energy system including residential electrical appliances and gas equipment was constructed, and optimal knowledge rules for residential equipment were designed based on system energy management objectives. Then, to cope with the uncertainties of renewable energy output and user demand in the system, a deep reinforcement learning optimization model of residential hybrid energy system was constructed, and the proximal policy optimization (PPO) method based on discrete-continuous hybrid strategy was used to make system energy management decisions. Furthermore, a residential hybrid energy system energy optimization management framework that embeds domain knowledge into deep reinforcement learning was constructed. By embedding domain knowledge into the training process of deep reinforcement learning, the advantage of deep reinforcement learning method in efficiently extracting domain optimization knowledge was fully utilized, improving the training efficiency of residential energy optimization strategy. Accordingly, a linkage training mechanism based on exponential probability function was developed to coordinate the probabilities of the random exploration, PPO exploration and knowledge-based exploration. Finally, the effectiveness and superiority of the proposed method were verified through simulation results. Simulation results show that the domain knowledge embedded PPO method improves the training efficiency by 72.5% and reduces the training time by 94.1 minutes compared with the conventional proximal policy optimization method, which verifies the effectiveness of the proposed method in improving the training efficiency of residential optimization strategy. The residential energy optimization management results under different test days are analyzed to verify that the proposed method can adapt to system uncertainties. By making real-time energy optimization management decisions for residential gas/electricity equipment, users' energy cost can be reduced while ensuring their thermal and visual comfort. The energy optimization management results of different methods show that the total cost of the proposed method is reduced by 3.47%, 14.74% and 15.59% compared with the PPO method, DQN method and PSO-LSTM method, respectively. The simulation analysis draws the following conclusions: (1) By embedding domain knowledge into the development and training of the PPO framework, the training efficiency of the residential optimization strategy can be improved. (2) The proposed method avoids the discretization of continuous actions in the system and the prediction of uncertainty factors, further improving the real-time optimization performance of residential hybrid energy system. (3) The proposed method can flexibly use heterogeneous energy to meet the multi-type load demands of residential users based on external energy price signals, further reducing the energy cost of users.
赵黎媛, 李金泽, 张献, 陈婷. 领域知识嵌入深度强化学习的住宅混合能源系统能量优化管理[J]. 电工技术学报, 2026, 41(5): 1724-1737.
Zhao Liyuan, Li Jinze, Zhang Xian, Chen Ting. Optimal Energy Management of Residential Hybrid Energy System by Embedding Domain Knowledge into Deep Reinforcement Learning. Transactions of China Electrotechnical Society, 2026, 41(5): 1724-1737.
[1] Wang Zixuan, Xiao Fu, Ran Yi, et al.Scalable energy management approach of residential hybrid energy system using multi-agent deep reinforcement learning[J]. Applied Energy, 2024, 367: 123414. [2] 杨雪莹, 祁琪, 李启明, 等. 奖励机制与用户意愿结合的高峰期负荷博弈调度策略[J]. 电工技术学报, 2024, 39(16): 5060-5074. Yang Xueying, Qi Qi, Li Qiming, et al.Peak load game scheduling strategy combining reward mechanism and user willingness[J]. Transactions of China Electrotechnical Society, 2024, 39(16): 5060-5074. [3] Zhao Liyuan, Yang Ting, Li Wei, et al.Deep reinforcement learning-based joint load scheduling for household multi-energy system[J]. Applied Energy, 2022, 324: 119346. [4] Wang Jidong, Liu Jianxin, Li Chenghao, et al.Optimal scheduling of gas and electricity consumption in a smart home with a hybrid gas boiler and electric heating system[J]. Energy, 2020, 204: 117951. [5] 李延珍, 王海鑫, 杨子豪, 等. 基于非侵入式负荷分解的家庭负荷两阶段超短期负荷预测模型[J]. 电工技术学报, 2024, 39(11): 3379-3391. Li Yanzhen, Wang Haixin, Yang Zihao, et al.Two-stage ultra-short-term load forecasting model of household appliances based on non-intrusive load disaggregation[J]. Transactions of China Electrotech-nical Society, 2024, 39(11): 3379-3391. [6] Hosseini S M, Carli R, Dotoli M.Robust optimal energy management of a residential microgrid under uncertainties on demand and renewable power generation[J]. IEEE Transactions on Automation Science and Engineering, 2021, 18(2): 618-637. [7] Javadi M S, Gough M, Lotfi M, et al.Optimal self-scheduling of home energy management system in the presence of photovoltaic power generation and batteries[J]. Energy, 2020, 210: 118568. [8] Yamamoto S, Furukakoi M, Uehara A, et al.MPC-based robust optimization of smart apartment building considering uncertainty for conservative reduction[J]. Energy and Buildings, 2024, 318: 114461. [9] Xiong Shengtao, Liu Dehong, Chen Yuan, et al.A deep reinforcement learning approach based energy management strategy for home energy system considering the time-of-use price and real-time control of energy storage system[J]. Energy Reports, 2024, 11: 3501-3508. [10] Zhang Yiwen, Lin Rui, Mei Zhen, et al.Interior-point policy optimization based multi-agent deep reinfo-rcement learning method for secure home energy management under various uncertainties[J]. Applied Energy, 2024, 376: 124155. [11] 张薇, 王浚宇, 杨茂, 等. 基于分布式双层强化学习的区域综合能源系统多时间尺度优化调度[J]. 电工技术学报, 2025, 40(11): 3529-3544. Zhang Wei, Wang Junyu, Yang Mao, et al.The multi-time-scale optimal scheduling for regional integrated energy system based on the distributed bi-layer reinforcement learning[J]. Transactions of China Electrotechnical Society, 2025, 40(11): 3529-3544. [12] 李鹏, 钟瀚明, 马红伟, 等. 基于深度强化学习的有源配电网多时间尺度源荷储协同优化调控[J]. 电工技术学报, 2025, 40(5): 1487-1502. Li Peng, Zhong Hanming, Ma Hongwei, et al.Multi-timescale optimal dispatch of source-load-storage coordination in active distribution network based on deep reinforcement learning[J]. Transactions of China Electrotechnical Society, 2025, 40(5): 1487-1502. [13] 张磊, 吴红斌, 何叶, 等. 基于深度强化学习的氢能综合能源系统优化调度方法[J]. 电力系统自动化, 2024, 48(16): 132-141. Zhang Lei, Wu Hongbin, He Ye, et al.Optimal scheduling method for integrated energy systems with hydrogen based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2024, 48(16): 132-141. [14] 张甜, 赵奇, 陈中, 等. 基于深度强化学习的家庭能量管理分层优化策略[J]. 电力系统自动化, 2021, 45(21): 149-158. Zhang Tian, Zhao Qi, Chen Zhong, et al.Hierarchical optimization strategy for home energy management based on deep reinforcement learning[J]. Automation of Electric Power Systems, 2021, 45(21): 149-158. [15] 侯慧, 陈跃, 吴细秀, 等. 非预测机制下计及碳交易的家庭能量低碳优化实时管理[J]. 电网技术, 2023, 47(3): 1066-1078. Hou Hui, Chen Yue, Wu Xixiu, et al.Low-carbon optimal real-time management strategy for home energy considering carbon trading under non-prediction mechanisms[J]. Power System Technology, 2023, 47(3): 1066-1078. [16] Ye Yujian, Qiu Dawei, Wu Xiaodong, et al.Model-free real-time autonomous control for a residential multi-energy system using deep reinforcement learning[J]. IEEE Transactions on Smart Grid, 2020, 11(4): 3068-3082. [17] Li Hepeng, Wan Zhiqiang, He Haibo.Real-time residential demand response[J]. IEEE Transactions on Smart Grid, 2020, 11(5): 4144-4154. [18] Lu Qing, Lü Shuaikang, Leng Yajun, et al.Optimal household energy management based on smart residential energy hub considering uncertain behaviors[J]. Energy, 2020, 195: 117052. [19] 程晨, 张永熙, 邓友均, 等. 考虑多重舒适度的分时间尺度家庭能量管理策略[J]. 电力建设, 2024, 45(3): 160-172. Cheng Chen, Zhang Yongxi, Deng Youjun, et al.Different time-scale optimization strategy for home energy management considering multiple comforts[J]. Electric Power Construction, 2024, 45(3): 160-172. [20] Bennett G, Elwell C.Effect of boiler oversizing on efficiency: a dynamic simulation study[J]. Building Services Engineering Research & Technology, 2020, 41(6): 709-726. [21] 潘玺安, 艾欣, 胡俊杰, 等. 考虑网络安全约束的分布式智能电网边云协同优化调度方法[J]. 电工技术学报, 2024, 39(19): 6104-6118. Pan Xi'an, Ai Xin, Hu Junjie, et al.Network security constrained distributed smart grid edge-cloud collaborative optimization scheduling[J]. Transactions of China Electrotechnical Society, 2024, 39(19): 6104-6118. [22] Chen Yurou, Zhang Fengyi, Liu Zhiyong.Adaptive bias-variance trade-off in advantage estimator for Actor-Critic algorithms[J]. Neural Networks, 2024, 169: 764-777. [23] McKenna E, Thomson M. High-resolution stochastic integrated thermal-electrical domestic demand model[J]. Applied Energy, 2016, 165: 445-461. [24] Open Power System Data[DB/OL]. [2025-04-30].https://data.open-power-system-data.org. [25] Xu Xu, Jia Youwei, Xu Yan, et al.A multi-agent reinforcement learning-based data-driven method for home energy management[J]. IEEE Transactions on Smart Grid, 2020, 11(4): 3201-3211.
2026, Vol. 41 
