多源异构的智能配用电数据存储处理技术

摘要
图/表
参考文献
相关文章 (5)

全文: PDF (36132 KB)
输出: BibTeX | EndNote (RIS)

摘要针对智能配用电数据具有海量、多源异构等特点,提出一种基于Hadoop的智能配用电数据存储处理的框架设计方案。在对智能配用电数据组成进行梳理的基础上,提出了智能配用电的资源、存储和查询三层数据存储框架。其中,资源层采用Hadoop集群架构,结合电力系统的行业特色,完成IT存储资源管理;存储层则先采用XML技术进行数据预处理,使非结构化数据归一化,然后利用NoSQL数据库通过Map和Reduce的有效结合,完成海量数据的快速存储;查询层利用Top-k检索技术,实现海量数据的快速查寻。该设计方案能够对结构化和非结构化数据进行统一存储,可为智能配用电大数据技术的应用提供基础支撑。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	葛磊蛟
	王守相
	王尧
	郭乃网

关键词 ：大数据, 海量数据, 框架设计, 智能配用电

Abstract：A Hadoop-based management framework design is proposed for massive and multi-source isomerized data in smart power distribution and utilization. This framework consists of resource layer, storage layer and query layer, based on the data composition of smart power distribution and utilization. The resource layer uses Hadoop-cluster architecture. Combined with the characteristics of power system, the management of IT storage resource is completed. The storage layer firstly adopts XML technology framework to preprocess unstructured data aiming to normalize the isomerized data, and then the fast storage process of mass data can be done through the effective combination of Map and Reduce based on NoSQL. The query layer is able to fast achieve the searching of mass data through the implementation of Top-k technology. The proposed scheme can give a unified treatment to the storage and processing of structured and unstructured data, and provide a basic support for the application of big data technology for smart power distribution and utilization.

Key words： Big data massive data framework design smart power distribution and utilization

收稿日期: 2015-09-18 出版日期: 2016-08-18

PACS:

TM711

基金资助:国家高技术研究发展计划（863计划）资助项目（2015AA050203）

作者简介: 葛磊蛟男,1984年生,博士,讲师,研究方向为智能配用电和不确定性仿真、云计算和大数据技术等。E-mail: legendglj99@tju.edu.cn（通信作者）;王守相男,1973年生,教授,博士生导师,研究方向为智能电网、分布式发电等。E-mail: sxwang@tju.edu.cn

引用本文:

葛磊蛟, 王守相, 王尧, 郭乃网. 多源异构的智能配用电数据存储处理技术[J]. 电工技术学报, 2015, 30(增刊2): 159-168. Ge Leijiao, Wang Shouxiang, Wang Yao, Guo Naiwang. Storage and Processing Technology of the Multi-Source Isomerized Data for Smart Power Distribution and Utilization. Transactions of China Electrotechnical Society, 2015, 30(增刊2): 159-168.

链接本文:

https://dgjsxb.ces-transaction.com/CN/Y2015/V30/I增刊2/159

[1] 王守相, 王成山. 现代配电系统分析[M]. 2版. 北京: 高等教育出版社, 2014.
[2] 何颖鹏. 非结构化数据统一存储平台的设计与实现[D]. 杭州: 浙江大学, 2013.
[3] 余斌. 海量非结构化数据分布式分析与检索[D]. 杭州: 浙江大学, 2012.
[4] 曲朝阳, 陈帅, 杨帆, 等. 基于云计算技术的电力大数据预处理属性简约方法[J]. 电力系统自动化, 2014, 38(8): 67-71.
Qu Zhaoyang, Chen Shuai, Yang Fan, et al. An attribute reducing method for electric power big data preprocessing based on cloud computing techno- logy[J]. Automation of Electric Power Systems, 2014, 38(8): 67-71.
[5] 丁杰, 朱力鹏, 胡斌, 等. 面向多级调度管理的融合型搜索引擎[J]. 电力系统自动化, 2014, 38(3): 150-154.
Ding Jie, Zhu Lipeng, Hu Bin, et al. A fusion-type search engine for multilevel scheduling manage- ment[J]. Automation of Electric Power Systems, 2014, 38(3): 150-154.
[6] 屈志坚, 郭亮, 陈秋琳, 等. Hadoop 云构架的智能调度无损集群压缩技术[J]. 电力系统自动化, 2013, 37(18): 93-98.
Qu Zhijian, Guo Liang, Chen Qiulin, et al. Intelligent dispatching lossless cluster compression technology based on Hadoop cloud framework[J]. Automation of Electric Power Systems, 2013, 37(18): 93-98.
[7] 刘道新, 胡航海, 张健, 等. 大数据全生命周期中关键问题研究及应用[J]. 中国电机工程学报, 2015, 35(1): 23-28.
Liu Daoxin, Hu Hanghai, Zhang Jian, et al. Research on key issues of big data lifecycle and its application[J]. Proceeding of the CSEE, 2015, 35(1): 23-28.
[8] He G, Ren S Y, Yu D C, et al. Analysis of enterprise user behavior on hadoop[C]//6th International Con- ference on Intelligent Human-Machine Systems and Cybernetics, Hangzhou, 2014: 230-233.
[9] 屈志坚, 郭亮, 刘明光, 等. 智能配电网量测信息变断面柔性压缩新算法[J]. 中国电机工程学报, 2013, 33(19): 191-199.
Qu Zhijian, Guo Liang, Liu Mingguang, et al. New variable section flexible compression algorithm for measurement information in intelligent distribution network[J]. Proceeding of the CSEE, 2013, 33(19): 191-199.
[10] 王德文. 基于云计算的电力数据中心基础架构及其关键技术[J]. 电力系统自动化, 2012, 36(11): 67-71, 107.
Wang Dewen. Basic framework and key technology for a new generation of data center in electric power corporation based on cloud computation[J]. Auto- mation of Electric Power Systems, 2012, 36(11): 67-71, 107.
[11] 高涛. 基于云计算的配电自动化系统设计[J]. 电气技术, 2014, 15(7): 103-105.
Gao Tao. Design of distribution automation system based on cloud computing[J]. Electrical Technique, 2014, 15(7): 103-105.
[12] 王守相, 张卫涛, 葛磊蛟. 智能配电网统一模型云与应用服务技术系统[J]. 电力自动化设备, 2015, 35(2): 49-54.
Wang Shouxiang, Zhang Weitao, Ge Leijiao. Unified model cloud and application service system for smart distribution network[J]. Electric Power Automation Equipment, 2015, 35(2): 49-54.
[13] 王保义, 赵硕, 张少敏. 基于云计算和极限学习机的分布式电力负荷预测算法[J]. 电网技术, 2014, 38(2): 526-531.
Wang Baoyi, Zhao Shuo, Zhang Shaomin. A distributed load forecasting algorithm based on cloud computing and extreme learning machine[J]. Power System Technology, 2014, 38(2): 526-531.
[14] 宋亚奇, 周国亮, 朱永利, 等. 云平台下输变电设备状态监测大数据存储优化与并行处理[J]. 中国电机工程学报, 2015, 35(2): 255-266.
Song Yaqi, Zhou Guoliang, Zhu Yongli, et al. Storage optimization and parallel processing of condition monitoring big data of transmission and transforming equipment based on cloud platform[J]. Proceedings of the CSEE, 2015, 35(2): 255-266.
[15] 曲朝阳, 朱莉, 张士林. 基于Hadoop的广域测量系统数据处理[J]. 电力系统自动化, 2013, 37(4): 92-97.
Qu Zhaoyang, Zhu Li, Zhang Shilin. Data processing of Hadoop based wide area measurement system[J]. Automation of Electric Power Systems, 2013, 37(4): 92-97.
[16] Anam A, Jamil A. Hadoop architecture and its issues[C]//International Conference on Computa- tional Science and Computational Intelligence, Las Vegas, NV, 2014: 288-291.
[17] Sadasivam G S, Dharini S. A novel parallel hybrid PSO-GA using MapReduce to schedule jobs in Hadoop data grids[C]//Second World Congress on Nature and Biologically Inspired Computing, Fukuoka, 2010: 377-382.
[18] 宋亚奇, 周国亮, 朱永利. 智能电网大数据处理技术现状与挑战[J]. 电网技术, 2013, 37(4): 927-935.
Song Yaqi, Zhou Guoliang, Zhu Yongli. Present status and challenges of big data processing in smart grid[J]. Power System Technology, 2013, 37(4): 927-935.
[19] 赵腾, 张焰, 张东霞. 智能配电网大数据应用技术与前景分析[J]. 电网技术, 2014, 38(12): 3305-3312.
Zhao Teng, Zhang Yan, Zhang Dongxia. Application technology of big data in smart distribution grid and its prospect analysis[J]. Power System Technology, 2014, 38(12): 3305-3312.
[20] 刘巍, 黄曌, 李鹏, 等. 面向智能配电网的大数据统一支持平台体系与架构[J]. 电工技术学报, 2014, 29(增1): 486-491.
Liu Wei, Huang Zhao, Li Peng, et al. Summary about system and framework of unified supporting platform of big data for smart distribution grid[J]. Transa- ctions of China Electrotechnical Society, 2014, 29(S1): 486-491.
[21] 周国亮, 朱永利, 王桂兰, 等. 实时大数据处理技术在状态监测领域中的应用[J]. 电工技术学报, 2014, 29(增1): 432-437.
Zhou Guoliang, Zhu Yongli, Wang Guilan, et al. Real-time big data processing technology application in the field of state monitoring[J]. Transactions of China Electrotechnical Society, 2014, 29(S1): 432- 437
[22] 韩璞, 袁世通. 基于大数据和双量子粒子群算法的多变量系统辨识[J]. 中国电机工程学报, 2014, 34(32): 5779-5787.
Han Pu, Yuan Shitong. Multivariable system identification based on double quantum particle swarm optimization and big data[J]. Proceedings of the CSEE, 2014, 34(32): 5779-5787.
[23] Labrinidis A, Jagadish H V. Challenges and oppor- tunities with big data[J]. Proceedings of the VLDB Endowment, 2012, 5(12): 2032-2033.
[24] Kamalpreet S, Ravinder K. Hadoop: addressing challenges of big data[C]//IEEE International Advance Computing Conference, Gurgaon, 2014: 686-689.
[25] 刘伯颖, 李志刚, 黄晓勤, 等. 复杂系统的可靠性优化分配方法及应用[J]. 电工技术学报, 2014, 29(增1): 497-501.
Liu Boying, Li Zhigang, Huang Xiaoqin, et al. Reliabiliy optimal allocation methods and application of complex systems[J]. Transactions of China Elec- trotechnical Society, 2014, 29(S1): 497-501.
[26] 熊小伏, 陈星田, 翁世杰. 支持大数据分析的发电厂变电站全息录波方法[J]. 电力系统保护与控制, 2015, 43(22): 17-22.
Xiong Xiaofu, Chen Xingtian, Weng Shijie. A holographic record method supporting big data analysis for power plant and substation[J]. Power System Protection and Control, 2015, 43(22): 17-22.
[27] 杨永标, 周立秋, 丁孝华, 等. 智能配用电园区技术集成方案[J]. 电力系统自动化, 2012, 36(10): 74- 78.
Yang Yongbiao, Zhou Liqiu, Ding Xiaohua, et al. Technology integration scheme of smart power distribution and utilization park[J]. Automation of Electric Power Systems, 2012, 36(10): 74-78.
[28] 杨凯, 余高旺, 宋勇辉, 等. 智能变电站冗余数据处理方案的研究与应用[J]. 电力系统保护与控制, 2015, 43(12): 150-154.
Yang Kai, Yu Gaowang, Song Yonghui, et al. Research and application of redundant data pro- cessing scheme for smart substation[J]. Power System Protection and Control, 2015, 43(12): 150- 154.
[29] 唐志军, 邹贵彬, 高厚磊, 等. 含分布式电源的智能配电网保护控制方案[J]. 电力系统保护与控制, 2014, 42(8): 9-14.
Tang Zhijun, Zou Guibin, Gao Houlei, et al. Protection and control scheme for smart distribution grid with distribution resource[J]. Power System Protection and Control, 2014, 42(8): 9-14.
[30] Li W, Lang B. A tetrahedral data model for unstructured data management[J]. Science China Information Sciences, 2010, 53(8): 1497-1510.
[31] Amin A, Farshad K, Reza A. A preliminary study of incorporating GPUs in the Hadoop framework[C]// The 16th CSI International Symposium on Computer Architecture and Digital Systems, Shiraz, Fars, 2012: 178-185.
[32] Jungkyu H, Masakuni I, Hiroyuki M. A Hadoop performance model for multi-rack clusters[C]//5th International Conference on Computer Science and Information Technology, Amman, 2013: 265-274.
[33] Muhammad A, Muhammad A U, Samreen A, et al. Framework for analysis of power system operation in smart cities[J]. Wireless Pers Commun, 2014, 76(3): 399-408.
[34] Yu Y X, Zeng Y, Liu H, et al. Challenges and R&D opportunities of smart distribution grids in China[J]. Science China: Technological Sciences, 2014, 57(8): 1588-1593.
[35] Elias B, Claude F, Makan P, et al. Communication security for smart grid distribution networks[J]. IEEE Communication Magazine, 2013, 51(1): 42-49.
[36] Bharadwaj R S, Gerald T H. Sensitivity based pricing and optimal storage utilization in distribution systems[J]. IEEE Transactions on Power Delivery, 2013, 28(2): 1073-1082.
[37] 欧阳柳波, 李学勇, 杨贯中, 等. 基于近似匹配模型的XML元数据检索[J]. 计算机应用, 2005, 25(4): 820-823, 826.
Ouyang Liubo, Li Xueyong, Yang Guanzhong, et al. XML metadata retrieval based on approximately matching model[J]. Computer Application, 2005, 25(4): 820-823, 826.