Abstract Computing performance is one of the key issues existing in the applications of big power data,such as fault diagnosis and prediction.Distributed storage and parallel computing are currently as the effective measures to accelerate the data-intensive applications.This paper describes an open distributed processing service(ODPS)from Ali Cloud,is used to store and accelerate the analytic process of monitoring big data about electrical equipment.Taking the phase resolved partial discharge(PRPD)processing of a partial discharge(PD)signal as example,a method for storing the signal with high sampling rate and time series data,and extracting the feature of the signal through the extended MapReduce model(MR2)of ODPS is proposed in this paper.The paralleled PRPD procedure(ODPS-PRPD)implements amounts of PD signals parallel basic parameters calculation and discharge type recognition,statistics features.To verify the effectiveness of the proposed method,a large number of partial discharge signals of four types from laboratory tests are respectively analyzed on ODPS and Hadoop.Because ODPS-PRPD stores the large amounts of middle data in the primary memory,its computing procedure is much faster.The results show that ODPS-PRPD has obviously better performance in data reliabltity,service anailabilty and cost than that of Hadoop.
Zhu Yongli,Li Li,Song Yaqi等. Storage and Parallel Processing of Big Data of Power Equipment Condition Monitoring on ODPS Platform[J]. Transactions of China Electrotechnical Society, 2017, 32(9): 199-210.
[1] 宋亚奇,周国亮,朱永利.智能电网大数据处理技术现状与挑战[J].电网技术,2013,37(4):927-935. Song Yaqi,Zhou Guoliang,Zhu Yongli.Present status and challenges of big data processing in smart grid[J].Power System Technology,2013,37(4):927-935. [2] Williams J W,Aggour K S,Interrante J,et al.Bridging high velocity and high volume industrial big data through distributed in-memory storage & analytics[C]//IEEE International Conference on Big Data(Big Data),Washington,DC,USA,2014:932-41. [3] Han Liangxiu,Ong H Y.Parallel data intensive applications using MapReduce:a data mining case study in biomedical sciences[J].Cluster Comput,2015,18(1):403-418. [4] Agrawal D,Bernstein P,Bertino E,et al.Challenges and opportunities with big data[J].Proceedings of the VLDB Endowment,2012,5(12):2032-2033. [5] Rob P,Sean D,Robert G,et al.Interpreting the data:parallel analysis with Sawzal[J].Scientific Programming,2005,13(4):277-298. [6] Tom White.Hadoop权威指南[M].2版.曾大聃,周傲英,译.北京:清华大学出版社,2011:260-262. [7] Zhao Yong,Hategan M,Clifford B,et al.Swift:fast,reliable,loosely coupled parallel computation[C]//2007 IEEE Congress on Services,Salt Lake City,UT,USA,2007:199-206. [8] Beynon M D,Kurc T,Catalyurek U,et al.Distributed processing of very large datasets with DataCutter[J].Parallel Computing,2001,27(11):1457-1478. [9] LINQ:The LINQ project[EB/OL].2014-04-19.http://msdn.microsoft.com/netframework/future/linq/. [10] Microsoft Research.Dryad[EB/OL].2013-12-23.http://research.microsoft.com/en-us/projects/Dryad/. [11] Teradata.Teradata homepage[EB/OL].2013-12-23.http: //www.teradata.com/. [12] Vertica.Vertica homepage[EB/OL].2013-12-23.http: //www.vertica.com/. [13] Amazon.Amazon homepage[EB/OL].http://aws.amazon.com/cn/. [14] Aliyun.大数据计算服务ODPS[EB/OL].http://www.aliyun.com/. [15] 宋亚奇,周国亮,朱永利,等.云平台下输变电设备状态监测大数据存储优化与并行处理[J].中国电机工程学报,2015,35(2):255-267. Song Yaqi,Zhou Guoliang,Zhu Yongli,et al.Storage optimization and parallel processing of condition monitoring big data of transmission and transforming equipment based on cloud platform[J].Proceedings of the CSEE,2015,35(2):255-267. [16] Ma Yan,Guo Zhihong,Chen Yufeng,et al.Multi-sourced data storage and index construction for equipment condition assessment[C]//The 6th International Conference on Computational Intelligence and Communi-cation Networks,2014:681-685. [17] 葛磊蛟,王守相,王尧,等.多源异构的智能配用电数据存储处理技术[J].电工技术学报,2015,30(增刊2):159-168. Ge Leijiao,Wang Shouxiang,Wang Yao,et al.Storage and processing technology of the multi-source isomerized data for smart power distribution and utilization[J].Transactions of China Electrotechnical Society,2015,30(S2):159-168. [18] Kawasoe S,Igarashi Y,Shibayama K,et al.Examples of distributed information platforms constructed by power utilities in Japan[C]//44th International Conference on Large High Voltage Electric Systems,Paris,France,2012:108-113. [19] 宋亚奇,周国亮,朱永利,等.云平台下并行总体经验模态分解局部放电信号去噪方法研究[J].电工技术学报,2015,30(18):213-222. Song Yaqi,Zhou Guoliang,Zhu Yongli,et al.Research on parallel ensemble empirical mode decomposition denoising method for partial discharge signals[J].Transactions of China Electrotechnical Society,2015,30(18):213-222. [20] 屈志坚,郭亮,刘明光,等.智能配电网量测信息变断面柔性压缩新算法[J].中国电机工程学报,2013,33(19):191-199. Qu Zhijian,Guo Liang,Liu Mingguang,et al.New variable section flexible compression algorithm for measurement information in intelligent distribution network[J].Proceedings of the CSEE,2013,33(19):191-199. [21] 曲广龙,杨洪耕,张逸.采用Map-Reduce模型的海量电能质量数据交换格式文件快速解析方案[J].电网技术,2014,38(6):1705-1711. Qu Guanglong,Yang Honggeng,Zhang Yi.A fast parallel parsing scheme for massive PQDIF files with map-reduce model[J].Power System Technology,2014,38(6):1705-1711. [22] 周国亮,朱永利,王桂兰,等.实时大数据处理技术在状态监测领域中的应用[J].电工技术学报,2014,29(增刊1):432-437. Zhou Guoliang,Zhu Yongli,Wang Guilan,et al.Real-time big data processing technology application in the field of state monitoring[J].Transactions of China Electrotechnical Society,2014,29(S1):432-437. [23] 张少敏,赵硕,王保义.基于云计算和量子粒子群算法的电力负荷曲线聚类算法研究[J].电力系统保护与控制,2014,42(21):93-98. Zhang Shaomin,Zhao Shuo,Wang Baoyi.Research of power load curve clustering algorithm based on cloud computing and quantum particle swarm optimization[J].Power System Protection & Control,2014,42(21):93-98. [24] 刘巍,黄曌,李鹏,等.面向智能配电网的大数据统一支撑平台体系与构架[J].电工技术学报,2014,29(增刊1):486-491. Liu Wei,Huang Zhao,Li Peng,et al.Summary about system and framework of unified supporting platform of big data for smart distribution grid[J].Transactions of China Electro technical Society,2014,29(S1):486-491. [25] 金亮,邱运涛,杨庆新,等.基于云计算的电磁问题并行计算方法[J].电工技术学报,2016,31(22):5-11. Jin Liang,Qiu Yuntao,Yang Qingxin.A parallel computing method to electromagnetic problems based on cloud computing[J].Transactions of China Electro-technical Society,2016,31(22):5-11. [26] Nobuyuki O.A threshold selection method from gray-level histograms[J].IEEE Transactions on Systems,Man and Cybernetics,1979,9(1):62-66. [27] Chang Wen-Yeau.Partial discharge pattern recognition of cast resin current transformers using radial basis function neural network[J].Journal of Electrical Engineering & Technology,2014,9(1):293-300. [28] Cover T,Hart P.Nearest neighbor pattern classification[J].IEEE Transcations on Information Theory,1967,30(1):21-27.
2017, Vol. 32 
