流计算与内存计算架构下的运营状态监测分析

doi:10.11772/j.issn.1001-9081.2017.10.3029

计算机应用 ›› 2017, Vol. 37 ›› Issue (10): 3029-3033.DOI: 10.11772/j.issn.1001-9081.2017.10.3029

• 应用前沿、交叉与综合 • 上一篇下一篇

流计算与内存计算架构下的运营状态监测分析

赵永彬¹, 陈硕¹, 刘明¹, 王佳楠^2,3, 贲驰⁴

1. 国网辽宁省电力有限公司信息通信调度监控中心, 沈阳 110004;
2. 中国科学院沈阳计算技术研究所, 沈阳 110168;
3. 中国科学院大学, 北京 100049;
4. 国家电网公司东北电力调控分中心, 沈阳 110180

收稿日期:2017-05-02 修回日期:2017-07-11 出版日期:2017-10-10 发布日期:2017-10-16
通讯作者: 陈硕(1983-),男,辽宁沈阳人,高级工程师,博士,主要研究方向:智能电网、Web工程、信息集成,E-mail:258098970@qq.com
作者简介:赵永彬(1975-),男,辽宁沈阳人,高级工程师,硕士,主要研究方向:智能电网、Web工程、信息集成;陈硕(1983-),男,辽宁沈阳人,高级工程师,博士,主要研究方向:智能电网、Web工程、信息集成;刘明(1979-),男,辽宁沈阳人,高级会计师,硕士,主要研究方向:电力信息;王佳楠(1993-),男,河南洛阳人,硕士研究生,主要研究方向:智能电网、电网大数据;贲驰(1965-),女,辽宁沈阳人,高级工程师,主要研究方向:电量采集与计费统计.
基金资助:
辽宁电力公司科技项目（SGLNXT00DKJS1600242）。

Monitoring and analysis of operation status under architecture of stream computing and memory computing

ZHAO Yongbin¹, CHEN Shuo¹, LIU Ming¹, WANG Jianan^2,3, BEN Chi⁴

1. Information & Telecommunication Branch, State Grid Liaoning Electric Power Company, Shenyang Liaoning 110004, China;
2. Shenyang Institute of Computing Technology, Chinese Academy of Sciences, Shenyang Liaoning 110168, China;
3. University of Chinese Academy of Sciences, Beijing 100049, China;
4. Electric Power Control Northeast Branch Center, State Grid Corporation of China, Shenyang Liaoning 110180, China

Received:2017-05-02 Revised:2017-07-11 Online:2017-10-10 Published:2017-10-16
Supported by:
This work is partially supported by the Science and Technology of Liaoning Electric Power Company (SGLNXT00DKJS1600242).

摘要/Abstract

摘要： 为满足对电网实时运营状态分析过程中对用户实时用电量数据等大规模实时数据进行实时分析处理的需求，实现对电网运营决策提供快速准确的数据分析支持，提出一种流计算与内存计算相结合的大规模数据分析处理的系统架构。将经过时间窗划分的用户实时用电量数据进行离散傅里叶变换（DFT），实现对异常用电行为评价指标的构建；将基于抽样统计分析构造出的用户用电行为特征，采用K-Means聚类算法实现对用户用电行为类别的划分。从实际业务系统中抽取实验数据，验证了提出的异常用电行为和用户用电分析评价指标的准确性。同时，在实验数据集上与传统的数据处理策略进行对比，实验结果表明流计算与内存计算相结合的系统架构在大规模数据分析处理方面更具优势。

关键词: 流计算, 内存计算, 特征构建, 异常监测, 行为划分

Abstract: In real-time operation state analysis of power grid, in order to meet the requirements of real-time analysis and processing of large-scale real-time data, such as real-time electricity consumption data, and provide fast and accurate data analysis support for power grid operation decision, the system architecture for large-scale data analysis and processing based on stream computing and memory computing was proposed. The Discrete Fourier Transform (DFT) was used to construct abnormal electricity behavior evaluation index based on the real-time electricity consumption data of the users by time window. The K-Means clustering algorithm was used to classify the users' electricity behavior based on the characteristics of user electricity behavior constructed by sampling statistical analysis. The accuracy of the proposed evaluation indicators of abnormal behavior and user electricity behavior was verified by the experimental data extracted from the actual business system. At the same time, compared with the traditional data processing strategy, the system architecture combined with stream computing and memory computing has good performance in large-scale data analysis and processing.

Key words: stream computing, memory computing, feature construction, anomaly detection, behavior partition

中图分类号:

TP39

赵永彬, 陈硕, 刘明, 王佳楠, 贲驰. 流计算与内存计算架构下的运营状态监测分析[J]. 计算机应用, 2017, 37(10): 3029-3033.

ZHAO Yongbin, CHEN Shuo, LIU Ming, WANG Jianan, BEN Chi. Monitoring and analysis of operation status under architecture of stream computing and memory computing[J]. Journal of Computer Applications, 2017, 37(10): 3029-3033.

参考文献

[1] 蔡勇.数据挖掘技术在电网运营监控平台建设中的研究与应用[D]. 上海: 上海交通大学, 2012: 5-6. (CAI Y. Research and application of data mining technology in grid operational monitoring platform [D]. Shanghai: Shanghai Jiao Tong University, 2012: 5-6.)
[2] 陈云.分布式电力大数据计算分析平台设计与实现[D]. 成都: 电子科技大学, 2016. (CHEN Y. The design and implementation of the distributed computing and analysis platform for power system [D]. Chengdu: University of Electronic Science and Technology of China, 2016.)
[3] 程学旗, 靳小龙, 王元卓, 等.大数据系统和分析技术综述[J]. 软件学报, 2014, 25(9): 1889-1908. (CHENG X Q, JIN X L, WANG Y Z, et al. Survey on big data system and analytic technology [J]. Journal of Software, 2014, 25(9): 1889-1908.)
[4] 李洋, 何宝灵, 刘海涛, 等.面向全球能源互联网的分布式电源云服务与大数据分析平台研究[J]. 电力信息与通信技术, 2016(3): 30-36. (LI Y, HE B L, LIU H T, et al. Research on distributed generation cloud service and big data analysis platform for global energy interconnection [J]. Electric Power Information and Communication Technology, 2016(3): 30-36.)
[5] 程敏.基于PostgreSQL和Spark的可扩展大数据分析平台[D]. 北京: 中国科学院大学, 2016. (CHEN M. Scalable big data analysis platform based on Postgre SQL and Spark [D]. Beijing: University of Chinese Academy of Sciences, 2016.)
[6] Apache Software Foundation. Storm documentation [EB/OL]. [2016-05-23]. http://storm.apache.org/releases/1.0.3/index.html.
[7] SAP Corporation. SAP HANA introduction [EB/OL]. [2016-06-14]. https://www.sap.com/china/product/technology-platform/hana.html.
[8] 熊元新, 陈允平.离散傅里叶变换的定义研究[J]. 武汉大学学报 (工学版), 2006, 39(1): 89-91. (XIONG Y X, CHEN Y P. Research on definition of discrete Fourier transform [J]. Engineering Journal of Wuhan University, 2006, 39(1): 89-91.)
[9] LIKAS A, VLASSIS N, J. VERBEEK J. The global k-means clustering algorithm [J]. Pattern Recognition, 2003, 36(2): 451-461.
[10] Apache Software Foundation. Kafka introduction [EB/OL]. [2016-07-08]. http://kafka.apache.org/intro.
[11] 王铭坤, 袁少光, 朱永利, 等.基于Storm的海量数据实时聚类[J]. 计算机应用, 2014, 34(11): 3078-3081. (WANG M K, YUAN S G, ZHU Y L, et al. Real-time clustering for massive data using Storm [J]. Journal of Computer Applications, 2014, 34(11): 3078-3081.)
[12] 李一辰, 李绪志, 阎镇.实时流计算在航天地面数据处理系统中的应用[J]. 微电子学与计算机, 2014, 31(9): 15-19. (LI Y C, LI X Z, YAN Z. Real-time stream computing in aerospace system's data disposing [J]. Microelectronics & Computer, 2014, 31(9): 15-19.)
[13] 孙大为, 张广艳, 郑纬民.大数据流式计算: 关键技术及系统实例[J]. 软件学报, 2014, 25(4): 839-862. (SUN D W, ZHANG G Y, ZHENG W M. Big data stream computing: technologies and instances [J]. Journal of Software, 2014, 25(4): 839-862.)
[14] 嵇智源, 潘巍.面向大数据的内存数据管理研究现状与展望[J]. 计算机工程与设计, 2014, 35(10): 3549-3506. (JI Z Y, PAN W. Present research status and prospects of in-memory data management in big data era [J]. Computer Engineering and Design, 2014, 35(10): 3549-3506.)
[15] 黄岚, 孙珂, 陈晓竹, 等.内存集群计算: 交互式数据分析[J]. 华东师范大学学报 (自然科学版), 2014(5): 216-227. (HUANG L, SUN K, CHEN X Z, et al. In-memory cluster computing: Interactive data analysis [J]. Journal of East China Normal University (Natural Science), 2014(5): 216-227.)
[16] 张延松, 王珊, 周烜.内存数据仓库集群技术研究[J]. 华东师范大学学报 (自然科学版), 2014(5): 117-132. (ZHANG Y S, WANG S, ZHOU X. Research on in-memory data warehouse cluster technologies [J]. Journal of East China Normal University (Natural Science), 2014(5): 117-132.)

流计算与内存计算架构下的运营状态监测分析

Monitoring and analysis of operation status under architecture of stream computing and memory computing

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 7

编辑推荐

Metrics

[1]	龚鸣清, 叶煌, 张鉴, 卢兴敬, 陈伟. 基于ARMv8架构的面向机器翻译的单精度浮点通用矩阵乘法优化[J]. 计算机应用, 2019, 39(6): 1557-1562.
[2]	刘靖, 肖冠烽. 基于Spark与粒子滤波算法的公交到站时间预测系统[J]. 计算机应用, 2019, 39(2): 429-435.
[3]	赵文芳, 刘旭林. Spark Streaming框架下的气象自动站数据实时处理系统[J]. 计算机应用, 2018, 38(1): 38-43.
[4]	卞琛, 于炯, 修位蓉, 英昌甜, 钱育蓉. 基于迭代填充的内存计算框架分区映射算法[J]. 计算机应用, 2017, 37(3): 647-653.
[5]	丁梦苏, 陈世敏. 轻量级大数据运算系统Helius[J]. 计算机应用, 2017, 37(2): 305-310.
[6]	王桂兰, 周国亮, 萨初日拉, 朱永利. Spark环境下的并行模糊C均值聚类算法[J]. 计算机应用, 2016, 36(2): 342-347.
[7]	赵军, 徐晓燕. 基于GraphX的分布式幂迭代聚类[J]. 计算机应用, 2016, 36(10): 2710-2714.