基于Storm的海量数据实时聚类

doi:10.11772/j.issn.1001-9081.2014.11.3078

计算机应用 ›› 2014, Vol. 34 ›› Issue (11): 3078-3081.DOI: 10.11772/j.issn.1001-9081.2014.11.3078

• 2014年全国开放式分布与并行计算学术年会(DPCS 2014)论文 • 上一篇下一篇

基于Storm的海量数据实时聚类

王铭坤,袁少光,朱永利,王德文

华北电力大学(保定) 控制与计算机工程学院,河北保定 071003

收稿日期:2014-07-28 修回日期:2014-08-04 出版日期:2014-11-01 发布日期:2014-12-01
通讯作者: 王铭坤
作者简介:王铭坤(1990-),男,山东泰安人,硕士研究生,主要研究方向:云计算、大数据处理;袁少光(1989-),男,河南洛阳人,硕士研究生,主要研究方向:分布式计算、并行计算;朱永利(1963-),男,河北冀州人,教授,博士生导师,CCF高级会员,主要研究方向:网络化监控、智能信息处理;王德文(1973-),男,黑龙江克山人,副教授,博士,主要研究方向:智能信息处理、计算机网络。
基金资助:
基于IEC61850与云计算的智能电网状态监测集成平台关键问题的研究

Real-time clustering for massive data using Storm

WANG Mingkun,YUAN Shaoguang,ZHU Yongli,WANG Dewen

School of Control and Computer Engineering, North China Electric Power University, Baoding Hebei 071003, China

Received:2014-07-28 Revised:2014-08-04 Online:2014-11-01 Published:2014-12-01
Contact: WANG Mingkun

摘要/Abstract

摘要：

针对现有平台处理海量数据实时响应能力普遍较差的问题,引入Storm分布式实时计算平台进行大规模数据的聚类分析,设计了基于Storm框架的DBSCAN算法。该算法将整个过程分为数据接入、聚类分析、结果输出等阶段,在框架预定义的组件中分别编程实现,各组件通过数据流连通形成任务实体,提交到集群运行完成。通过对比分析和性能监测,验证了所提方案具有低延迟和高吞吐量的优势,集群运行状况良好,负载均衡。实验结果表明Storm平台处理海量数据实时性较高,能够胜任大数据背景下的数据挖掘任务。

Abstract:

In order to improve the real-time response ability of massive data processing, Storm distributed real-time platform was introduced to process data mining, and the Density-Based Spatial Clustering of Application with Noise (DBSCAN) clustering algorithm based on Storm was designed to deal with massive data. The algorithm was divided into three main steps: data collection, clustering analysis and result output. All procedures were realized under the pre-defined component of Storm and submitted to the Storm cluster for execution. Through comparative analysis and performance monitoring, the system shows the advantages of low latency and high throughput capacity. It proves that Storm suits for real-time processing of massive data.

中图分类号:

王铭坤袁少光朱永利王德文. 基于Storm的海量数据实时聚类[J]. 计算机应用, 2014, 34(11): 3078-3081.

WANG Mingkun YUAN Shaoguang ZHU Yongli WANG Dewen. Real-time clustering for massive data using Storm[J]. Journal of Computer Applications, 2014, 34(11): 3078-3081.

参考文献

[1]JIN R, YANG G, AGRAWAL G. Shared memory parallelization of data mining algorithms techniques programming interface and performance[J]. IEEE Transactions on Knowledge and Data Engineering,2005,17(1):71-89.
[2]CUI J, LI T, LAN H. Design and development of the mass data storage platform based on Hadoop [J]. Journal of Computer Research and Development, 2012, 49(S1): 12-18.(崔杰, 李陶深, 兰红星. 基于Hadoop的海量数据存储平台设计与开发 [J]. 计算机研究与发展, 2012, 49(增刊1): 12-18.)
[3]ANDERSON Q. Storm real-time processing cookbook[M]. Birmingham: Packt Publishing, 2013.
[4]EKANAYAKE J, PALLICKARA S. MapReduce for data intensive scientific analysis[C]// eScience 2008: Proceedings of the Fourth IEEE International Conference on eScience. Piscataway: IEEE Press, 2008: 277-284.
[5]ZHOU Q, CHEN L, MA J, et al.Improved DPhyp algorithm based Impala query optimization[J]. Journal of Computer Research and Development, 2013,50(S2): 114-120.(周强,陈岭,马骄阳,等.基于改进 DPhyp算法的Impala查询优化[J]. 计算机研究与发展, 2013,50(S2): 114-120.)
[6]WU G. Spark: the light of big data era[J]. Programmer, 2013(7):100-104.(吴甘沙. Spark: 大数据的“电光石火”[J]. 程序员, 2013(7): 100-104.)
[7]LEIBIUSKY J, EISBRUCH G, SIMONASSI D. Getting started with storm[M]. Sebastopol: OReilly Media, 2012.
[8]YE D, ZHANG R, QIAN X. Marking method for cell fluorescent images based on density extraction [J]. Journal of Tsinghua University: Science and Technology, 2013(1): 129-132.(叶大田,张锐,钱翔.基于密度提取的细胞荧光图像标识算法[J]. 清华大学学报:自然科学版, 2013 (1): 129-132.)
[9]HOU R, ZHU B, FENG M. Prediction model for lightning now casting based on DBSCAN[J]. Journal of Computer Applications,2012,32(3):847-851.(侯荣涛,朱斌,冯民学. 基于DBSCAN聚类算法的闪电临近预报模型[J].计算机应用,2012,32(3):847-851.)
[10]OUYANG J, LIN P. Webpage content extraction based on DBSCAN [J]. Computer Engineering,2011,37(3):64-69.(欧阳佳, 林丕源.基于DBSCAN算法的网页正文提取[J].计算机工程,2011,37(3):64-69.)
[11]LI J, HU X. Efficient mixed clustering algorithm and its application in anomaly detection [J]. Journal of Computer Applications,2010,30(7):1916-1918.(李建国,胡学钢.高效的混合聚类算法及其在异常检测中的应用[J].计算机应用,2010,30(7):1916-1918.)
[12]HAN J W, KAMBER M, PEI J. Data mining: concepts and techniques [M]. 3rd ed. San Francisco: Morgan Kaufmann Publishers, 2011.
[13]PAN Y, LI G, XU Y. Abnormal data detection method for environment wireless sensor networks based on DBSCAN [J]. Computer Applications and Software,2012,29(11):69-72.(潘渊洋,李光辉,徐勇军. 基于DBSCAN 的环境传感器网络异常数据检测方法[J].计算机应用与软件,2012,29(11):69-72.)

基于Storm的海量数据实时聚类

Real-time clustering for massive data using Storm

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 0

编辑推荐

Metrics