Real-time data analysis system based on Spark Streaming and its application

doi:10.11772/j.issn.1001-9081.2017.05.1263

Journal of Computer Applications ›› 2017, Vol. 37 ›› Issue (5): 1263-1269.DOI: 10.11772/j.issn.1001-9081.2017.05.1263

Previous Articles Next Articles

Real-time data analysis system based on Spark Streaming and its application

HAN Dezhi¹, CHEN Xuguang¹, LEI Yuxin², DAI Yongtao¹, ZHANG Xiao¹

1. College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China;
2. School of Information Engineering, Zhengzhou University, Zhengzhou Henan 450001, China

Received:2016-07-15 Revised:2016-11-26 Online:2017-05-10 Published:2017-05-16
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61672338,61373028).

基于Spark Streaming的实时数据分析系统及其应用

韩德志¹, 陈旭光¹, 雷雨馨², 戴永涛¹, 张肖¹

1. 上海海事大学信息工程学院, 上海 201306;
2. 郑州大学信息工程学院, 郑州 450001

通讯作者: 韩德志
作者简介:韩德志(1966-),男,河南信阳人,教授,博士,CCF会员,主要研究方向:云计算、云存储及其安全、大数据应用;陈旭光(1993-),男,河南信阳人,硕士研究生,主要研究方向:云计算、大数据实时分析;雷雨馨(1996-),女,河南郑州人,主要研究方向:数据挖掘、网络安全;戴永涛(1991-),男,湖南邵阳人,硕士研究生,主要研究方向:云计算、分布式计算、数据挖掘、网络安全;张肖(1994-),女,安徽蚌埠人,硕士研究生,主要研究方向:云计算、大数据实时分析。
基金资助:
国家自然科学基金资助项目（61373028，61672338）。

Abstract

Abstract: In order to realize the rapid analysis of massive real-time data, a Distributed Real-time Data Analysis System (DRDAS) was designed, which resolved the collection, storage and real-time analysis for mass concurrent data. And according to the operation principle of Spark Streaming, a dynamic sampling K-means parallel algorithm was proposed, which could quickly and efficiently detect all kinds of DDoS (Distributed Denial of Service) attacks. The experimental results show that the DRDAS has good scalability, fault tolerance and real-time processing ability, and along with new K-means parallel algorithm, the DRDAS can real-time detect various DDoS attacks, and shorten the detecting time of attacks.

Key words: Spark Streaming framework, distributed stream processing, network data analysis, Distributed Denial of Service (DDoS) attack

摘要： 为了实现对实时网络数据流的快速分析，设计一种分布式实时数据流分析系统（DRDAS），能有效解决并发访问数据流的收集、存储和实时分析问题，为大数据环境的网络安全检测提供了一种有效的数据分析平台；根据Spark Streaming运行的原理设计一种动态采样的K-Means并行算法，与DRDAS结合能实时有效地检测大数据环境下的各种分布式拒绝服务（DDoS）攻击。实验结果显示：DRDAS具有好的可扩展性、容错性和实时处理能力，与动态采样的K-Means并行算法结合能实时地检测各种DDoS攻击，缩短了攻击的检测时间。

关键词: Spark Streaming框架, 分布式流处理, 网络数据分析, 分布式拒绝服务攻击

CLC Number:

TP316.2

HAN Dezhi, CHEN Xuguang, LEI Yuxin, DAI Yongtao, ZHANG Xiao. Real-time data analysis system based on Spark Streaming and its application[J]. Journal of Computer Applications, 2017, 37(5): 1263-1269.

韩德志, 陈旭光, 雷雨馨, 戴永涛, 张肖. 基于Spark Streaming的实时数据分析系统及其应用[J]. 计算机应用, 2017, 37(5): 1263-1269.

References

[1] Incapsula. Report:2014 DDoS Trends-Botnet activity is up by 240%[EB/OL].[2014-06-20].https://www.incapsula.com/blog/ddos-threat-landscape-report-2014.html.
[2] Incapsula. Lax security opens the door for mass-scale abuse of SOHO routers[EB/OL].[2015-09-20].https://www.incapsula.com/blog/ddos-botnet-soho-router.html.
[3] Incapsula. DDoS impact survey reveals the actual cost of DDoS attacks[EB/OL].[2015-08-20].https://www.incapsula.com/blog/ddos-impact-cost-of-ddos-attack.html.
[4] ZAHARIA M, DAS T, LI H, et al. Discretized streams:an efficient and fault-tolerant model for stream processing on large clusters[C]//Proceedings of the 4th USENIX Conference on Hot Topics in Cloud Ccomputing. Berkeley, CA:USENIX Association, 2012:10.
[5] 孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.(SUN D W, ZHANG G Y, ZHENG W M. Big data stream computing:technologies and instances[J]. Journal of Software, 2014,25(4):839-862.)
[6] LEE Y, KANG W, SON H. An Internet traffic analysis method with MapReduce[C]//Proceedings of the 2010 IEEE/IFIP Network Operations & Management Symposium Workshops. Piscataway, NJ:IEEE, 2010:357-361.
[7] KHATTAK R, BANO S, HUSSAIN S, et al. DOFUR:DDoS forensics using MapReduce[C]//Proceedings of the 2011 Frontiers of Information Technology. Washington, DC:IEEE Computer Society, 2011:117-120.
[8] GOODHOPE K, KOSHY J, KREPS J. Building LinkedIn's real-time activity data pipeline[C]//Proceedings of the 2012 IEEE Computer Society Technical Committee on Data Engineering. Washington, DC:IEEE Computer Society, 2012:33-45.
[9] KREPS J, CORP L, NARKHEDE N, et al. Kafka:a distributed messaging system for log processing[C]//Proceedings of the 2011 ACM SIGMOD Workshop on Networking Meets Databases. New York:ACM, 2011:231-240.
[10] Apache Software Foundation. Apache Kafka[EB/OL].[2015-09-16]. http://kafka.apache.org/.
[11] Apache Software Foundation. Apache Spark:Lightning-fast cluster computing[EB/OL].[2015-09-16]. http://spark.apache.org/.
[12] Apache Software Foundation. Welcome to Apache Hadoop[EB/OL].[2015-09-16]. http://hadoop.apache.org/.
[13] ZAHARIA M, CHOWDHURY M, DAS T, et al. Resilient distributed datasets:a fault-tolerant abstraction for in-memory cluster computing[C]//Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. Piscataway, NJ:IEEE, 2011:141-146.
[14] Apache Software Foundation. S4:distributed stream compute platform[EB/OL].[2015-09-16]. http://incubator.apache.org/s4/.
[15] Apache Software Foundation. Storm:distributed and fault-tolerance real-time computation[EB/OL].[2015-09-16]. http://storm.apache.org/.
[16] Apache Software Foundation. Apache Kafka[EB/OL].[2015-09-16]. http://kafka.apache.org/.
[17] 崔星灿,禹晓辉,刘洋,等. 分布式流处理技术综述[J]. 计算机研究与发展, 2015, 52(2):318-332.(CUI X C, YU X H, LIU Y, et al. Distributed stream processing:a survey[J]. Journal of Computer Research and Development, 2015, 52(2):318-332.)
[18] Apache Software Foundation. Apache Flume[EB/OL].[2016-01-20]. http://flume.apache.org/.
[19] Databricks. Spark-perf[EB/OL].[2015-09-16]. https://github.com/databricks/spark-perf.
[20] 虞立军, 王建光, 倪力. 使用VisualVM进行性能分析及调优[EB/OL].[2013-02-18]. http://www.ibm.com/developerworks/cn/java/j-lo-visualvm/.(YU L J, WANG J G, NI L. performance analysis and tuning by using VisualVM[EB/OL].[2013-02-18]. http://www.ibm.com/developerworks/cn/java/j-lo-visualvm/.)
[21] 周丽娟, 王慧, 王文伯,等. 面向海量数据的并行K-Means算法[J]. 华中科技大学学报(自然科学版), 2012,40(S1):150-152.(ZHOU L J, WANG H, WANG W B, et al. Parallel K-Means algorithm for mass data[J]. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2012,40(S1):150-152.)
[22] 刘罕. 基于Spark框架的DDoS攻击检测系统研究[D].上海:上海海事大学, 2016:5.(LIU H. Research of the DDoS Attack Detection System base on the Spark Framework[M]. Shanghai:Shanghai Mritime University, 2016:5.)
[23] HETTICH S, BAY S D. KDD cup 1999 data[EB/OL].[1999-10-20]. http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

Real-time data analysis system based on Spark Streaming and its application

基于Spark Streaming的实时数据分析系统及其应用

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 9

Recommended Articles

Metrics

[1]	LIU Xiangju, LIU Pengcheng, XU Hui, ZHU Xiaojuan. Distributed denial of service attack detection method based on software defined Internet of things [J]. Journal of Computer Applications, 2020, 40(3): 753-759.
[2]	FU Mou, YANG Hekun, WU Tangmei, HE Run, FENG Chaosheng, KANG Sheng. Fast video transcoding method based on Spark Streaming [J]. Journal of Computer Applications, 2018, 38(12): 3500-3508.
[3]	WU Ruohao, DONG Ping, ZHENG Tao. Malicious scanning protection technology based on OpenDayLight [J]. Journal of Computer Applications, 2018, 38(1): 188-193.
[4]	MA Linjin, WAN Liang, MA Shaoju, YANG Ting, YI Huifan. Distributed denial of service attack recognition based on bag of words model [J]. Journal of Computer Applications, 2017, 37(6): 1644-1649.
[5]	SHEN Xueli, SHEN Jie. Optimization method of tracing distributed denial of service attacks based on autonomous system and dynamic probabilistic packet-marking [J]. Journal of Computer Applications, 2015, 35(6): 1705-1709.
[6]	Yan GAO WANG Taihua GUO Fan YU Min. DDoS detection with non-iterative Apriori algorithm [J]. Journal of Computer Applications, 2011, 31(06): 1521-1524.
[7]	. CDPM: An Improved Packet Marking tracking Scheme [J]. Journal of Computer Applications, 2009, 29(12): 3185-3187.
[8]	. Improved route packet marking tracking scheme [J]. Journal of Computer Applications, 2009, 29(05): 1316-1320.
[9]	. DDoS attack detection based on global network properties of network traffic anomaly [J]. Journal of Computer Applications, 2007, 27(2): 314-317.