基于Spark Streaming的实时数据分析系统及其应用

doi:10.11772/j.issn.1001-9081.2017.05.1263

计算机应用 ›› 2017, Vol. 37 ›› Issue (5): 1263-1269.DOI: 10.11772/j.issn.1001-9081.2017.05.1263

• 第22届全国信息存储技术学术会议 • 上一篇下一篇

基于Spark Streaming的实时数据分析系统及其应用

韩德志¹, 陈旭光¹, 雷雨馨², 戴永涛¹, 张肖¹

1. 上海海事大学信息工程学院, 上海 201306;
2. 郑州大学信息工程学院, 郑州 450001

收稿日期:2016-07-15 修回日期:2016-11-26 出版日期:2017-05-10 发布日期:2017-05-16
通讯作者: 韩德志
作者简介:韩德志(1966-),男,河南信阳人,教授,博士,CCF会员,主要研究方向:云计算、云存储及其安全、大数据应用;陈旭光(1993-),男,河南信阳人,硕士研究生,主要研究方向:云计算、大数据实时分析;雷雨馨(1996-),女,河南郑州人,主要研究方向:数据挖掘、网络安全;戴永涛(1991-),男,湖南邵阳人,硕士研究生,主要研究方向:云计算、分布式计算、数据挖掘、网络安全;张肖(1994-),女,安徽蚌埠人,硕士研究生,主要研究方向:云计算、大数据实时分析。
基金资助:
国家自然科学基金资助项目（61373028，61672338）。

Real-time data analysis system based on Spark Streaming and its application

HAN Dezhi¹, CHEN Xuguang¹, LEI Yuxin², DAI Yongtao¹, ZHANG Xiao¹

1. College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China;
2. School of Information Engineering, Zhengzhou University, Zhengzhou Henan 450001, China

Received:2016-07-15 Revised:2016-11-26 Online:2017-05-10 Published:2017-05-16
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61672338,61373028).

摘要/Abstract

摘要： 为了实现对实时网络数据流的快速分析，设计一种分布式实时数据流分析系统（DRDAS），能有效解决并发访问数据流的收集、存储和实时分析问题，为大数据环境的网络安全检测提供了一种有效的数据分析平台；根据Spark Streaming运行的原理设计一种动态采样的K-Means并行算法，与DRDAS结合能实时有效地检测大数据环境下的各种分布式拒绝服务（DDoS）攻击。实验结果显示：DRDAS具有好的可扩展性、容错性和实时处理能力，与动态采样的K-Means并行算法结合能实时地检测各种DDoS攻击，缩短了攻击的检测时间。

关键词: Spark Streaming框架, 分布式流处理, 网络数据分析, 分布式拒绝服务攻击

Abstract: In order to realize the rapid analysis of massive real-time data, a Distributed Real-time Data Analysis System (DRDAS) was designed, which resolved the collection, storage and real-time analysis for mass concurrent data. And according to the operation principle of Spark Streaming, a dynamic sampling K-means parallel algorithm was proposed, which could quickly and efficiently detect all kinds of DDoS (Distributed Denial of Service) attacks. The experimental results show that the DRDAS has good scalability, fault tolerance and real-time processing ability, and along with new K-means parallel algorithm, the DRDAS can real-time detect various DDoS attacks, and shorten the detecting time of attacks.

Key words: Spark Streaming framework, distributed stream processing, network data analysis, Distributed Denial of Service (DDoS) attack

中图分类号:

TP316.2

韩德志, 陈旭光, 雷雨馨, 戴永涛, 张肖. 基于Spark Streaming的实时数据分析系统及其应用[J]. 计算机应用, 2017, 37(5): 1263-1269.

HAN Dezhi, CHEN Xuguang, LEI Yuxin, DAI Yongtao, ZHANG Xiao. Real-time data analysis system based on Spark Streaming and its application[J]. Journal of Computer Applications, 2017, 37(5): 1263-1269.

参考文献

[1] Incapsula. Report:2014 DDoS Trends-Botnet activity is up by 240%[EB/OL].[2014-06-20].https://www.incapsula.com/blog/ddos-threat-landscape-report-2014.html.
[2] Incapsula. Lax security opens the door for mass-scale abuse of SOHO routers[EB/OL].[2015-09-20].https://www.incapsula.com/blog/ddos-botnet-soho-router.html.
[3] Incapsula. DDoS impact survey reveals the actual cost of DDoS attacks[EB/OL].[2015-08-20].https://www.incapsula.com/blog/ddos-impact-cost-of-ddos-attack.html.
[4] ZAHARIA M, DAS T, LI H, et al. Discretized streams:an efficient and fault-tolerant model for stream processing on large clusters[C]//Proceedings of the 4th USENIX Conference on Hot Topics in Cloud Ccomputing. Berkeley, CA:USENIX Association, 2012:10.
[5] 孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.(SUN D W, ZHANG G Y, ZHENG W M. Big data stream computing:technologies and instances[J]. Journal of Software, 2014,25(4):839-862.)
[6] LEE Y, KANG W, SON H. An Internet traffic analysis method with MapReduce[C]//Proceedings of the 2010 IEEE/IFIP Network Operations & Management Symposium Workshops. Piscataway, NJ:IEEE, 2010:357-361.
[7] KHATTAK R, BANO S, HUSSAIN S, et al. DOFUR:DDoS forensics using MapReduce[C]//Proceedings of the 2011 Frontiers of Information Technology. Washington, DC:IEEE Computer Society, 2011:117-120.
[8] GOODHOPE K, KOSHY J, KREPS J. Building LinkedIn's real-time activity data pipeline[C]//Proceedings of the 2012 IEEE Computer Society Technical Committee on Data Engineering. Washington, DC:IEEE Computer Society, 2012:33-45.
[9] KREPS J, CORP L, NARKHEDE N, et al. Kafka:a distributed messaging system for log processing[C]//Proceedings of the 2011 ACM SIGMOD Workshop on Networking Meets Databases. New York:ACM, 2011:231-240.
[10] Apache Software Foundation. Apache Kafka[EB/OL].[2015-09-16]. http://kafka.apache.org/.
[11] Apache Software Foundation. Apache Spark:Lightning-fast cluster computing[EB/OL].[2015-09-16]. http://spark.apache.org/.
[12] Apache Software Foundation. Welcome to Apache Hadoop[EB/OL].[2015-09-16]. http://hadoop.apache.org/.
[13] ZAHARIA M, CHOWDHURY M, DAS T, et al. Resilient distributed datasets:a fault-tolerant abstraction for in-memory cluster computing[C]//Proceedings of the 9th USENIX Conference on Networked Systems Design and Implementation. Piscataway, NJ:IEEE, 2011:141-146.
[14] Apache Software Foundation. S4:distributed stream compute platform[EB/OL].[2015-09-16]. http://incubator.apache.org/s4/.
[15] Apache Software Foundation. Storm:distributed and fault-tolerance real-time computation[EB/OL].[2015-09-16]. http://storm.apache.org/.
[16] Apache Software Foundation. Apache Kafka[EB/OL].[2015-09-16]. http://kafka.apache.org/.
[17] 崔星灿,禹晓辉,刘洋,等. 分布式流处理技术综述[J]. 计算机研究与发展, 2015, 52(2):318-332.(CUI X C, YU X H, LIU Y, et al. Distributed stream processing:a survey[J]. Journal of Computer Research and Development, 2015, 52(2):318-332.)
[18] Apache Software Foundation. Apache Flume[EB/OL].[2016-01-20]. http://flume.apache.org/.
[19] Databricks. Spark-perf[EB/OL].[2015-09-16]. https://github.com/databricks/spark-perf.
[20] 虞立军, 王建光, 倪力. 使用VisualVM进行性能分析及调优[EB/OL].[2013-02-18]. http://www.ibm.com/developerworks/cn/java/j-lo-visualvm/.(YU L J, WANG J G, NI L. performance analysis and tuning by using VisualVM[EB/OL].[2013-02-18]. http://www.ibm.com/developerworks/cn/java/j-lo-visualvm/.)
[21] 周丽娟, 王慧, 王文伯,等. 面向海量数据的并行K-Means算法[J]. 华中科技大学学报(自然科学版), 2012,40(S1):150-152.(ZHOU L J, WANG H, WANG W B, et al. Parallel K-Means algorithm for mass data[J]. Journal of Huazhong University of Science and Technology(Natural Science Edition), 2012,40(S1):150-152.)
[22] 刘罕. 基于Spark框架的DDoS攻击检测系统研究[D].上海:上海海事大学, 2016:5.(LIU H. Research of the DDoS Attack Detection System base on the Spark Framework[M]. Shanghai:Shanghai Mritime University, 2016:5.)
[23] HETTICH S, BAY S D. KDD cup 1999 data[EB/OL].[1999-10-20]. http://kdd.ics.uci.edu/databases/kddcup99/kddcup99.html.

基于Spark Streaming的实时数据分析系统及其应用

Real-time data analysis system based on Spark Streaming and its application

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	王波, 任英琦, 黄冬艳. H-Algorand:基于多块输出的公有链共识机制[J]. 计算机应用, 2020, 40(7): 2150-2154.
[2]	刘向举, 刘鹏程, 徐辉, 朱晓娟. 基于软件定义物联网的分布式拒绝服务攻击检测方法[J]. 计算机应用, 2020, 40(3): 753-759.
[3]	付眸, 杨贺昆, 吴唐美, 何润, 冯朝胜, 康胜. 基于Spark Streaming的快速视频转码方法[J]. 计算机应用, 2018, 38(12): 3500-3508.
[4]	吴若豪, 董平, 郑涛. 基于OpenDayLight的恶意扫描防护技术[J]. 计算机应用, 2018, 38(1): 188-193.
[5]	马林进, 万良, 马绍菊, 杨婷, 易辉凡. 基于词袋模型的分布式拒绝服务攻击检测[J]. 计算机应用, 2017, 37(6): 1644-1649.
[6]	刘泽宇, 夏阳, 张义龙, 任远. 基于Web行为轨迹的应用层DDoS攻击防御模型[J]. 计算机应用, 2017, 37(1): 128-133.
[7]	沈学利, 申杰. 基于自治系统与动态概率包标记的DDoS攻击溯源优化方法[J]. 计算机应用, 2015, 35(6): 1705-1709.
[8]	程建张明清刘小虎范涛. 基于人工免疫的分布式入侵检测模型[J]. 计算机应用, 2014, 34(1): 86-89.
[9]	顾晓清王洪元倪彤光丁辉. 基于时间序列分析的应用层DDoS攻击检测[J]. 计算机应用, 2013, 33(08): 2228-2231.
[10]	闫巧宁土文. 基于确定线性网络编码的IPv6追踪[J]. 计算机应用, 2011, 31(09): 2352-2355.
[11]	高琰王台华郭帆余敏. 应用非迭代Apriori算法检测分布式拒绝服务攻击[J]. 计算机应用, 2011, 31(06): 1521-1524.
[12]	李扬王景中杨义先. 综合安全管理平台中日志格式化系统的设计与实现[J]. 计算机应用, 2010, 30(06): 1708-1710.
[13]	袁志勇周肆清. 一种基于分片包标记的改进方案[J]. 计算机应用, 2009, 29(2): 357-359.
[14]	徐劲松黄汝成王元捷. 一种改进的数据包标记追踪方案--CDPM[J]. 计算机应用, 2009, 29(12): 3185-3187.
[15]	徐劲松. 一种改进的路由包标记追踪方案[J]. 计算机应用, 2009, 29(05): 1316-1320.