大数据环境下的分布式数据流处理关键技术探析

doi:10.11772/j.issn.1001-9081.2017.03.620

计算机应用 ›› 2017, Vol. 37 ›› Issue (3): 620-627.DOI: 10.11772/j.issn.1001-9081.2017.03.620

• 第四届大数据学术会议(CCF BIGDATA2016) • 上一篇下一篇

大数据环境下的分布式数据流处理关键技术探析

陈付梅, 韩德志, 毕坤, 戴永涛

上海海事大学信息工程学院, 上海 201306

收稿日期:2016-09-20 修回日期:2016-10-18 发布日期:2017-03-22 出版日期:2017-03-10
通讯作者: 韩德志
作者简介:陈付梅(1989-),女,山东临沂人,硕士研究生,主要研究方向:云计算、大数据实时分析;韩德志(1966-),男,河南信阳人,教授,博士,CCF高级会员,主要研究方向:云计算、云存储及其安全技术、大数据应用技术;毕坤(1981-),男,山东青岛人,讲师,博士,主要研究方向:云计算、云存储、大数据应用技术;戴永涛(1991-),男,湖南邵阳人,硕士研究生,主要研究方向:云计算、分布式计算、数据挖掘、网络安全技术。
基金资助:
国家自然科学基金资助项目（61373028，61672338）。

Key technologies of distributed data stream processing based on big data

CHEN Fumei, HAN Dezhi, BI Kun, DAI Yongtao

College of Information Engineering, Shanghai Maritime University, Shanghai 201306, China

Received:2016-09-20 Revised:2016-10-18 Online:2017-03-22 Published:2017-03-10
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61672338, 61373028).

摘要/Abstract

摘要： 大数据环境下的数据流处理实时性要求高，数据计算要求持续性和高可靠性。分布式数据流处理系统（DDSPS）能解决大数据环境下的数据流处理问题，它除具备分布式系统的可扩展性和容错性优势外，还具有高的实时处理能力。详细介绍了组成基于大数据的分布式数据流处理系统的四个子系统及其关键技术，讨论和比较了各个子系统的不同技术方案；同时介绍一种分布式拒绝服务（DDoS）攻击检测数据流处理系统结构案例，其研究内容能为大数据环境下的数据流处理理论研究和应用技术开发提供技术参考。

关键词: 大数据, 流处理, 消息队列, 数据处理, 数据存储

Abstract: In the big data environment, the real-time processing requirement of data stream is high, and data calculations require persistence and high reliability. Distributed Data Stream Processing System (DDSPS) can solve the problem of data stream processing in big data environment. Besides, it has the advantages of scalability and fault-tolerance of distributed system, and also has high real-time processing capability. Four subsystems and their key technologies of the DDSPS based on big data were introduced in detail. The different technical schemes of each subsystem were discussed and compared. At the same time, an example of data stream processing system structure to detect Distributed Denial of Service (DDoS) attacks was introduced, which can provide the technical reference for data stream processing theory research and application technology development under big data environment.

Key words: big data, stream processing, message queue, data processing, data storage

中图分类号:

陈付梅, 韩德志, 毕坤, 戴永涛. 大数据环境下的分布式数据流处理关键技术探析[J]. 计算机应用, 2017, 37(3): 620-627.

CHEN Fumei, HAN Dezhi, BI Kun, DAI Yongtao. Key technologies of distributed data stream processing based on big data[J]. Journal of Computer Applications, 2017, 37(3): 620-627.

参考文献

[1] 国家图书馆研究院.CNNIC发布第37次《中国互联网络发展状况统计报告》[J].国家图书馆学刊,2016(2):20.(The Research Institute of the National Library.The 37th China Internet network development state statistic report issued by CNNIC[J]. Journal of the National Library of China, 2016(2):20.)
[2] IDC. The digital universe of opportunities:rich data and the increasing value of the Internet of things[EB/OL].[2014-04-15]. http://www.emc.com/leadership/digital-universe/2014iview/executive-summary.htm.
[3] 戴永涛.分布式流处理系统研究与应用[D].上海:上海海事大学,2016:1-40.(DAI Y T. Research and application of distributed streaming system[D]. Shanghai:Shanghai Maritime University, 2016:1-40.)
[4] 赵勇.架构大数据——大数据技术及算法解析[M].北京:电子工业出版社,2015:394-410.(ZHAO Y. Big Data Structure-The Technology and Algorithm Analysis of Big Data[M]. Beijing:Publishing House of Electronics Industry, 2015:394-410.)
[5] 孟小峰,慈祥.大数据管理:概念、技术与挑战[J].计算机研究与发展,2013,50(1):146-169.(MENG X F, CI X. Big data management:concepts, technology and challenges[J]. Journal of Computer Research and Development, 2013, 50(1):146-169.)
[6] 孙大为,张广艳,郑纬民.大数据流式计算:关键技术及系统实例[J].软件学报,2014,25(4):839-862.(SUN D W, ZHANG G Y, ZHENG W M. Big data flow calculation:the key technology and system instance[J]. Journal of Software, 2014, 25(4):839-862.)
[7] MICHAEL K, MILLER K W. Big data:new opportunities and new challenges[J]. Computer, 2013, 46(6):22-24.
[8] Facebook. Scribe Wiki[EB/OL].[2015-02-03]. https://github.com/facebookarchive/scribe/wiki.
[9] Apache Software Foundation. Apache Thrift[EB/OL].[2016-05-01]. http://thrift.apache.org/.
[10] Apache Software Foundation. Apache Flume[EB/OL].[2016-04-09]. http://flume.apache.org/.
[11] Apache Software Foundation. Apache Chukwa[EB/OL].[2016-04-05]. http://chukwa.apache.org/.
[12] Elasticsearch. Elasticsearch Logstash[EB/OL].[2016-04-11]. https://www.elastic.co/products/logstash.
[13] Pivotal Software. RabbitMQ[EB/OL].[2016-04-11]. http://www.rabbitmq.com/.
[14] Erlang. Erlang introduction[EB/OL].[2016-04-11]. http://www.erlang.org/.
[15] HINTJENS P. ZeroMQ:Messaging for Many Applications[M]. Sebastopol:O'Reilly Media, 2013.
[16] SCHMIDT D C. Reactor:an object behavioral pattern for demultiplexing and dispatching handles for synchronous events[J]. Compilers Principles Techniques & Tools, 1999, 261(2):201-208.
[17] Apache Software Foundation. Kafka[EB/OL].[2016-04-11]. http://kafka.apache.org/.
[18] Apache Software Foundation. Apache Zookeeper[EB/OL].[2016-04-11]. http://zookeeper.apache.org/.
[19] PALANIAPPAN S K, NAGARAJA P B. Efficient data transfer through zero copy[EB/OL].[2016-03-13]. https://www.ibm.com/developerworks/linux/library/j-zerocopy/j-zerocopy-pdf.pdf.
[20] Apache Software Foundation. Apache Storm[EB/OL].[2016-04-07]. http://storm.apache.org/.
[21] Apache Software Foundation. Apache Samza[EB/OL].[2016-04-08]. http://samza.apache.org/.
[22] MURTHY A. Apache Hadoop YARN-background and an overview[EB/OL].[2016-04-08]. http://hortonworks.com/blog/apache-hadoop-yarn-background-and-an-overview/.
[23] Apache Software Foundation. Apache Flink[EB/OL].[2016-04-08]. http://flink.apache.org/.
[24] Apache Software Foundation. Apache Spark[EB/OL].[2016-04-08]. http://spark.apache.org/.
[25] ZAHARIA M, CHOWDHURY M, DAS T, et al. Resilient distributed datasets:a fault-tolerant abstraction for in-memory cluster computing[C]//Proceedings of the 9th Usenix Conference on Networked Systems Design and Implementation. Berkely:USENIX Association, 2012:2.
[26] ZAHARIA M, DAS T, LI H, et al. Discretized streams:an efficient and fault-tolerant model for stream processing on large clusters[C]//Proceedings of the 4th USENIX Conference on Hot Topics in Cloud Computing. Berkeley, CA:USENIX Association, 2012:10.
[27] Apache Software Foundation. Apache Hbase[EB/OL].[2016-04-06]. http://hbase.apache.org/.
[28] CHANG F, DEAN J, GHEMAWAT S, et al. Bigtable:a distributed storage system for structured data[C]//Proceedings of the 7th USENIX Symposium on Operating Systems Design and Implementation. Berkeley, CA:USENIX Association, 2006:15.
[29] 百度百科.CAP原则[EB/OL].[2016-03-18]. http://baike.baidu.com/link?url=i-7VhglR7AO5k63IHraug1jk0t6LE03jVJsKJQiUL4VU22oNGDa3u2vr_PT8m27-b4ZG_vpxtY7laL8jHNzq9q. (Baidu Encyclopedia. CAP principle[EB/OL].[2016-03-18]. http://baike.baidu.com/link?url=i-7VhglR7AO5k63IHraug1jk0t6LE03jVJsKJQiUL4VU22oNGDa3u2vr_PT8m27-b4ZG_vpxtY7laL8jHNzq9q.)
[30] LAKSHMAN A, MALIK P. Cassandra:a decentralized structured storage system[J]. ACM SIGOPS Operating Systems Review, 2010, 44(2):35-40.
[31] DECANDIA G, HASTORUN D, JAMPANI M, et al. Dynamo:amazon's highly available key-value store[J]. ACM SIGOPS Operating Systems Review, 2007, 41(6):205-220.
[32] Redis Labs. Redis[EB/OL].[2016-04-11]. http://redis.io/.
[33] Redis Labs. Redis cluster tutorial[EB/OL].[2016-04-11]. http://redis.io/topics/cluster-tutorial.

大数据环境下的分布式数据流处理关键技术探析

Key technologies of distributed data stream processing based on big data

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	李旭, 何玉林, 崔来中, 黄哲学, PHILIPPE Fournier‑Viger. 基于大数据随机样本划分的分布式观测点分类器[J]. 《计算机应用》唯一官方网站, 2024, 44(6): 1727-1733.
[2]	陈国祥, 于自强, 赵浩宇. 面向动态路网的移动对象分布式k近邻查询算法[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3403-3410.
[3]	曹萌, 余孙婕, 曾辉, 史红周. 基于区块链的医疗数据分级访问控制与共享系统[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1518-1526.
[4]	杨力, 陈建廷, 向阳. 基于HBase的工业时序大数据分布式存储性能优化策略[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 759-766.
[5]	凌宇, 单志龙. 基于兴趣增强的知识概念推荐系统[J]. 《计算机应用》唯一官方网站, 2023, 43(12): 3697-3702.
[6]	陈虎, 周鹏灵. 面向国产高性能众核处理器的编程模型[J]. 《计算机应用》唯一官方网站, 2023, 43(11): 3517-3526.
[7]	孙志于, 王琪, 高彬, 梁中军, 徐晓斌, 王尚广. 面向边缘智能计算的数据场分类算法[J]. 《计算机应用》唯一官方网站, 2022, 42(11): 3473-3478.
[8]	刘阳, 张扬扬, 周号益. 面向流式数据处理系统的高效故障恢复方法[J]. 《计算机应用》唯一官方网站, 2022, 42(11): 3337-3345.
[9]	廖佳, 陈扬, 包秋兰, 廖雪花, 朱洲森. 计算密集型大流量数据的接力计算与动态分流处理[J]. 计算机应用, 2021, 41(9): 2646-2651.
[10]	谷正川, 郭渊博, 方晨. 基于代理重加密的消息队列遥测传输协议端到端安全解决方案[J]. 计算机应用, 2021, 41(5): 1378-1385.
[11]	周翔, 翟俊海, 黄雅婕, 申瑞彩, 侯璎真. 基于随机森林和投票机制的大数据样例选择算法[J]. 计算机应用, 2021, 41(1): 74-80.
[12]	张晨曦, 唐曙, 唐珂. 迁移学习下的火箭发动机参数异常检测策略[J]. 计算机应用, 2020, 40(9): 2774-2780.
[13]	曹策俊, 刘桔. 灾害运作管理中应急组织决策建模方法综述[J]. 计算机应用, 2020, 40(7): 2142-2149.
[14]	朱小杰, 赵子豪, 杜一. 模型驱动的大数据流水线框架PiFlow[J]. 计算机应用, 2020, 40(6): 1638-1647.
[15]	吴文莉, 刘国华, 张君宝. 大数据上函数查询解答的复杂度分析[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 416-419.