模型驱动的大数据流水线框架PiFlow

doi:10.11772/j.issn.1001-9081.2019101793

计算机应用 ›› 2020, Vol. 40 ›› Issue (6): 1638-1647.DOI: 10.11772/j.issn.1001-9081.2019101793

模型驱动的大数据流水线框架PiFlow

朱小杰¹, 赵子豪^1,2, 杜一^1,2

1.中国科学院计算机网络信息中心，北京 100190
2.中国科学院大学，北京 100049

收稿日期:2019-10-22 修回日期:2020-01-13 出版日期:2020-06-10 发布日期:2020-06-18
通讯作者: 杜一(1988—)
作者简介:朱小杰(1985—)，女，天津人，工程师，硕士，主要研究方向：大数据处理、大数据流水线。赵子豪(1994—)，男，辽宁阜新人，硕士研究生，CCF会员，主要研究方向：大数据处理、图数据系统、人工智能数据库。杜一(1988—)，男，山东聊城人，副研究员，博士，CCF会员，主要研究方向：可视分析、数据挖掘。
基金资助:
国家重点研发计划云计算与大数据重点专项(2018YFB1004001);国家自然科学基金重点项目(61836013);中国烟草总公司科技重大专项（110201801019(SJ-01)）。

PiFlow: model driven big data pipeline framework

ZHU Xiaojie¹, ZHAO Zihao^1,2, DU Yi^1,2

1. Computer Network Information Center, Chinese Academy of Sciences, Beijing 100190, China
2. University of Chinese Academy of Sciences, Beijing 100049, China

Received:2019-10-22 Revised:2020-01-13 Online:2020-06-10 Published:2020-06-18
Contact: DU Yi,born in 1988,Ph. D.,associate research fellow. His research interests include visual analysis,data mining
About author:ZHU Xiaojie， born in 1985， M. S.， engineer. Her research interests include big data processing，big data pipeline.ZHAO Zihao，born in 1994，M. S. candidate. His research interests include big data processing，graph data system，artificial intelligence database.DU Yi，born in 1988，Ph. D.，associate research fellow. His research interests include visual analysis，data mining.
Supported by:
Cloud Computing and Big Data Key Program of the National Key Research and Development Plan of China(2018YFB1004001), the Key Project of National Natural Science Foundation of China (61836013), the Science and Technology Major Project of China National Tobacco Corporation (110201801019(SJ-01)).

摘要/Abstract

摘要： 复杂流程的大数据处理多依托于流水线系统，但大数据处理的流水线系统在易用性、功能复用性、扩展性以及处理性能等方面存在不足。针对上述问题，为提高大数据处理环境的构建与开发效率,优化处理流程，提出了一种模型驱动的大数据流水线框架PiFlow。首先，将大数据处理过程抽象为有向无环图；然后，开发一系列组件用于构建数据处理流水线，并设计了流水线任务执行机制。同时，为规范和简化流水线框架的描述，设计了基于模型驱动的大数据流水线描述语言——PiFlowDL，该语言以模块化、层次化的方式对大数据处理任务进行描述。PiFlow以所见即所得(WYSIWYG)的方式配置流水线，集成了状态监控、模板配置、组件集成等功能，与Apache NiFi相比有2~7倍的性能提升。

关键词: 大数据, 流水线, 流水线调度, 模型驱动的开发方法, 数据处理

Abstract: Big data processing with complex process mostly relies on pipeline systems. However, the pipeline systems of big data processing have some shortcomings in usability, function reusability, expansibility and processing performance. In order to solve the problems and improve the construction and development efficiency of big data processing environment and optimize the processing flow, a model driven big data pipeline framework called PiFlow was proposed. Firstly, the big data processing process was abstracted as a directed acyclic graph. Then, a series of components were developed to construct the data processing pipeline, and the pipeline task execution mechanism was designed. At the same time, in order to standardize and simplify the pipeline framework description, a model driven big data pipeline description language called PiFlowDL was designed, which described the big data processing tasks in a modular and hierarchical way. PiFlow configures the pipeline in a What You See Is What You Get (WYSIWYG) way, and integrates the functions such as status monitoring, template configuration, and component integration. Compared with Apache NiFi, it has the performance improvement of 2-7 times.

Key words: big data, pipeline, pipeline scheduling, model driven development method, data processing

中图分类号:

TP311.56

朱小杰, 赵子豪, 杜一. 模型驱动的大数据流水线框架PiFlow[J]. 计算机应用, 2020, 40(6): 1638-1647.

ZHU Xiaojie, ZHAO Zihao, DU Yi. PiFlow: model driven big data pipeline framework[J]. Journal of Computer Applications, 2020, 40(6): 1638-1647.

参考文献

1 宋杰,孙宗哲,毛克明,等.MapReduce大数据处理平台与算法研究进展[J].软件学报,2017,28(3):514-543. SONGJ, SUNZ Z, MAOK M, et al. Research advance on MapReduce based big data processing platforms and algorithms [J]. Journal of Software, 2017, 28(3):514-543.
2 宫学庆,金澈清,王晓玲,等.数据密集型科学与工程:需求和挑战[J].计算机学报,2012,35(8):1563-1578. GONGX Q, JINC Q, WANGX L, et al. Data-intensive science and engineering: requirements and challenges [J]. Chinese Journal of Computers, 2012, 35(8): 1563-1578.
3 MANYIKAJ, CHUIM, BROWNB, et al. Big Data: The Next Frontier for Innovation, Competition, and Productivity [M]. Hawthorn: McKinsey Global Institute, 2011:1-13.
4 JINX, WAH B W, CHENGX, et al. Significance and challenges of big data research [J]. Big Data Research, 2015, 2(2): 59-64.
5 CHENH, CHIANGR H L, STOREYV C. Business intelligence and analytics: from big data to big impact [J]. MIS Quarterly, 2012, 36(4):1165-1188.
6 沈志宏,姚畅,侯艳飞,等.关联大数据管理技术:挑战、对策与实践[J].数据分析与知识发现,2018,2(1):9-20. SHENZ H, YAOC, HOUY F, et al. Big linked data management: challenges, solutions and practices [J]. Data Analysis and Knowledge Discovery, 2018, 2(1): 9-20.
7 KLEPPEA G, WARMERJ, BASTW. MDA Explained: The Model Driven Architecture: Practice and Promise [M]. Boston: Addison-Wesley Professional, 2003: 1-192.
8 BRAMBILLAM, CABOTJ, WIMMERM, et al. Model-Driven Software Engineering in Practice [M]. 2nd ed. San Rafael: Morgan & Claypool Publishers, 2017: 1-207.
9 WHITTLEJ, HUTCHINSONJ, ROUNCEFIELDM. The state of practice in model-driven engineering [J]. IEEE Software, 2014, 31(3): 79-85.
10 VANDERDONCKTJ, LIMBOURGQ, MICHOTTEB, et al. USIXML: a user interface description language supporting multiple levels of independence [EB/OL]. [2019-01-10]. https://www.w3.org/2004/02/mmi-workshop/vanderdonckt-louvain.pdf.
11 杜一,邓昌智,田丰,等.一种可扩展的用户界面描述语言[J].软件学报,2013,24(5):1127-1142. DUY, DENGC Z, TIANF, et al. Extensible user interface description language [J]. Journal of Software, 2013, 24(5): 1127-1142.
12 PUERTAA, EISENSTEINJ. Towards a general computational framework for model-based interface development systems [C]// Proceedings of the 4th International Conference on Intelligent User Interfaces. New York: ACM, 1999: 171-178.
13 杜一,郭旦怀,陈昕,等.一种模型驱动的可视化生成系统.软件学报,2016,27(5):1199-1211.(DU Y, GUO D H, CHEN X,et al. Model-driven visualization generation system [J]. Journal of Software, 2016,27(5): 1199-1211.)
14 DEANJ, GHEMAWZATS. MapReduce: simplified data processing on large clusters [J]. Communications of the ACM, 2008, 51(1): 107-113.
15 ZAHARIAM, CHOWDHURYM, FRANKLINM J, et al. Spark: cluster computing with working sets [C]// Proceedings of the 2nd USENIX Conference on Hot Topics in Cloud Computing. Berkeley: USENIX Association, 2010:1-7.
16 SHVACHKOK, KUANGH, RADIAS, et al. The Hadoop distributed file system [C]// Proceedings of the IEEE 26th Symposium on Mass Storage Systems and Technologies. Piscataway: IEEE, 2010: 1-10.
17 WANGD, ZHOUF, LIJ. Cloud-based parallel power flow calculation using resilient distributed datasets and directed acyclic graph [J]. Journal of Modern Power Systems and Clean Energy, 2019, 7(1): 65-77.
18 MOFFAG, CATONEG, KUIPERSJ, et al. Using directed acyclic graphs in epidemiological research in psychosis: an analysis of the role of bullying in psychosis [J]. Schizophrenia Bulletin, 2017, 43(6): 1273-1279.
19 HEP, ZHUJ, XUP, et al. A directed acyclic graph approach to online log parsing [EB/OL]. [2019-01-10]. https://arxiv.org/pdf/1806.04356.pdf.
20 TOSHNIWALA, TANEJAS, SHUKLAA, et al. Storm@twitter [C]// Proceedings of the 2014 ACM SIGMOD International Conference on Management of Data. New York: ACM, 2014: 147-156.
21 CARBONEP, KATSIFODIMOSA, EWENS, et al. Apache Flink: stream and batch processing in a single engine [J]. Bulletin of the IEEE Computer Society Technical Committee on Data Engineering, 2015, 36(4): 28-38.
22 NEUMEYERL, ROBBINSB, NAIRA, et al. S4: distributed stream computing platform [C]// Proceedings of the 2010 IEEE International Conference on Data Mining Workshops. Piscataway: IEEE, 2010: 170-177.
23 VENKATARAMANS, PANDAA, OUSTERHOUTK, et al. Drizzle: Fast and adaptable stream processing at scale [C]// Proceedings of the 26th Symposium on Operating Systems Principles. New York: ACM, 2017: 374-389.
24 ARMBRUSTM, DAS T, TORRESJ, et al. Structured streaming: a declarative API for real-time applications in Apache Spark [C]// Proceedings of the 2018 International Conference on Management of Data. New York: ACM, 2018: 601-613.
25 INTERLANDIM, EKMEKJIA, SHAHK, et al. Adding data provenance support to Apache Spark [J]. The VLDB Journal, 2018, 27(5): 595-615.
26 HABIB-UR-REHMANM, AHMEDE, YAQOOBI, et al. Big data analytics in industrial IoT using a concentric computing model[J]. IEEE Communications Magazine, 2018, 56(2): 37-43.
27 AKIDAUT, BRADSHAWR, CHAMBERSC, et al. The dataflow model: a practical approach to balancing correctness, latency, and cost in massive-scale, unbounded, out-of-order data processing [J]. Proceedings of the VLDB Endowment, 2015, 8(12): 1792-1803.
28 KRISHNANS P T, GONZALEZJ L U. Google cloud dataflow [M]// KRISHNAN S P T, GONZALEZ J L U. Building Your Next Big Thing with Google Cloud Platform. Berkeley: Apress, 2015: 255-275.
29 KREPSJ, NARKHEDEN, RAOJ. Kafka: a distributed messaging system for log processing [EB/OL]. [2019-01-10]. http://notes.stephenholiday.com/Kafka.pdf.
30 ISAHH, ZULKERNINEF. A scalable and robust framework for data stream ingestion [C]// Proceedings of the 2018 IEEE International Conference on Big Data. Piscataway: IEEE, 2018: 2900-2905.
31 ARDAGNAC A, BELLANDIV, CERAVOLOP, et al. A model-driven methodology for big data analytics-as-a-service [C]// Proceedings of the 2017 IEEE International Congress on Big Data. Piscataway: IEEE, 2017: 105-112.

[1]	黄程程, 董霄霄, 李钊. 基于二维Winograd算法的深流水线5×5卷积方法[J]. 计算机应用, 2021, 41(8): 2258-2264.
[2]	周翔, 翟俊海, 黄雅婕, 申瑞彩, 侯璎真. 基于随机森林和投票机制的大数据样例选择算法[J]. 计算机应用, 2021, 41(1): 74-80.
[3]	张晨曦, 唐曙, 唐珂. 迁移学习下的火箭发动机参数异常检测策略[J]. 计算机应用, 2020, 40(9): 2774-2780.
[4]	曹策俊, 刘桔. 灾害运作管理中应急组织决策建模方法综述[J]. 计算机应用, 2020, 40(7): 2142-2149.
[5]	吴文莉, 刘国华, 张君宝. 大数据上函数查询解答的复杂度分析[J]. 计算机应用, 2020, 40(2): 416-419.
[6]	李孜颖, 石振国. 面向大数据任务的调度方法[J]. 计算机应用, 2020, 40(10): 2923-2928.
[7]	章永来, 周耀鉴. 聚类算法综述[J]. 计算机应用, 2019, 39(7): 1869-1882.
[8]	孙小涓, 石涛, 胡玉新, 佟继周, 李冰, 宋峣. 基于流式计算的空间科学卫星数据实时处理[J]. 计算机应用, 2019, 39(6): 1563-1568.
[9]	马建刚, 马应龙. 语义驱动的司法文档学习分类方法[J]. 计算机应用, 2019, 39(6): 1696-1700.
[10]	纪丽娜, 陈凯, 于彦伟, 宋鹏, 王淑莹, 王成锐. 基于城市交通大数据的车辆类别挖掘及应用分析[J]. 计算机应用, 2019, 39(5): 1343-1350.
[11]	张译天, 于炯, 鲁亮, 李梓杨. 大数据流式计算框架Heron环境下的流分类任务调度策略[J]. 计算机应用, 2019, 39(4): 1106-1116.
[12]	常征, 吕勇. 基于正则表达式的海量数据清洗系统[J]. 计算机应用, 2019, 39(10): 2942-2947.
[13]	王雪菲, 丁维龙. 面向高速公路大数据的短时流量预测方法[J]. 计算机应用, 2019, 39(1): 87-92.
[14]	徐垚, 李卓然, 孟金龙, 赵利坡, 温建新, 王桂玲. 基于大规模船舶轨迹数据的航道边界提取方法[J]. 计算机应用, 2019, 39(1): 105-112.
[15]	唐小川, 罗亮. 基于析因设计的大数据相关关系挖掘算法[J]. 计算机应用, 2018, 38(9): 2507-2510.

模型驱动的大数据流水线框架PiFlow

PiFlow: model driven big data pipeline framework

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics