基于多阶段划分的MapReduce模型

doi:10.11772/j.issn.1001-9081.2015.12.3374

计算机应用 ›› 2015, Vol. 35 ›› Issue (12): 3374-3377.DOI: 10.11772/j.issn.1001-9081.2015.12.3374

基于多阶段划分的MapReduce模型

李振举, 李学军, 杨晟, 刘涛

解放军装备学院信息装备系, 北京 101416

收稿日期:2015-06-09 修回日期:2015-08-27 出版日期:2015-12-10 发布日期:2015-12-10
通讯作者: 李振举(1987-),男,河南安阳人,助理工程师,博士研究生,CCF会员,主要研究方向:云计算、遥感数据管理
作者简介:李学军(1967-),男,湖北监利人,教授,博士,CCF会员,主要研究方向:通信与信息系统、计算机图形学;杨晟(1985-),男,陕西汉中人,讲师,博士,主要研究方向:遥感影像匹配;刘涛(1979-),男,陕西渭南人,讲师,博士,主要研究方向:计算机图形学、遥感图像并行处理。
基金资助:
总装备部预研项目(513150701)。

MapReduce performance model based on multi-phase dividing

LI Zhenju, LI Xuejun, YANG Sheng, LIU Tao

Department of Information Equipment, Equipment Academy of PLA, Beijing 101416, China

Received:2015-06-09 Revised:2015-08-27 Online:2015-12-10 Published:2015-12-10

摘要/Abstract

摘要： 针对已有的MapReduce模型阶段划分粒度不合理导致模型精度和复杂度存在的问题,提出了阶段划分粒度为5的多阶段MapReduce模型(MR-Model)。首先综述了MapReduce模型的研究现状;然后将MapReduce划分为Read、Map、Shuffle、Reduce、Write共5个阶段,并对每个阶段的具体运行时间进行研究;最后通过实验对模型的预测性能进行验证。实验结果表明,提出的MR-Model可用来描述MapReduce实际任务的执行过程,与另外两种不同划分粒度的模型P-Model和H-Model相比,MR-Model模型的运行时间预测精度可以提高10%~30%,在Reduce阶段的运行时间预测精度可以提高2~3倍,综合性能较好。

关键词: 云计算, MapReduce, 性能模型, 多阶段划分, 划分粒度

Abstract: In order to resolve the low precision and complexity problem of the existing MapReduce model caused by the reasonable phase partitioning granularity, a multi-phase MapReduce Model (MR-Model) with 5 partition granularities was proposed. Firstly, the research status of MapReduce model was reviewed. Secondly, the MapReduce job was divided into 5 phases of Read, Map, Shuffle, Reduce, Write and the specific processing time of each phase was studied. Finally, the MR-model prediction performance was tested by experiments. The experimental results show that MR-Model is suitable for the MapReduce actual job execution process. Compared with the two existing models of P-Model and H-Model, the time accuracy precision of MR-Model can be improved by 10%-30%; in the Reduce phase, its time accuracy precision can be improved by 2-3 times, the comprehensive property of the MR-Model is better.

Key words: cloud computing, MapReduce, performance model, multi-phase division, partition granularity

中图分类号:

TP391

李振举, 李学军, 杨晟, 刘涛. 基于多阶段划分的MapReduce模型[J]. 计算机应用, 2015, 35(12): 3374-3377.

LI Zhenju, LI Xuejun, YANG Sheng, LIU Tao. MapReduce performance model based on multi-phase dividing[J]. Journal of Computer Applications, 2015, 35(12): 3374-3377.

参考文献

[1] DEAN J, GHEMAWAT S. MapReduce:simplified data processing on large clusters[J]. Communications of the ACM, 2008, 51(1):107-113.
[2] QIN X, WANG H, DU X, et al. Big data analysis-competition and symbiosis of RDBMS and MapReduce[J]. Journal of Software, 2012, 23(1):32-45. (覃雄派,王会举,杜小勇,等.大数据分析--RDBMS与MapReduce的竞争与共生[J].软件学报,2012,23(1):32-45.)
[3] JAHANI E, CAFARELLA M J, RÉ C. Automatic optimization for MapReduce programs[J]. Proceedings of the VLDB Endowment, 2011, 4(6):385-396.
[4] SONG G, MENG Z, HUET F, et al. A Hadoop MapReduce performance prediction method[C]//HPCC_EUC 2013:Proceedings of the 2013 IEEE 10th International Conference on High Performance Computing and Communications & 2013 IEEE International Conference on Embedded and Ubiquitous Computing. Piscataway:IEEE, 2013:820-825.
[5] YANG X, SUN J. An analytical performance model of MapReduce[C]//Proceedings of the 2011 IEEE International Conference on Cloud Computing and Intelligence Systems. Piscataway:IEEE, 2011:306-310.
[6] BERLINSKA J, DROZDOWSKI M. Scheduling divisible MapReduce computations[J]. Journal of Parallel and Distributed Computing, 2011, 71(3):450-459.
[7] ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environments[C]//OSDI'08:Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley:USENIX Association, 2008:29-42.
[8] VIANNA E, COMARELA G, PONTES T, et al. Analytical performance models for MapReduce workloads[J]. International Journal of Parallel Programming, 2013, 41(4):495-525.
[9] VERMA A, CHERKASOVA L, CAMPBELL R H. ARIA:automatic resource inference and allocation for MapReduce environments[C]//Proceedings of the 8th ACM International Conference on Autonomic Computing. New York:ACM, 2011:235-244.
[10] LI B, MAZUR E, DIAO Y, et al. A platform for scalable one-pass analytics using MapReduce[C]//Proceedings of the 2011 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2011:985-996.
[11] HAMMOUD M, SAKR M F. MC2:Map concurrency characterization for MapReduce on the cloud[C]//Proceedings of 2013 IEEE 6th International Conference on Cloud Computing. Piscataway:IEEE, 2013:17-26.
[12] TIAN F, CHEN K. Towards optimal resource provisioning for running MapReduce programs in public clouds[C]//Proceedings of the 2011 IEEE International Conference on Cloud Computing. Piscataway:IEEE, 2011:155-162.
[13] LIN X, MENG Z, XU C, et al. A practical performance model for Hadoop MapReduce[C]//Proceedings of the 2012 IEEE International Conference on Cluster Computing Workshops. Piscataway:IEEE, 2012:231-239.
[14] HERODOTOU H. Hadoop performance models, CS-2011-05[R]. Durham:Duke University, 2011:18-19.
[15] BLANAS S, PATEL J M, ERCEGOVAC V, et al. A comparison of join algorithms for log processing in MapReduce[C]//Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2010:975-986.

基于多阶段划分的MapReduce模型

MapReduce performance model based on multi-phase dividing

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	陈家豪, 殷新春. 基于云雾计算的可追踪可撤销密文策略属性基加密方案[J]. 计算机应用, 2021, 41(6): 1611-1620.
[2]	葛丽娜, 胡雨谷, 张桂芬, 陈园园. 云计算环境基于客体属性匹配的逆向混合访问控制方案[J]. 计算机应用, 2021, 41(6): 1604-1610.
[3]	杨翎, 姜春茂. 基于三支决策的虚拟机节能迁移策略[J]. 计算机应用, 2021, 41(4): 990-998.
[4]	孙晓玲, 杨光, 沈焱萍, 杨秋格, 陈涛. 基于可拆分倒排索引的可搜索加密方案[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3288-3294.
[5]	吕佳玉, 竺智荣, 姚志强. 云计算环境下的双通道数据动态加密策略[J]. 计算机应用, 2020, 40(8): 2268-2273.
[6]	陈程军, 毛莺池, 王绎超. 基于激活-熵的分层迭代剪枝策略的CNN模型压缩[J]. 计算机应用, 2020, 40(5): 1260-1265.
[7]	郭曙杰, 李志华, 蔺凯青. 云环境下基于模糊隶属度的虚拟机放置算法[J]. 计算机应用, 2020, 40(5): 1374-1381.
[8]	许英鑫, 孙磊, 赵建成, 郭松辉. 基于蚁群优化算法的虚拟现场可编程门阵列部署策略[J]. 计算机应用, 2020, 40(3): 747-752.
[9]	王庆永, 毛莺池, 王绎超, 王龙宝. 基于多微云协作的计算任务卸载[J]. 计算机应用, 2020, 40(2): 328-334.
[10]	刘福鑫, 李劲巍, 王熠弘, 李琳. 基于Kubernetes的云原生海量数据存储系统设计与实现[J]. 计算机应用, 2020, 40(2): 547-552.
[11]	杨哂哂, 吴慧珍, 庄黎丽, 吕宏武. 基于Markov过程的IaaS系统可用性建模与分析方法[J]. 计算机应用, 2020, 40(10): 3013-3018.
[12]	林立, 熊金波, 肖如良, 林铭炜, 陈秀华. Gaming@Edge:基于边缘节点的低延迟云游戏系统[J]. 计算机应用, 2019, 39(7): 2001-2007.
[13]	徐雅斌, 彭宏恩. 基于需求预测的PaaS平台资源分配方法[J]. 计算机应用, 2019, 39(6): 1583-1588.
[14]	李启锐, 彭志平, 崔得龙, 何杰光. 容器云环境虚拟资源配置策略的优化[J]. 计算机应用, 2019, 39(3): 784-789.
[15]	李磊, 薛洋, 吕念玲, 冯敏. 基于李雅普诺夫优化的容器云队列在线任务和资源调度设计[J]. 计算机应用, 2019, 39(2): 494-500.