云环境下影响数据分布并行应用执行效率的因素分析

doi:10.11772/j.issn.1001-9081.2017.07.1883

计算机应用 ›› 2017, Vol. 37 ›› Issue (7): 1883-1887.DOI: 10.11772/j.issn.1001-9081.2017.07.1883

云环境下影响数据分布并行应用执行效率的因素分析

马生俊, 陈旺虎, 俞茂义, 李金溶, 郏文博

西北师范大学计算机科学与工程学院, 兰州 730070

收稿日期:2017-01-16 修回日期:2017-03-11 出版日期:2017-07-10 发布日期:2017-07-18
通讯作者: 马生俊
作者简介:马生俊(1989-),男,甘肃广河人,硕士研究生,主要研究方向:大数据与云计算;陈旺虎(1973-),男,甘肃静宁人,教授,博士,CCF会员,主要研究方向:大数据与云计算;俞茂义(1991-),男,安徽铜陵人,硕士研究生,主要研究方向:大数据与云计算;李金溶(1989-),女,山东肥城人,硕士研究生,主要研究方向:大数据与云计算;郏文博(1992-),男,江苏丰县人,硕士研究生,主要研究方向:大数据与云计算。
基金资助:
国家自然科学基金资助项目（61462076）。

Analysis of factors affecting efficiency of data distributed parallel application in cloud environment

MA Shengjun, CHEN Wanghu, YU Maoyi, LI Jinrong, JIA Wenbo

College of Computer Science and Engineering, Northwest Normal University, Lanzhou Gansu 730070, China

Received:2017-01-16 Revised:2017-03-11 Online:2017-07-10 Published:2017-07-18
Supported by:
This work is supported by the National Natural Science Foundation of China (61462076).

摘要/Abstract

摘要： 云环境下，类似MapReduce的数据分布并行应用被广泛运用。针对此类应用执行效率低、成本高的问题，以Hadoop为例，首先，分析该类应用的执行方式，发现数据量、节点数和任务数是影响其效率的主要因素；其次，探讨以上因素对应用效率的影响；最后，通过实验得出在数据量一定的情况下，增加节点数不会明显提高应用的执行效率，反而极大地增加执行成本；当任务数接近节点数时，应用的执行效率较高、成本较低。该结论为云环境中类似MapReduce的数据分布并行应用的效率优化提供借鉴，并为用户租用云资源提供参考。

关键词: 云环境, 数据分布并行应用, MapReduce, 效率, 成本

Abstract: Data distributed parallel applications like MapReduce are widely used. Focusing on the issues such as low execution efficiency and high cost of such applications, a case analysis of Hadoop was given. Firstly, based on the analyses of the execution processes of such applications, it was found that the data volume, the numbers of the nodes and tasks were the main factors that affected their execution efficiency. Secondly, the impacts of the factors mentioned above on the execution efficiency of an application were explored. Finally, based on a set of experiments, two important novel rules were derived as follows. Given a specific volume of data, the execution efficiency of a data distributed parallel application could not be improved remarkably only by increasing the number of nodes, but the execution cost would raise on the contrary. However, when the number of tasks was nearly equal to that of the nodes, a higher efficiency and lower cost could be got for such an application. The conclusions are useful for users to optimize their data distributed parallel applications and to estimate the necessary computing resources to be rented in a cloud environment.

Key words: cloud environment, data distributed parallel application, MapReduce, efficiency, cost

中图分类号:

马生俊, 陈旺虎, 俞茂义, 李金溶, 郏文博. 云环境下影响数据分布并行应用执行效率的因素分析[J]. 计算机应用, 2017, 37(7): 1883-1887.

MA Shengjun, CHEN Wanghu, YU Maoyi, LI Jinrong, JIA Wenbo. Analysis of factors affecting efficiency of data distributed parallel application in cloud environment[J]. Journal of Computer Applications, 2017, 37(7): 1883-1887.

参考文献

[1] 冯登国,张敏,李昊.大数据安全与隐私保护[J].计算机学报,2014,37(1):246-258.(FENG D G, ZHANG M, LI H. Big data security and privacy protection[J]. Chinese Journal of Computers, 2014, 37(1):246-258.)
[2] VAQUERO L M, RODERO-MERINO L, CACERES J, et al. A break in the clouds:towards a cloud definition[J]. ACM SIGCOMM Computer Communication Review, 2008, 39(1):50-55.
[3] JIANG D, PIERRE G, CHI C H. EC2 performance analysis for resource provisioning of service-oriented applications[C]//NFPSLAM-SOC 2009:Proceedings of the 3rd Workshop on Non-functional Properties and Service Level Agreements Management in Service Oriented Computing. Berlin:Springer, 2010:197-207.
[4] BYUN E K, KEE Y S, KIM J S, et al. BTS:resource capacity estimate for time-targeted science workflows[J]. Journal of Parallel & Distributed Computing, 2011, 71(6):848-862.
[5] 刘丹丹,陈俊,梁锋,等.云计算异构环境下Hadoop性能分析[J].集成技术,2012,1(4):46-51.(LIU D D, CHEN J, LIANG F, et al. A performance analysis for Hadoop under heterogeneous cloud computing environments[J]. Journal of Integration Technology, 2012, 1(4):46-51.)
[6] AHMAD N M, YAACOB A H, AMIN A H M, et al. Performance analysis of MapReduce on OpenStack-based Hadoop virtual cluster[C]//ISTT 2014:Proceedings of the 2014 IEEE 2nd International Symposium on Telecommunication Technologies. Piscataway, NJ:IEEE, 2014:132-137.
[7] GOHIL P, GARG D, PANCHAL B. A performance analysis of MapReduce applications on big data in cloud based Hadoop[C]//ICICES 2014:Proceedings of the 2014 International Conference on Information Communication and Embedded Systems. Piscataway, NJ:IEEE, 2015:1-6.
[8] 丁泽柳,郭得科,申建伟,等.面向云计算的数据中心网络拓扑研究[J].国防科技大学学报,2011,33(6):1-6.(DING Z L, GUO D K, SHEN J W, et al. Researching data center networking topology for cloud computing[J]. Journal of National University of Defense Technology, 2011, 33(6):1-6.)
[9] 李立耀,赵少卡,许华荣.基于云平台的MapReduce性能优化策略[J].兰州大学学报(自然科学版),2015,51(5):752-758.(LI L Y, ZHAO S K, XU H R. MapReduce performance optimization strategy based on a cloud platform[J]. Journal of Lanzhou University (Natural Sciences), 2015, 51(5):752-758.)
[10] PREMCHAISWADI W, ROMSAIYUD W. Optimizing and tuning MapReduce jobs to improve the large-scale data analysis process[J]. International Journal of Intelligent Systems, 2013, 28(2):185-200.
[11] 李三淼,李龙澍.Hadoop中处理小文件的四种方法的性能分析[J].计算机工程与应用,2016,52(9):44-49.(LI S M, LI L S. Performance analysis of four methods for handing small files in Hadoop[J]. Computer Engineering and Applications, 2016, 52(9):44-49.)
[12] 杨浩.Hadoop平台性能优化的研究与实现[D].成都:西南交通大学,2015:25-37.(YANG H. Research and implementation of Hadoop platform performance optimization[D]. Chengdu:Southwest Jiaotong University, 2015:25-37)
[13] 王春梅,胡玉平,易叶青,等.Hadoop云计算平台的参数优化算法[J].华中师范大学学报(自然科学版),2016,50(2):183-189.(WANG C M, HU Y P, YI Y Q, et al. Cross layer parameter optimization algorithm for Hadoop cloud computing platform[J]. Journal of Central China Normal University (Natural Sciences), 2016, 50(2):183-189.)
[14] 李千慧,魏海平,窦雪英.基于Hadoop的排序性能优化研究[J].电子设计工程,2016,24(2):45-47.(LI Q H, WEI H P, DOU X Y. Optimization of sorting performance based on Hadoop[J]. Electronic Design Engineering, 2016, 24(2):45-47.)
[15] PRINCE J D. Introduction to cloud computing[J]. Journal of Electronic Resources in Medical Libraries, 2011, 8(4):449-458.
[16] GHEMAWAT S, GOBIOFF H, LEUNG S T. The Google file system[J]. ACM SIGOPS Operating Systems Review, 2003, 37(5):29-43.
[17] WHITE T, CUTTING D. Hadoop:The Definitive Guide[M]. 4th ed. Sebastopol:O'Reilly Media, 2012:1-4.
[18] DEAN J, GHEMAWAT S. MapReduce:simplified data processing on large clusters[J]. Communications of the ACM, 2008, 51(1):107-113.
[19] GORDON A W, LU P. Elastic phoenix:malleable MapReduce for shared-memory systems[C]//NPC 2011:Proceedings of the 8th IFIP International Conference on Network and Parallel Computing. Berlin:Springer, 2011:1-16.
[20] WOOD D A, HILL M D. Cost-effective parallel computing[J]. Computer, 1995, 28(2):69-72.

云环境下影响数据分布并行应用执行效率的因素分析

Analysis of factors affecting efficiency of data distributed parallel application in cloud environment

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	张豪, 朱睿, 宋栿尧, 方鹏, 夏秀峰. 距离-关键字相似度约束的双色反k近邻查询方法[J]. 计算机应用, 2021, 41(6): 1686-1693.
[2]	邹志文, 秦程. 基于k-means++的动态构建空间主题R树方法[J]. 计算机应用, 2021, 41(3): 733-737.
[3]	吕佳玉, 竺智荣, 姚志强. 云计算环境下的双通道数据动态加密策略[J]. 计算机应用, 2020, 40(8): 2268-2273.
[4]	王波, 任英琦, 黄冬艳. H-Algorand:基于多块输出的公有链共识机制[J]. 计算机应用, 2020, 40(7): 2150-2154.
[5]	葛宇, 梁静. 基于相遇概率时效性和重复扩散感知的机会网络消息转发算法[J]. 计算机应用, 2020, 40(5): 1397-1402.
[6]	陈永红, 郭莉莉, 张士兵, 杨洁. 基于微基站发射功率的异构蜂窝网络能效优化[J]. 计算机应用, 2020, 40(4): 1115-1118.
[7]	毛敏俐, 梁承姬, 胡筱渊. 基于船时效率的岸桥配置优化[J]. 计算机应用, 2020, 40(4): 1223-1230.
[8]	段宇君, 王耀力, 常青, 刘鑫. 改进型蚁群算法融合混沌优化的pSPIEL算法的无线传感器布局优化[J]. 计算机应用, 2020, 40(3): 793-798.
[9]	江泽涛, 徐娟娟. 云环境下基于签密的异构跨域身份认证方案[J]. 计算机应用, 2020, 40(3): 740-746.
[10]	赵志学, 李夏苗, 周鲜成. 考虑拥堵区域的多车型绿色车辆路径问题优化[J]. 计算机应用, 2020, 40(3): 883-890.
[11]	张青, 郑岩. 基于蛙跳退火粒子群算法的民航发动机单元体修理级别决策及成本优化[J]. 计算机应用, 2020, 40(12): 3541-3549.
[12]	张航, 刘善政, 唐聃, 蔡红亮. 分布式存储系统中的低修复成本纠删码[J]. 计算机应用, 2020, 40(10): 2942-2950.
[13]	杨正清, 周朝荣, 袁姝. 移动群智感知系统中基于离散布谷鸟搜索算法的任务分配[J]. 计算机应用, 2019, 39(9): 2778-2783.
[14]	王可可, 陈志德, 徐健. 基于联盟区块链的农产品质量安全高效追溯体系[J]. 计算机应用, 2019, 39(8): 2438-2443.
[15]	王守华, 李云柯, 孙希延, 纪元法. 基于低成本接收机的双天线测姿算法[J]. 计算机应用, 2019, 39(8): 2381-2385.