基于微操作的Hadoop参数自动调优方法

doi:10.11772/j.issn.1001-9081.2018122592

计算机应用 ›› 2019, Vol. 39 ›› Issue (6): 1589-1594.DOI: 10.11772/j.issn.1001-9081.2018122592

• 2018全国高性能计算学术年会(HPC China 2018)论文 • 上一篇下一篇

基于微操作的Hadoop参数自动调优方法

李耘书¹, 滕飞^1,2, 李天瑞^1,2

1. 西南交通大学信息科学与技术学院, 成都 611756;
2. 四川省云计算与智能技术高校重点实验室(西南交通大学), 成都 611756

收稿日期:2018-12-12 修回日期:2019-03-25 发布日期:2019-06-17 出版日期:2019-06-10
通讯作者: 滕飞
作者简介:李耘书(1991-),男,四川广元人,硕士研究生,主要研究方向:云计算、机器学习:滕飞(1984-),女,山东泰安人,副教授,博士,CCF会员,主要研究方向:云计算、医学信息学、工业大数据挖掘:李天瑞(1969-),男,福建莆田人,教授,博士,CCF会员,主要研究方向:智能信息处理、数据挖掘、云计算。
基金资助:
国家自然科学基金资助项目（61573292）；四川省科技计划项目（2019YJ0214）。

Microoperation-based parameter auto-optimization method of Hadoop

LI Yunshu¹, TENG Fei^1,2, LI Tianrui^1,2

1. School of Information Science and Technology, Southwest Jiaotong University, Chengdu Sichuan 611756, China;
2. Key Laboratory of Cloud Computing and Intelligent Technology of Sichuan Province(Southwest Jiaotong University), Chengdu Sichuan 611756, China

Received:2018-12-12 Revised:2019-03-25 Online:2019-06-17 Published:2019-06-10
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61573292), the Science and Technology Project of Sichuan Province (2019YJ0214).

摘要/Abstract

摘要： Hadoop作为大规模分布式数据处理框架已经在工业界得到广泛的应用，针对手动和经验调优方法中参数空间庞大和运行流程复杂的问题，提出了一种Hadoop参数自动优化的方法和分析框架。首先，对作业运行流程进行解耦，从可变参数直接影响的更细粒度的角度定义微操作，从而分析参数和单次微操作执行时间的关系；然后，利用微操作对作业运行流程进行重构，建立参数和作业运行时间关系的模型；最后，在此模型上应用各类搜索优化算法高效快速得出优化后的系统参数。在terasort和wordcount两个作业类型上进行了实验，实验结果表明，相对于默认参数情况，该方法使作业执行时间分别缩短了至少41%和30%。该方法能够有效提高Hadoop作业执行效率，缩短作业执行时间。

关键词: Hadoop, 参数调优, 微操作, 重构, 搜索算法

Abstract: As a large-scale distributed data processing framework, Hadoop has been widely used in industry during the past few years. Currently manual parameter optimization and experience-based parameter optimization are ineffective due to complex running process and large parameter space. In order to solve this problem, a method and an analytical framework for Hadoop parameter auto-optimization were proposed. Firstly, the operation process of a job was broken down into several microoperations and the microoperations were determined from the angle of finer granularity directly affected by variable parameters, so that the relationship between parameters and the execution time of a single microoperation was able to be analyzed. Then, by reconstructing the job operation process based on microoperations, a model of the relationship between parameters and the execution time of whole job was established. Finally, various searching optimization algorithms were applied on this model to efficiently and quickly obtain the optimized system parameters. Experiments were conducted with two types of jobs, terasort and wordcount. The experimental results show that, compared with the default parameters condition, the proposed method reduce the job execution time by at least 41% and 30% respectively. The proposed method can effectively improve the job execution efficiency of Hadoop and shorten the job execution time.

Key words: Hadoop, parameter optimization, microoperation, reconstitution, search algorithm

中图分类号:

TP311.13

李耘书, 滕飞, 李天瑞. 基于微操作的Hadoop参数自动调优方法[J]. 计算机应用, 2019, 39(6): 1589-1594.

LI Yunshu, TENG Fei, LI Tianrui. Microoperation-based parameter auto-optimization method of Hadoop[J]. Journal of Computer Applications, 2019, 39(6): 1589-1594.

参考文献

[1] DEAN J, GHEMAWAT S. MapReduce:simplified data processing on large clusters[C]//Proceedings of the 6th Conference on Symposium on Opearting Systems Design and Implementation. Berkeley, CA:USENIX Association, 2004:137-149.
[2] CUTTING D. Apache Hadoop[EB/OL]. (2015-02-25)[2018-08-12]. http://hadoop.apache.org.
[3] BABU S. Towards automatic optimization of MapReduce programs[C]//Proceedings of the 1st ACM Symposium on Cloud Computing. New York:ACM, 2010:137-142.
[4] TIPCON T. 7 tips for improving MapReduce performance[EB/OL]. (2009-12-17)[2018-08-12]. http://www.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/.
[5] HERODOTOU H, LIM H, LUO G, et al. Starfish:a self-tuning system for big data analytics[C]//Proceedings of the 20115th Biennial Conference on Innovative Data Systems Research. Asilomar, CA:[s.n.], 2011:261-272.
[6] WANG J H, QIU M K, GUO B, et al. Phase-reconfigurable shuffle optimization for Hadoop MapReduce[EB/OL].[2018-08-12]. https://www.onacademic.com/detail/journal_1000038191224210_fd14.html.
[7] YIGITBASI N, WILLKE T L, LIAO G, et al. Towards machine learning-based auto-tuning of MapReduce[C]//Proceedings of the IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems. Piscataway, NJ:IEEE, 2013:11-20.
[8] CHAUDHURI S, NARASAYYA V. Self-tuning database systems:a decade of progress[C]//Proceedings of the 2007 International Conference on Very Large Data Bases. Framingham, MA:VLDB Endowment, 2007:3-14.
[9] IPEK E, de SUPINSKI B R, SCHULZ M, et al. An approach to performance prediction for parallel applications[C]//Proceedings of the 2005 European Conference on Parallel Processing, LNCS 3648. Berlin:Springer, 2005:196-205.
[10] SINGER J, KOVOOR G, BROWN G, et al. Garbage collection auto-tuning for Java MapReduce on multi-cores[C]//Proceedings of the 2011 International Symposium on Memory Management. New York:ACM, 2011:109-118.
[11] CHENG D Z, RAO J, GUO Y F, et al. Improving performance of heterogeneous MapReduce clusters with adaptive task tuning[J]. IEEE Transactions on Parallel & Distributed Systems, 2017, 28(3):774-786.
[12] WASI-UR-RAHMAN M, ISLAM N S, LU X, et al. MR-Advisor:a comprehensive tuning tool for advising HPC users to accelerate MapReduce applications on supercomputers[C]//Proceedings of the 28th International Symposium on Computer Architecture and High Performance Computing. Piscataway, NJ:IEEE, 2016:198-205.
[13] 童颖.基于机器学习的Hadoop参数调优方法[D].武汉:华中科技大学,2016:1-52.(TONG Y. Hadoop parameters tuning method based on machine learning[D]. Wuhan:Huazhong University of Science and Technology, 2016:1-52.)
[14] LIAO G, DATTA K, WILLKE T L. Gunther:search-based auto-tuning of MapReduce[C]//Proceedings of the 2013 European Conference on Parallel Processing, LNCS 8097. Berlin:Springer, 2013:406-419.
[15] DEB K. An introduction to genetic algorithms[J]. Sadhana, 1999, 24(4/5):293-315.
[16] 祝春祥,陈世平,陈敏刚.基于递归随机抽样的Hadoop配置优化[J].计算机工程,2016,42(2):26-32.(ZHU C X, CHEN S P, CHEN M G. Configuration optimization of Hadoop based on recursive random sampling[J]. Computer Engineering, 2016, 42(2):26-32.)

基于微操作的Hadoop参数自动调优方法

Microoperation-based parameter auto-optimization method of Hadoop

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	力尚龙, 刘建华, 贾鹤鸣. 融合多狩猎协调策略的爬行动物搜索算法[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2818-2828.
[2]	穆凌霞, 周政君, 王斑, 张友民, 薛向宏, 宁凯凯. 多无人机编队避障和编队重构方法[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2938-2946.
[3]	李牧, 骆宇, 柯熙政. 基于调频连续波雷达的人体生命体征检测算法[J]. 《计算机应用》唯一官方网站, 2024, 44(6): 1978-1986.
[4]	韦修喜, 彭茂松, 黄华娟. 基于多策略改进蝴蝶优化算法的无线传感网络节点覆盖优化[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1009-1017.
[5]	孙林, 刘梦含. 基于自适应布谷鸟优化特征选择的K-means聚类[J]. 《计算机应用》唯一官方网站, 2024, 44(3): 831-841.
[6]	赵星宇, 王晓峰, 杨易, 庞立超, 杨澜. 求解恰当可满足性问题的随机局部搜索算法[J]. 《计算机应用》唯一官方网站, 2024, 44(3): 842-848.
[7]	邓辅秦, 谭朝恩, 黎俊炜, 钟家铭, 付兰慧, 张建民, 王宏民, 李楠楠, 姜炳春, 林天麟. 面向大型仓储环境的基于冲突搜索算法[J]. 《计算机应用》唯一官方网站, 2024, 44(12): 3854-3860.
[8]	李大海, 詹美欣, 王振东. 基于多个改进策略的增强麻雀搜索算法[J]. 《计算机应用》唯一官方网站, 2023, 43(9): 2845-2854.
[9]	曾泽芝, 杨建伟. 零点分布更均匀的广义极谐复指数变换[J]. 《计算机应用》唯一官方网站, 2023, 43(8): 2499-2504.
[10]	钟静, 林晨, 盛志伟, 张仕斌. 基于汉明距离的量子K-Means算法[J]. 《计算机应用》唯一官方网站, 2023, 43(8): 2493-2498.
[11]	程美英, 钱乾, 熊伟清. 信息迁移多任务优化共生生物搜索算法[J]. 《计算机应用》唯一官方网站, 2023, 43(7): 2237-2247.
[12]	徐清海, 丁世飞, 孙统风, 张健, 郭丽丽. 改进的基于多路径特征的胶囊网络[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1330-1335.
[13]	张玉杰, 王帆. 基于改进麻雀搜索算法的照明控制优化[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 835-841.
[14]	陈田, 鲁建勇, 刘军, 梁华国, 鲁迎春. 基于三维线性反馈移位寄存器的三维堆叠集成电路可重构测试方案[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 949-955.
[15]	周玉清, 韩晓龙. 双循环策略下岸桥与跨运车的联合调度[J]. 《计算机应用》唯一官方网站, 2023, 43(2): 645-653.