基于微操作的Hadoop参数自动调优方法

doi:10.11772/j.issn.1001-9081.2018122592

计算机应用 ›› 2019, Vol. 39 ›› Issue (6): 1589-1594.DOI: 10.11772/j.issn.1001-9081.2018122592

• 2018全国高性能计算学术年会(HPC China 2018)论文 • 上一篇下一篇

基于微操作的Hadoop参数自动调优方法

李耘书¹, 滕飞^1,2, 李天瑞^1,2

1. 西南交通大学信息科学与技术学院, 成都 611756;
2. 四川省云计算与智能技术高校重点实验室(西南交通大学), 成都 611756

收稿日期:2018-12-12 修回日期:2019-03-25 出版日期:2019-06-10 发布日期:2019-06-17
通讯作者: 滕飞
作者简介:李耘书(1991-),男,四川广元人,硕士研究生,主要研究方向:云计算、机器学习:滕飞(1984-),女,山东泰安人,副教授,博士,CCF会员,主要研究方向:云计算、医学信息学、工业大数据挖掘:李天瑞(1969-),男,福建莆田人,教授,博士,CCF会员,主要研究方向:智能信息处理、数据挖掘、云计算。
基金资助:
国家自然科学基金资助项目（61573292）；四川省科技计划项目（2019YJ0214）。

Microoperation-based parameter auto-optimization method of Hadoop

LI Yunshu¹, TENG Fei^1,2, LI Tianrui^1,2

1. School of Information Science and Technology, Southwest Jiaotong University, Chengdu Sichuan 611756, China;
2. Key Laboratory of Cloud Computing and Intelligent Technology of Sichuan Province(Southwest Jiaotong University), Chengdu Sichuan 611756, China

Received:2018-12-12 Revised:2019-03-25 Online:2019-06-10 Published:2019-06-17
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61573292), the Science and Technology Project of Sichuan Province (2019YJ0214).

摘要/Abstract

摘要： Hadoop作为大规模分布式数据处理框架已经在工业界得到广泛的应用，针对手动和经验调优方法中参数空间庞大和运行流程复杂的问题，提出了一种Hadoop参数自动优化的方法和分析框架。首先，对作业运行流程进行解耦，从可变参数直接影响的更细粒度的角度定义微操作，从而分析参数和单次微操作执行时间的关系；然后，利用微操作对作业运行流程进行重构，建立参数和作业运行时间关系的模型；最后，在此模型上应用各类搜索优化算法高效快速得出优化后的系统参数。在terasort和wordcount两个作业类型上进行了实验，实验结果表明，相对于默认参数情况，该方法使作业执行时间分别缩短了至少41%和30%。该方法能够有效提高Hadoop作业执行效率，缩短作业执行时间。

关键词: Hadoop, 参数调优, 微操作, 重构, 搜索算法

Abstract: As a large-scale distributed data processing framework, Hadoop has been widely used in industry during the past few years. Currently manual parameter optimization and experience-based parameter optimization are ineffective due to complex running process and large parameter space. In order to solve this problem, a method and an analytical framework for Hadoop parameter auto-optimization were proposed. Firstly, the operation process of a job was broken down into several microoperations and the microoperations were determined from the angle of finer granularity directly affected by variable parameters, so that the relationship between parameters and the execution time of a single microoperation was able to be analyzed. Then, by reconstructing the job operation process based on microoperations, a model of the relationship between parameters and the execution time of whole job was established. Finally, various searching optimization algorithms were applied on this model to efficiently and quickly obtain the optimized system parameters. Experiments were conducted with two types of jobs, terasort and wordcount. The experimental results show that, compared with the default parameters condition, the proposed method reduce the job execution time by at least 41% and 30% respectively. The proposed method can effectively improve the job execution efficiency of Hadoop and shorten the job execution time.

Key words: Hadoop, parameter optimization, microoperation, reconstitution, search algorithm

中图分类号:

TP311.13

李耘书, 滕飞, 李天瑞. 基于微操作的Hadoop参数自动调优方法[J]. 计算机应用, 2019, 39(6): 1589-1594.

LI Yunshu, TENG Fei, LI Tianrui. Microoperation-based parameter auto-optimization method of Hadoop[J]. Journal of Computer Applications, 2019, 39(6): 1589-1594.

参考文献

[1] DEAN J, GHEMAWAT S. MapReduce:simplified data processing on large clusters[C]//Proceedings of the 6th Conference on Symposium on Opearting Systems Design and Implementation. Berkeley, CA:USENIX Association, 2004:137-149.
[2] CUTTING D. Apache Hadoop[EB/OL]. (2015-02-25)[2018-08-12]. http://hadoop.apache.org.
[3] BABU S. Towards automatic optimization of MapReduce programs[C]//Proceedings of the 1st ACM Symposium on Cloud Computing. New York:ACM, 2010:137-142.
[4] TIPCON T. 7 tips for improving MapReduce performance[EB/OL]. (2009-12-17)[2018-08-12]. http://www.cloudera.com/blog/2009/12/7-tips-for-improving-mapreduce-performance/.
[5] HERODOTOU H, LIM H, LUO G, et al. Starfish:a self-tuning system for big data analytics[C]//Proceedings of the 20115th Biennial Conference on Innovative Data Systems Research. Asilomar, CA:[s.n.], 2011:261-272.
[6] WANG J H, QIU M K, GUO B, et al. Phase-reconfigurable shuffle optimization for Hadoop MapReduce[EB/OL].[2018-08-12]. https://www.onacademic.com/detail/journal_1000038191224210_fd14.html.
[7] YIGITBASI N, WILLKE T L, LIAO G, et al. Towards machine learning-based auto-tuning of MapReduce[C]//Proceedings of the IEEE 21st International Symposium on Modeling, Analysis and Simulation of Computer and Telecommunication Systems. Piscataway, NJ:IEEE, 2013:11-20.
[8] CHAUDHURI S, NARASAYYA V. Self-tuning database systems:a decade of progress[C]//Proceedings of the 2007 International Conference on Very Large Data Bases. Framingham, MA:VLDB Endowment, 2007:3-14.
[9] IPEK E, de SUPINSKI B R, SCHULZ M, et al. An approach to performance prediction for parallel applications[C]//Proceedings of the 2005 European Conference on Parallel Processing, LNCS 3648. Berlin:Springer, 2005:196-205.
[10] SINGER J, KOVOOR G, BROWN G, et al. Garbage collection auto-tuning for Java MapReduce on multi-cores[C]//Proceedings of the 2011 International Symposium on Memory Management. New York:ACM, 2011:109-118.
[11] CHENG D Z, RAO J, GUO Y F, et al. Improving performance of heterogeneous MapReduce clusters with adaptive task tuning[J]. IEEE Transactions on Parallel & Distributed Systems, 2017, 28(3):774-786.
[12] WASI-UR-RAHMAN M, ISLAM N S, LU X, et al. MR-Advisor:a comprehensive tuning tool for advising HPC users to accelerate MapReduce applications on supercomputers[C]//Proceedings of the 28th International Symposium on Computer Architecture and High Performance Computing. Piscataway, NJ:IEEE, 2016:198-205.
[13] 童颖.基于机器学习的Hadoop参数调优方法[D].武汉:华中科技大学,2016:1-52.(TONG Y. Hadoop parameters tuning method based on machine learning[D]. Wuhan:Huazhong University of Science and Technology, 2016:1-52.)
[14] LIAO G, DATTA K, WILLKE T L. Gunther:search-based auto-tuning of MapReduce[C]//Proceedings of the 2013 European Conference on Parallel Processing, LNCS 8097. Berlin:Springer, 2013:406-419.
[15] DEB K. An introduction to genetic algorithms[J]. Sadhana, 1999, 24(4/5):293-315.
[16] 祝春祥,陈世平,陈敏刚.基于递归随机抽样的Hadoop配置优化[J].计算机工程,2016,42(2):26-32.(ZHU C X, CHEN S P, CHEN M G. Configuration optimization of Hadoop based on recursive random sampling[J]. Computer Engineering, 2016, 42(2):26-32.)

基于微操作的Hadoop参数自动调优方法

Microoperation-based parameter auto-optimization method of Hadoop

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	张成, 万源, 强浩鹏. 基于知识蒸馏的深度无监督离散跨模态哈希[J]. 计算机应用, 2021, 41(9): 2523-2531.
[2]	王丹, 梁家敏, 刘金枝, 张友寿. 6G可重构智能表面的相移模型设计[J]. 计算机应用, 2021, 41(9): 2694-2698.
[3]	石雪松, 李宪华, 孙青, 宋韬. 基于人工蜂群与模糊C均值的自适应小波变换的噪声图像分割[J]. 计算机应用, 2021, 41(8): 2312-2317.
[4]	李洋莹, 陈智军, 张子豪, 游兰. 基于改进Elman神经网络的制糖企业原糖需求预测模型[J]. 计算机应用, 2021, 41(7): 2113-2120.
[5]	汤安迪, 韩统, 徐登武, 谢磊. 基于混沌麻雀搜索算法的无人机航迹规划方法[J]. 计算机应用, 2021, 41(7): 2128-2136.
[6]	王先武, 张挺, 吉欣, 杜奕. 基于带梯度惩罚深度卷积生成对抗网络的页岩三维数字岩心重构方法[J]. 计算机应用, 2021, 41(6): 1805-1811.
[7]	蒋林, 施佳琪, 李远成. 可重构结构下合成视点失真变化算法并行设计与实现[J]. 计算机应用, 2021, 41(6): 1734-1740.
[8]	贾鹤鸣, 李瑶, 姜子超, 孙康健. 基于改进共生生物搜索算法的林火图像多阈值分割[J]. 计算机应用, 2021, 41(5): 1465-1470.
[9]	张伟伟, 陶聪, 范岩, 于坤杰, 文笑雨, 张卫正. 改进回溯搜索算法解决光伏模型参数识别问题[J]. 计算机应用, 2021, 41(4): 1199-1206.
[10]	沈忱, 邰凌翔, 彭煜玮. 面向自动参数调优的动态负载匹配方法[J]. 计算机应用, 2021, 41(3): 657-661.
[11]	邓文瀚, 张铭, 王李进, 钟一文. 混合群体增量学习算法求解闭环布局问题[J]. 计算机应用, 2021, 41(1): 95-102.
[12]	王守华, 吴黎荣, 纪元法, 孙希延. 基于格理论的模糊度快速解算方法[J]. 计算机应用, 2020, 40(8): 2299-2304.
[13]	徐志强, 蒋铁钢, 杨立波. 基于随机支撑挑选的广义正交匹配追踪算法[J]. 计算机应用, 2020, 40(4): 1104-1108.
[14]	许英鑫, 孙磊, 赵建成, 郭松辉. 基于蚁群优化算法的虚拟现场可编程门阵列部署策略[J]. 计算机应用, 2020, 40(3): 747-752.
[15]	杨立波, 蒋铁钢, 徐志强. 基于混合梯度的硬阈值追踪算法[J]. 计算机应用, 2020, 40(3): 912-916.