基于Hadoop的三队列作业调度算法

doi:10.11772/j.issn.1001-9081.2014.11.3227

计算机应用 ›› 2014, Vol. 34 ›› Issue (11): 3227-3230.DOI: 10.11772/j.issn.1001-9081.2014.11.3227

基于Hadoop的三队列作业调度算法

朱洁,赵红,李雯睿

南京晓庄学院数学与信息技术学院,南京 211171

收稿日期:2014-05-16 修回日期:2014-06-27 出版日期:2014-11-01 发布日期:2014-12-01
通讯作者: 朱洁
作者简介:朱洁(1979-),女,江苏泰州人,讲师,硕士,主要研究方向:云计算、分布式计算;赵红(1982-),女,黑龙江哈尔滨人,讲师,博士,主要研究方向:人工智能、分布式计算;李雯睿(1981-),女,河南开封人,讲师,博士,主要研究方向:云计算、服务计算。
基金资助:
江苏高校优势学科建设工程项目;江苏省高校自然科学研究项目;南京晓庄学院科研项目

Three-queue job scheduling algorithm based on Hadoop

ZHU Jie,ZHAO Hong,LI Wenrui

School of Mathematics and Information Technology, Nanjing Xiaozhuang University, Nanjing Jiangsu 211171, China

Received:2014-05-16 Revised:2014-06-27 Online:2014-11-01 Published:2014-12-01
Contact: ZHU Jie

摘要/Abstract

摘要：

Hadoop集群单队列作业调度会产生短作业等待、资源利用率低的问题;采用多队列调度可兼顾公平、提高执行效率,但会带来手工配置参数、资源互占、算法复杂等问题。针对上述问题,提出三队列作业调度算法,利用区分作业类型、动态调整作业优先级、配置共享资源池、作业抢占等设计,达到平衡作业需求、简化一般作业调度流程、提升并行执行能力的目的。对短作业占比高,各作业占比均衡以及一般作业为主,偶尔出现长、短作业三种情况与先进先出(FIFO)算法进行了对比实验,结果三队列算法的运行时间均比FIFO算法要少。实验结果表明,在短作业聚集时,三队列算法的执行效率提升并不显著;但当各种作业并存且分布均衡时,效果很明显,这符合了算法设计时短作业优先、一般作业简化流程、兼顾长作业的初衷,提高了作业整体执行效率。

Abstract:

Single queue job scheduling algorithm in homogeneous Hadoop cluster causes short jobs waiting and low utilization rate of resources; multi-queue scheduling algorithms solve problems of unfairness and low execution efficiency, but most of them need setting parameters manually, occupy resources each other and are more complex. In order to resolve these problems, a kind of three-queue scheduling algorithm was proposed. The algorithm used job classifications, dynamic priority adjustment, shared resource pool and job preemption to realize fairness, simplify the scheduling flow of normal jobs and improve concurrency. Comparison experiments with First In First Out (FIFO) algorithm were given under three kinds of situations, including that the percentage of short jobs is high, the percentages of all types of jobs are similar, and the general jobs are major with occasional long and short jobs. The proposed algorithm reduced the running time of jobs. The experimental results show that the execution efficiency increase of the proposed algorithm is not obvious when the major jobs are short ones; however, when the assignments of all types of jobs are balanced, the performance is remarkable. This is consistent with the algorithm design rules: prioritizing the short jobs, simplifying the scheduling flow of normal jobs and considering the long jobs, which improves the scheduling performance.

中图分类号:

朱洁赵红李雯睿. 基于Hadoop的三队列作业调度算法[J]. 计算机应用, 2014, 34(11): 3227-3230.

ZHU Jie ZHAO Hong LI Wenrui. Three-queue job scheduling algorithm based on Hadoop[J]. Journal of Computer Applications, 2014, 34(11): 3227-3230.

参考文献

[1]Cloud computing[EB/OL]. [2013-03-10]. http://en.wikip-edia.org/wiki/Cloud_computing.
[2]DEAN J, GHEMAWAT S. MapReduce: a flexible data processing tool[J]. Communications of the ACM,2010,53(1):72-77.
[3]MapReduce[EB/OL]. [2013-03-10]. http://zh.wikipedia.o-rg/zh-cn/MapReduce.
[4]ZAHARIA M. Job scheduling with the fair and capacity schedulers[EB/OL]. [2013-05-10]. http://www.cs.berk-eley.edu/~matei/talks/2009/hadoop_summit_fair_sched-uler.pdf.
[5]DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters[J]. Communications of the ACM,2008,51(1):107-113.
[6]ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Job scheduling for multi-user MapReduce clusters, 55[R]. Berkeley: University of California, 2009.
[7]Fair scheduler guide[EB/OL]. [2013-04-08]. http://had-oop.apache.org/docs/r1.2.1/Fair_Scheduler.html.
[8]Capacity scheduler guide[EB/OL]. [2013-04-08]. http://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.html.
[9]GUO Z, FOX G, ZHOU M. Investigation of data locality and fairness in MapReduce[C]// Proceedings of the 3rd International Workshop on MapReduce and its Applications. New York: ACM Press, 2012:25-32.
[10]ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling[C]// EuroSys 2010: Proceedings of the 5th European Conference on Computer Systems. New York: ACM Press, 2010:265-278.
[11]
KE H, YANG Q, WANG L, et al. Improved delay-scheduler algorithm in homogeneous Hadoop cluster[J]. Application Research of Computers,2013,30(5):1397-1401.(柯何杨,杨群,王立松,等.同构Hadoop集群环境下改进的延迟调度算法[J].计算机应用研究,2013,30(5):1397-1401.)
[12]JIN W, WANG C. Iteration MapReduce framework for evolution algorithm[J]. Journal of Computer Applications,2013,33(12):3591-3595.(金伟健,王春枝.适于进化算法的迭代式MapReduce框架[J].计算机应用,2013,33(12):3591-3595.)
[13]ZHANG X, GONG K, ZHAO G. Parallel k-medoids algorithm based on MapReduce[J]. Journal of Computer Applications,2013,33(4):1023-1025.(张雪萍,龚康莉,赵广才.基于MapReduce的k-medoids并行算法[J].计算机应用,2013,33(4):1023-1025.)
[14]ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environment[C]// Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley: USENIX, 2008:29-42.
[15]LI L, TANG Z, LI R. New improvement of the Hadoop relevant data locality scheduling algorithm based on LATE[J]. Computer Science,2011,38(11):67-70.(李丽英,唐卓,李仁发.基于LATE的Hadoop数据局部性改进调度算法[J].计算机科学,2011,38(11):67-70.)
[16]POLO J, CARRERA D, BECERRA Y, et al. Performance-driven task co-scheduling for MapReduce environments[C]// Proceedings of the 12th IEEE/IFIP Network Operations and Management Symposium. Piscataway: IEEE Press, 2010:373-380.
[17]DENG C, FAN T, GAO F. A resource scheduler algorithm based on statistical optimization under Hadoop[J]. Application Research of Computers, 2013,30(2):417-422.(邓传华,范通让,高峰.Hadoop下基于统计最优的资源调度算法[J].计算机应用研究,2013,30(2):417-422.)
[18]Hadoop MapReduce[EB/OL]. [2013-04-12]. http://wiki. apache.org/hadoop/HadoopMapReduce.

基于Hadoop的三队列作业调度算法

Three-queue job scheduling algorithm based on Hadoop

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	肖智豪胡志华朱琳. 求解冷链物流时间依赖型车辆路径问题的混合自适应大邻域搜索算法[J]. 计算机应用, 0, (): 0-0.
[2]	沙林秀，聂凡，高倩，孟号. 基于布朗运动与梯度信息的交替优化算法[J]. 计算机应用, 0, (): 0-0.
[3]	董永峰孙跃华高立超韩鹏季海鹏. 基于改进一维卷积和双向长短期记忆神经网络的故障诊断方法[J]. 计算机应用, 0, (): 0-0.
[4]	李大海刘庆腾艾志刚王振东. 基于动态D向分割和混沌扰动的阴阳对算法[J]. 计算机应用, 0, (): 0-0.
[5]	雷鹰郑万波魏嵬夏云霓李晓波刘诚武谢洪. 基于概率性能感知演化博弈策略的混合“云+边”环境中任务卸载方法 [J]. 计算机应用, 0, (): 0-0.
[6]	乔钢柱王瑞孙超利. 基于分解的高维多目标改进进化算法[J]. 计算机应用, 0, (): 0-0.
[7]	朱诚潘旭华张勇. 基于趋化校正的哈里斯鹰优化算法[J]. 计算机应用, 0, (): 0-0.
[8]	平凡汤小春潘彦宇李战怀. 不规则任务在图形处理器集群上的调度策略[J]. 计算机应用, 0, (): 0-0.
[9]	杨杰张名扬芮晓彬王志晓. 融合节点覆盖范围和结构洞的影响力最大化算法[J]. 计算机应用, 0, (): 0-0.
[10]	汤安迪, 韩统, 徐登武, 谢磊. 混沌精英哈里斯鹰优化算法[J]. 计算机应用, 2021, 41(8): 2265-2272.
[11]	李蒙蒙, 秦伟, 刘艺, 刁兴春. 结合头脑风暴优化的混合蚁群优化算法[J]. 计算机应用, 2021, 41(8): 2412-2417.
[12]	张闻强, 邢征, 杨卫东. 基于多区域采样策略的混合粒子群优化求解多目标柔性作业车间调度问题[J]. 计算机应用, 2021, 41(8): 2249-2257.
[13]	张祥飞, 鲁宇明, 张平生. 基于协同进化的约束多目标优化算法[J]. 计算机应用, 2021, 41(7): 2012-2018.
[14]	张萌, 李维华. 用户互动表示下的影响力最大化算法[J]. 计算机应用, 2021, 41(7): 1964-1969.
[15]	王宇, 刘燕丽, 陈劭武. 基于顶点冲突学习的最大公共子图算法[J]. 计算机应用, 2021, 41(6): 1756-1760.