基于Hadoop的三队列作业调度算法

doi:10.11772/j.issn.1001-9081.2014.11.3227

计算机应用 ›› 2014, Vol. 34 ›› Issue (11): 3227-3230.DOI: 10.11772/j.issn.1001-9081.2014.11.3227

基于Hadoop的三队列作业调度算法

朱洁,赵红,李雯睿

南京晓庄学院数学与信息技术学院,南京 211171

收稿日期:2014-05-16 修回日期:2014-06-27 发布日期:2014-12-01 出版日期:2014-11-01
通讯作者: 朱洁
作者简介:朱洁(1979-),女,江苏泰州人,讲师,硕士,主要研究方向:云计算、分布式计算;赵红(1982-),女,黑龙江哈尔滨人,讲师,博士,主要研究方向:人工智能、分布式计算;李雯睿(1981-),女,河南开封人,讲师,博士,主要研究方向:云计算、服务计算。
基金资助:
江苏高校优势学科建设工程项目;江苏省高校自然科学研究项目;南京晓庄学院科研项目

Three-queue job scheduling algorithm based on Hadoop

ZHU Jie,ZHAO Hong,LI Wenrui

School of Mathematics and Information Technology, Nanjing Xiaozhuang University, Nanjing Jiangsu 211171, China

Received:2014-05-16 Revised:2014-06-27 Online:2014-12-01 Published:2014-11-01
Contact: ZHU Jie

摘要/Abstract

摘要：

Hadoop集群单队列作业调度会产生短作业等待、资源利用率低的问题;采用多队列调度可兼顾公平、提高执行效率,但会带来手工配置参数、资源互占、算法复杂等问题。针对上述问题,提出三队列作业调度算法,利用区分作业类型、动态调整作业优先级、配置共享资源池、作业抢占等设计,达到平衡作业需求、简化一般作业调度流程、提升并行执行能力的目的。对短作业占比高,各作业占比均衡以及一般作业为主,偶尔出现长、短作业三种情况与先进先出(FIFO)算法进行了对比实验,结果三队列算法的运行时间均比FIFO算法要少。实验结果表明,在短作业聚集时,三队列算法的执行效率提升并不显著;但当各种作业并存且分布均衡时,效果很明显,这符合了算法设计时短作业优先、一般作业简化流程、兼顾长作业的初衷,提高了作业整体执行效率。

Abstract:

Single queue job scheduling algorithm in homogeneous Hadoop cluster causes short jobs waiting and low utilization rate of resources; multi-queue scheduling algorithms solve problems of unfairness and low execution efficiency, but most of them need setting parameters manually, occupy resources each other and are more complex. In order to resolve these problems, a kind of three-queue scheduling algorithm was proposed. The algorithm used job classifications, dynamic priority adjustment, shared resource pool and job preemption to realize fairness, simplify the scheduling flow of normal jobs and improve concurrency. Comparison experiments with First In First Out (FIFO) algorithm were given under three kinds of situations, including that the percentage of short jobs is high, the percentages of all types of jobs are similar, and the general jobs are major with occasional long and short jobs. The proposed algorithm reduced the running time of jobs. The experimental results show that the execution efficiency increase of the proposed algorithm is not obvious when the major jobs are short ones; however, when the assignments of all types of jobs are balanced, the performance is remarkable. This is consistent with the algorithm design rules: prioritizing the short jobs, simplifying the scheduling flow of normal jobs and considering the long jobs, which improves the scheduling performance.

中图分类号:

朱洁赵红李雯睿. 基于Hadoop的三队列作业调度算法[J]. 计算机应用, 2014, 34(11): 3227-3230.

ZHU Jie ZHAO Hong LI Wenrui. Three-queue job scheduling algorithm based on Hadoop[J]. Journal of Computer Applications, 2014, 34(11): 3227-3230.

参考文献

[1]Cloud computing[EB/OL]. [2013-03-10]. http://en.wikip-edia.org/wiki/Cloud_computing.
[2]DEAN J, GHEMAWAT S. MapReduce: a flexible data processing tool[J]. Communications of the ACM,2010,53(1):72-77.
[3]MapReduce[EB/OL]. [2013-03-10]. http://zh.wikipedia.o-rg/zh-cn/MapReduce.
[4]ZAHARIA M. Job scheduling with the fair and capacity schedulers[EB/OL]. [2013-05-10]. http://www.cs.berk-eley.edu/~matei/talks/2009/hadoop_summit_fair_sched-uler.pdf.
[5]DEAN J, GHEMAWAT S. MapReduce: simplified data processing on large clusters[J]. Communications of the ACM,2008,51(1):107-113.
[6]ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Job scheduling for multi-user MapReduce clusters, 55[R]. Berkeley: University of California, 2009.
[7]Fair scheduler guide[EB/OL]. [2013-04-08]. http://had-oop.apache.org/docs/r1.2.1/Fair_Scheduler.html.
[8]Capacity scheduler guide[EB/OL]. [2013-04-08]. http://hadoop.apache.org/docs/r1.2.1/capacity_scheduler.html.
[9]GUO Z, FOX G, ZHOU M. Investigation of data locality and fairness in MapReduce[C]// Proceedings of the 3rd International Workshop on MapReduce and its Applications. New York: ACM Press, 2012:25-32.
[10]ZAHARIA M, BORTHAKUR D, SARMA J S, et al. Delay scheduling: a simple technique for achieving locality and fairness in cluster scheduling[C]// EuroSys 2010: Proceedings of the 5th European Conference on Computer Systems. New York: ACM Press, 2010:265-278.
[11]
KE H, YANG Q, WANG L, et al. Improved delay-scheduler algorithm in homogeneous Hadoop cluster[J]. Application Research of Computers,2013,30(5):1397-1401.(柯何杨,杨群,王立松,等.同构Hadoop集群环境下改进的延迟调度算法[J].计算机应用研究,2013,30(5):1397-1401.)
[12]JIN W, WANG C. Iteration MapReduce framework for evolution algorithm[J]. Journal of Computer Applications,2013,33(12):3591-3595.(金伟健,王春枝.适于进化算法的迭代式MapReduce框架[J].计算机应用,2013,33(12):3591-3595.)
[13]ZHANG X, GONG K, ZHAO G. Parallel k-medoids algorithm based on MapReduce[J]. Journal of Computer Applications,2013,33(4):1023-1025.(张雪萍,龚康莉,赵广才.基于MapReduce的k-medoids并行算法[J].计算机应用,2013,33(4):1023-1025.)
[14]ZAHARIA M, KONWINSKI A, JOSEPH A D, et al. Improving MapReduce performance in heterogeneous environment[C]// Proceedings of the 8th USENIX Conference on Operating Systems Design and Implementation. Berkeley: USENIX, 2008:29-42.
[15]LI L, TANG Z, LI R. New improvement of the Hadoop relevant data locality scheduling algorithm based on LATE[J]. Computer Science,2011,38(11):67-70.(李丽英,唐卓,李仁发.基于LATE的Hadoop数据局部性改进调度算法[J].计算机科学,2011,38(11):67-70.)
[16]POLO J, CARRERA D, BECERRA Y, et al. Performance-driven task co-scheduling for MapReduce environments[C]// Proceedings of the 12th IEEE/IFIP Network Operations and Management Symposium. Piscataway: IEEE Press, 2010:373-380.
[17]DENG C, FAN T, GAO F. A resource scheduler algorithm based on statistical optimization under Hadoop[J]. Application Research of Computers, 2013,30(2):417-422.(邓传华,范通让,高峰.Hadoop下基于统计最优的资源调度算法[J].计算机应用研究,2013,30(2):417-422.)
[18]Hadoop MapReduce[EB/OL]. [2013-04-12]. http://wiki. apache.org/hadoop/HadoopMapReduce.

基于Hadoop的三队列作业调度算法

Three-queue job scheduling algorithm based on Hadoop

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	侯阳张琼赵紫煊朱正宇张晓博. 基于YOLOv5s的复杂场景下高效烟火检测算法——YOLOv5s-MRD[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[2]	张奇业, 曾心蕊. 带高斯核的支持向量数据描述问题的高效积极集法[J]. 《计算机应用》唯一官方网站, 2024, 44(12): 3808-3814.
[3]	曹晓意罗煦琼李景贺恩锋. 改进人工势场法下的多无人机编队路径规划方法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[4]	袁志超杨磊田井林魏晓威李康顺. 面向复杂约束多目标优化问题的双种群双阶段进化算法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[5]	冷琴, 毛政元. 考虑设施规模决策的两级选址-路径优化[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3513-3520.
[6]	彭庆媛, 王晓峰, 王军霞, 华盈盈, 唐傲, 何飞. 可满足性问题相变研究综述[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3503-3512.
[7]	孙仁科, 皇甫志宇, 陈虎, 李仲年, 许新征. 神经架构搜索综述[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 2983-2994.
[8]	孙安泰, 刘烨, 徐冬梅. 多智能体系统的动态面渐近补偿算法[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 3151-3157.
[9]	颜超英, 张紫仪, 曲映楠, 李秋禹, 郑地翔, 孙丽珺. 基于联盟链的双向拍卖碳交易[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 3240-3245.
[10]	杨志龙邹德旋李灿邵莹莹马乐杰. 融入限制反向学习与柯西-高斯变异的蜣螂优化算法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[11]	姚光磊, 熊菊霞, 杨国武. 基于神经网络优化的花朵授粉算法[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2829-2837.
[12]	力尚龙, 刘建华, 贾鹤鸣. 融合多狩猎协调策略的爬行动物搜索算法[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2818-2828.
[13]	李焱, 潘大志, 郑思情. 多车场带时间窗车辆路径问题的改良自适应大邻域搜索算法[J]. 《计算机应用》唯一官方网站, 2024, 44(6): 1897-1904.
[14]	张倩婷胡丽莹陈黎飞. 时间序列的鲁棒形态表征方法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.
[15]	胡林波倪志伟程家乐刘文涛朱旭辉. 基于融合社区检测的复杂协作众包任务分配方法[J]. 《计算机应用》唯一官方网站, 0, (): 0-0.