计算机应用 ›› 2014, Vol. 34 ›› Issue (4): 1010-1013.DOI: 10.11772/j.issn.1001-9081.2014.04.1010

• 先进计算 • 上一篇    下一篇

基于匹配规则的MapReduce任务调度模型

金伟健1,王春枝2   

  1. 1. 义乌工商职业技术学院 机电信息分院,浙江 义乌 322000;
    2. 湖北工业大学 计算机学院,武汉 430068
  • 收稿日期:2013-10-23 修回日期:2013-12-13 出版日期:2014-04-01 发布日期:2014-04-29
  • 通讯作者: 金伟健
  • 作者简介:金伟健(1982-),男,浙江义乌人,讲师,硕士研究生,主要研究方向:计算机网络安全、云计算;
    王春枝(1963-),女,湖北武汉人,教授,博士,CCF高级会员,主要研究方向:计算机网络协同教育、计算机信息安全。
  • 基金资助:

    国家自然科学基金项目资助项目

MapReduce tasks scheduling model based on matching rules

JIN Weijian1,WANG Chunzhi2   

  1. 1. School of Electro-Mechanical and Information Technology, Yiwu Industrial and Commercial College, Yiwu Zhejiang 322000, China
    2. School of Computer Science, Hubei University of Technology, Wuhan Hubei 430068, China
  • Received:2013-10-23 Revised:2013-12-13 Online:2014-04-01 Published:2014-04-29
  • Contact: JIN Weijian

摘要:

基于开源云计算平台Hadoop的MapReduce是当前流行的分布式计算框架之一,然而其先进先出(FIFO)调度算法存在资源利用效率低下的问题。提出了一种基于资源匹配规则的MapReduce任务调度模型并进行了算法实现。该调度模型通过获取任务的资源需求与计算节点的剩余资源,依据资源的匹配性进行任务分配,提高了系统的资源使用效率。首先对MapReduce的调度过程进行建模,提出了资源及匹配度的量化定义和相应的计算公式;然后给出了资源测量的具体方法及算法实现;最后利用TeraSort、GrepCount和WordCount任务与FIFO调度算法进行实验对比,实验结果显示,最好的情况下,提出的调度模型任务完成时间减少了22.19%,而最差情况下的吞吐量也提高了25.39%。

Abstract:

MapReduce is one of the popular distributed computing frameworks based on an open source cloud platform named Hadoop. However, the First-In First-Out (FIFO) scheduling algorithm of MapReduce is inefficient in resources utilization. A new tasks scheduling model based on resources matching rules was proposed and implemented. After obtaining the tasks resources requirement and remainder resources on computing nodes, the model assigned tasks to computing nodes based on resources matching degree to improve the usage efficiency of system resources. First of all, the model for MapReduce scheduling was established, the quantitative definition of resources and matching degree were given, and the corresponding calculation formulas were put forward. Second, the specific methods of resource measurement and the implementation of the algorithm were introduced. Compared with FIFO scheduling algorithm on TeraSort, GrepCount and WordCount, the experimental results show that the proposed model reduces by 22.19% in tasks completion time in the best case, and increases by 25.39% in throughput even in the worst case.

中图分类号: