适应异构集群的Mesos多资源调度DRF增强算法

doi:10.11772/j.issn.1001-9081.2016.05.1216

计算机应用 ›› 2016, Vol. 36 ›› Issue (5): 1216-1221.DOI: 10.11772/j.issn.1001-9081.2016.05.1216

适应异构集群的Mesos多资源调度DRF增强算法

柯尊旺¹, 于炯¹, 廖彬²

1. 新疆大学软件学院, 乌鲁木齐 830008;
2. 新疆财经大学统计与信息学院, 乌鲁木齐 830012

收稿日期:2015-11-05 修回日期:2015-12-23 出版日期:2016-05-10 发布日期:2016-05-09
通讯作者: 柯尊旺
作者简介:柯尊旺(1984-),男,湖北黄石人,硕士研究生,主要研究方向:云资源调度、数据挖掘;于炯(1964-),男,北京人,教授,博士生导师,博士,高级CCF会员,主要研究方向:网络安全、分布式计算;廖彬(1986-),男,四川内江人,副教授,博士,CCF会员,主要研究方向:数据库系统理论、绿色计算、数据挖掘。
基金资助:
国家自然科学基金资助项目(61462079,61262088,61562086,61363083)。

Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters

KE Zunwang¹, YU Jiong¹, LIAO Bin²

1. College of Software, Xinjiang University, Urumqi Xinjiang 830008, China;
2. College of Statistics and Information, Xinjiang University of Finance and Economics, Urumqi Xinjiang 830012, China

Received:2015-11-05 Revised:2015-12-23 Online:2016-05-10 Published:2016-05-09
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61462079,61262088,61562086,61363083).

摘要/Abstract

摘要： 云计算集群环境下多资源分配的公平性是考量资源调度子系统最重要的指标之一,DRF作为通用的多资源公平分配算法,在异构异质的集群环境下可能有失公平性。在研究Mesos框架中DRF多资源公平分配算法的基础上,设计并实现了增加机器性能评估影响因子的meDRF分配算法。将计算节点的机器性能得分,作为DRF主导份额计算的因子,使得计算任务有均等的机会获得优质计算资源和劣质计算资源。通过选取K-means、Bayes及PageRank等多种作业进行实验,实验结果表明:meDRF较DRF分配算法更能体现多资源分配的公平性,且资源分配具有更好的稳定性,能有效提高系统资源的利用率。

关键词: 资源分配, DRF分配算法, 公平性, Mesos

Abstract: The fairness of multi-resource allocation is one of the most important indicators in the resource scheduling subsystem, Dominant Resource Fairness (DRF), as a general resource allocation algorithm for multi-resources scenarios, it may be unfair in heterogeneous cluster environment. On the basis of the research on the DRF multi-resource fair allocation algorithm under Mesos framework environment, meDRF allocation algorithm was designed and implemented to evaluate the influence factors of the performance of the server. The machine performance scores of computing nodes, as the dominant factor of DRF share calculation, made computing tasks have equal chance to obtain high quality computing resources and poor computing resources. Experiments were conducted by using K-means, Bayes and PageRank jobs under Hadoop. The experimental results show that, compared with DRF allocation algorithm, the meDRF algorithm can reflect more fairness of the allocation of resources, and the allocation of resources has better stability, which effectively improves the utilization of system resources.

Key words: resource allocation, Dominant Resource Fairness (DRF) allocation algorithm, fairness, Mesos

中图分类号:

TP311

柯尊旺, 于炯, 廖彬. 适应异构集群的Mesos多资源调度DRF增强算法[J]. 计算机应用, 2016, 36(5): 1216-1221.

KE Zunwang, YU Jiong, LIAO Bin. Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters[J]. Journal of Computer Applications, 2016, 36(5): 1216-1221.

参考文献

[1] SHVACHKO K, KUANG H, RADIA S, et al. The Hadoop distributed file system[C]//Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies. Piscataway, NJ:IEEE, 2010:1-10.
[2] WANG L, WANG Y, XIE Y. Implementation of a parallel algorithm based on a spark cloud computing platform[J]. Algorithms, 2015, 8(3):407-414.
[3] LEE B S, KANAGAVELU R, AUNG K M M. An efficient flow cache algorithm with improved fairness in software-defined data center networks[C]//Proceedings of the 2013 IEEE 2nd International Conference on Cloud Networking. Piscataway, NJ:IEEE, 2013:18-24.
[4] DEAN J, GHEMAWAT S. MapReduce:simplified data processing on large clusters[C]//Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation. Berkeley, CA:USENIX Association, 2004:107-113.
[5] VERNICA R, CAREY M J, LI C. Efficient parallel set-similarity joins using MapReduce[C]//Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2010:495-506.
[6] 覃雄派, 王会举, 杜小勇, 等.大数据分析:RDBMS与MapReduce的竞争与共生[J].软件学报, 2012, 23(1):32-45.(QIN X P, WANG H J, DU X Y, et al. Big data analysis:competition and symbiosis of RDBMS and MapReduce[J]. Journal of Software, 2012, 23(1):32-45.)
[7] 张雪萍, 龚康莉, 赵广才. 基于MapReduce的K-Medoids并行算法[J]. 计算机应用, 2013, 33(4):1023-1025. (ZHANG X P, GONG K L, ZHAO G C. Parallel K-Medoids algorithm based on MapReduce[J]. Journal of Computer Applications, 2013, 33(4):1023-1025.)
[8] 亓开元, 韩燕波, 赵卓峰, 等.支持高并发数据流处理的MapReduce中间结果缓存[J]. 计算机研究与发展, 2013, 50(1):111-121.(QI K Y, HAN Y B, ZHAO Z F, et al. MapReduce intermediate result cache for concurrent data stream processing[J]. Journal of Computer Research and Development, 2013, 50(1):111-121.)
[9] 顾荣, 王芳芳, 袁春风, 等. YARM:基于MapReduce的高效可扩展的语义推理引擎[J]. 计算机学报, 2015, 38(1):74-85. (GU R, WANG F F, YUAN C F, et al. YARM:efficient and scalable semantic reasoning engine based on MapReduce[J]. Chinese Journal of Computers, 2015, 38(1):74-85.)
[10] 王习特, 申德荣, 于戈, 等. MapReduce集群中最大收益问题的研究[J]. 计算机学报, 2015, 38(1):109-121.(WANG X T, SHEN D R, YU G, et al. Research on maximum benefit problem in a MapReduce cluster[J]. Chinese Journal of Computers, 2015, 38(1):109-121.)
[11] TANG S, LEE B S, HE B. DynamicMR:a dynamic slot allocation optimization framework for MapReduce clusters[J]. IEEE Transactions on Cloud Computing, 2014, 2(3):333-347.
[12] 夏祎. Hadoop平台下的作业调度算法研究与改进[D]. 广州:华南理工大学, 2010:30-40. (XIA Y. Research and improvement of Hadoop job scheduling algorithm[D]. Guangzhou:South China University of Technology, 2010:30-40.)
[13] 赵春燕.云环境下作业调度算法研究与实现[D]. 北京:北京交通大学, 2009:36-37. (ZHAO C Y. Research and implementation of a cloud environment job scheduling algorithm[D]. Beijing:Beijing Jiaotong University, 2009:36-37.)
[14] 最大最小公平算法[EB/OL].[2015-10-22].https://en.wikipedia.org/wiki/Max-min_fairness.(Max-min fairness[EB/OL].[2015-10-22]. https://en.wikipedia.org/wiki/Max-min_fairness.)
[15] ASADPOUR A, SABERI A. An approximation algorithm for max-min fair allocation of indivisible goods[J]. SIAM Journal on Computing, 2010, 39(7):2970-2989.
[16] GHODSI A, ZAHARIA M, HINDMAN B, et al. Dominant resource fairness:fair allocation of multiple resource types[C]//Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2011:323-336.
[17] 卢笛, 马建峰, 王一川, 等.云计算下保障公平性的多资源分配算法[J]. 西安电子科技大学学报(自然科学版), 2014, 41(3):162-168. (LU D, MA J F, WANG Y C, et al. Enhanced fairness-based multi-resource allocation algorithm for cloud computing[J]. Journal of Xidian University (Natural Science), 2014, 41(3):162-168.)
[18] Apache Mesos Documentation[EB/OL].[2015-10-03]. http://mesos.apache.org/documentation/latest/index.html.
[19] HINDMAN B, KONWINSKI A, ZAHARIA M, et al. Mesos:a platform for fine-grained resource sharing in the data center[C]//Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX, 2011:429-483.
[20] 霍菁, 石京燕, 孙功星, 等.一种改进的DRF算法对BESⅢ集群资源管理的优化[J]. 核电子学与探测技术, 2014, 34(10):1153-1158. (HUO J, SHI J Y, SUN G X, et al. The optimization of BESⅢ cluster resource management by using the improved DRF algorithm[J]. Nuclear Electronics & Detection Technology, 2014, 34(10):1153-1158.)
[21] BOUVERET S, LANG J. Efficiency and envy-freeness in fair division of indivisible goods:logical representation and complexity[J]. Journal of Artificial Intelligence Research, 2008, 32(1):525-564.
[22] BOLDI P, SANTINI M, VIGNA S. PageRank:functional dependencies[J]. ACM Transactions on Information Systems, 2009, 27(4):1139-1141.

适应异构集群的Mesos多资源调度DRF增强算法

Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	黄冬艳, 付中卫, 王波. 计算资源受限的移动边缘计算服务器收益优化策略[J]. 计算机应用, 2020, 40(3): 765-769.
[2]	黄秀丽, 黄进, 于鹏飞, 缪巍巍, 杨如侠, 李怡静, 喻鹏. 电力无线专网中面向安全风险的分布式资源分配方法[J]. 计算机应用, 2020, 40(12): 3586-3593.
[3]	程文亮, 王志宏, 周虞, 过弋, 赵俊锋. 面向外汇市场监测的分布式计算框架设计[J]. 计算机应用, 2020, 40(1): 173-180.
[4]	李叶飞, 徐超, 许道强, 邹云峰, 张晓达, 钱柱中. 基于累计工作量的在线大数据分析作业调度算法[J]. 计算机应用, 2019, 39(8): 2431-2437.
[5]	徐雅斌, 彭宏恩. 基于需求预测的PaaS平台资源分配方法[J]. 计算机应用, 2019, 39(6): 1583-1588.
[6]	安鑫, 张影, 康安, 陈田, 李建华. 基于机器学习的异构多核处理器系统在线映射方法[J]. 计算机应用, 2019, 39(6): 1753-1759.
[7]	余韵, 连晓灿, 朱宇航, 谭国平. 增强现实场景下移动边缘计算资源分配优化方法[J]. 计算机应用, 2019, 39(1): 22-25.
[8]	李中捷, 谢东朋. 异构蜂窝网络中联合功率控制的终端直通通信资源分配[J]. 计算机应用, 2018, 38(9): 2610-2615.
[9]	刘旭东, 张学杰, 张骥先, 李伟东, 张静. 基于竞价的租赁车辆资源分配和定价机制[J]. 计算机应用, 2018, 38(8): 2423-2430.
[10]	杜雪灵, 孟学雷, 杨贝, 汤霖. 考虑公平性的面向多灾点需求应急资源调度[J]. 计算机应用, 2018, 38(7): 2089-2094.
[11]	陈发堂, 邢苹苹, 杨艳娟. 窄带蜂窝物联网终端上行资源调度的分析与设计[J]. 计算机应用, 2018, 38(11): 3270-3274.
[12]	吴诗奎, 王艳. 软件定义网络的异构无线网络资源分配框架[J]. 计算机应用, 2018, 38(11): 3293-3298.
[13]	王运盛, 雷航. 综合模块化航空电子分区配置启动机制[J]. 计算机应用, 2017, 37(6): 1808-1813.
[14]	刘备, 谭新明, 曹文彬. Spark Streaming动态资源分配策略[J]. 计算机应用, 2017, 37(6): 1574-1579.
[15]	张海波, 彭星萤, 陈善学. 宏-飞蜂窝双层网络中基于毫微微基站分组的资源分配[J]. 计算机应用, 2017, 37(5): 1311-1316.