Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters

doi:10.11772/j.issn.1001-9081.2016.05.1216

Journal of Computer Applications ›› 2016, Vol. 36 ›› Issue (5): 1216-1221.DOI: 10.11772/j.issn.1001-9081.2016.05.1216

Previous Articles Next Articles

Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters

KE Zunwang¹, YU Jiong¹, LIAO Bin²

1. College of Software, Xinjiang University, Urumqi Xinjiang 830008, China;
2. College of Statistics and Information, Xinjiang University of Finance and Economics, Urumqi Xinjiang 830012, China

Received:2015-11-05 Revised:2015-12-23 Online:2016-05-09 Published:2016-05-10
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61462079,61262088,61562086,61363083).

适应异构集群的Mesos多资源调度DRF增强算法

柯尊旺¹, 于炯¹, 廖彬²

1. 新疆大学软件学院, 乌鲁木齐 830008;
2. 新疆财经大学统计与信息学院, 乌鲁木齐 830012

通讯作者: 柯尊旺
作者简介:柯尊旺(1984-),男,湖北黄石人,硕士研究生,主要研究方向:云资源调度、数据挖掘;于炯(1964-),男,北京人,教授,博士生导师,博士,高级CCF会员,主要研究方向:网络安全、分布式计算;廖彬(1986-),男,四川内江人,副教授,博士,CCF会员,主要研究方向:数据库系统理论、绿色计算、数据挖掘。
基金资助:
国家自然科学基金资助项目(61462079,61262088,61562086,61363083)。

Abstract

Abstract: The fairness of multi-resource allocation is one of the most important indicators in the resource scheduling subsystem, Dominant Resource Fairness (DRF), as a general resource allocation algorithm for multi-resources scenarios, it may be unfair in heterogeneous cluster environment. On the basis of the research on the DRF multi-resource fair allocation algorithm under Mesos framework environment, meDRF allocation algorithm was designed and implemented to evaluate the influence factors of the performance of the server. The machine performance scores of computing nodes, as the dominant factor of DRF share calculation, made computing tasks have equal chance to obtain high quality computing resources and poor computing resources. Experiments were conducted by using K-means, Bayes and PageRank jobs under Hadoop. The experimental results show that, compared with DRF allocation algorithm, the meDRF algorithm can reflect more fairness of the allocation of resources, and the allocation of resources has better stability, which effectively improves the utilization of system resources.

Key words: resource allocation, Dominant Resource Fairness (DRF) allocation algorithm, fairness, Mesos

摘要： 云计算集群环境下多资源分配的公平性是考量资源调度子系统最重要的指标之一,DRF作为通用的多资源公平分配算法,在异构异质的集群环境下可能有失公平性。在研究Mesos框架中DRF多资源公平分配算法的基础上,设计并实现了增加机器性能评估影响因子的meDRF分配算法。将计算节点的机器性能得分,作为DRF主导份额计算的因子,使得计算任务有均等的机会获得优质计算资源和劣质计算资源。通过选取K-means、Bayes及PageRank等多种作业进行实验,实验结果表明:meDRF较DRF分配算法更能体现多资源分配的公平性,且资源分配具有更好的稳定性,能有效提高系统资源的利用率。

关键词: 资源分配, DRF分配算法, 公平性, Mesos

CLC Number:

TP311

KE Zunwang, YU Jiong, LIAO Bin. Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters[J]. Journal of Computer Applications, 2016, 36(5): 1216-1221.

柯尊旺, 于炯, 廖彬. 适应异构集群的Mesos多资源调度DRF增强算法[J]. 计算机应用, 2016, 36(5): 1216-1221.

References

[1] SHVACHKO K, KUANG H, RADIA S, et al. The Hadoop distributed file system[C]//Proceedings of the 2010 IEEE 26th Symposium on Mass Storage Systems and Technologies. Piscataway, NJ:IEEE, 2010:1-10.
[2] WANG L, WANG Y, XIE Y. Implementation of a parallel algorithm based on a spark cloud computing platform[J]. Algorithms, 2015, 8(3):407-414.
[3] LEE B S, KANAGAVELU R, AUNG K M M. An efficient flow cache algorithm with improved fairness in software-defined data center networks[C]//Proceedings of the 2013 IEEE 2nd International Conference on Cloud Networking. Piscataway, NJ:IEEE, 2013:18-24.
[4] DEAN J, GHEMAWAT S. MapReduce:simplified data processing on large clusters[C]//Proceedings of the 6th Conference on Symposium on Opearting Systems Design & Implementation. Berkeley, CA:USENIX Association, 2004:107-113.
[5] VERNICA R, CAREY M J, LI C. Efficient parallel set-similarity joins using MapReduce[C]//Proceedings of the 2010 ACM SIGMOD International Conference on Management of Data. New York:ACM, 2010:495-506.
[6] 覃雄派, 王会举, 杜小勇, 等.大数据分析:RDBMS与MapReduce的竞争与共生[J].软件学报, 2012, 23(1):32-45.(QIN X P, WANG H J, DU X Y, et al. Big data analysis:competition and symbiosis of RDBMS and MapReduce[J]. Journal of Software, 2012, 23(1):32-45.)
[7] 张雪萍, 龚康莉, 赵广才. 基于MapReduce的K-Medoids并行算法[J]. 计算机应用, 2013, 33(4):1023-1025. (ZHANG X P, GONG K L, ZHAO G C. Parallel K-Medoids algorithm based on MapReduce[J]. Journal of Computer Applications, 2013, 33(4):1023-1025.)
[8] 亓开元, 韩燕波, 赵卓峰, 等.支持高并发数据流处理的MapReduce中间结果缓存[J]. 计算机研究与发展, 2013, 50(1):111-121.(QI K Y, HAN Y B, ZHAO Z F, et al. MapReduce intermediate result cache for concurrent data stream processing[J]. Journal of Computer Research and Development, 2013, 50(1):111-121.)
[9] 顾荣, 王芳芳, 袁春风, 等. YARM:基于MapReduce的高效可扩展的语义推理引擎[J]. 计算机学报, 2015, 38(1):74-85. (GU R, WANG F F, YUAN C F, et al. YARM:efficient and scalable semantic reasoning engine based on MapReduce[J]. Chinese Journal of Computers, 2015, 38(1):74-85.)
[10] 王习特, 申德荣, 于戈, 等. MapReduce集群中最大收益问题的研究[J]. 计算机学报, 2015, 38(1):109-121.(WANG X T, SHEN D R, YU G, et al. Research on maximum benefit problem in a MapReduce cluster[J]. Chinese Journal of Computers, 2015, 38(1):109-121.)
[11] TANG S, LEE B S, HE B. DynamicMR:a dynamic slot allocation optimization framework for MapReduce clusters[J]. IEEE Transactions on Cloud Computing, 2014, 2(3):333-347.
[12] 夏祎. Hadoop平台下的作业调度算法研究与改进[D]. 广州:华南理工大学, 2010:30-40. (XIA Y. Research and improvement of Hadoop job scheduling algorithm[D]. Guangzhou:South China University of Technology, 2010:30-40.)
[13] 赵春燕.云环境下作业调度算法研究与实现[D]. 北京:北京交通大学, 2009:36-37. (ZHAO C Y. Research and implementation of a cloud environment job scheduling algorithm[D]. Beijing:Beijing Jiaotong University, 2009:36-37.)
[14] 最大最小公平算法[EB/OL].[2015-10-22].https://en.wikipedia.org/wiki/Max-min_fairness.(Max-min fairness[EB/OL].[2015-10-22]. https://en.wikipedia.org/wiki/Max-min_fairness.)
[15] ASADPOUR A, SABERI A. An approximation algorithm for max-min fair allocation of indivisible goods[J]. SIAM Journal on Computing, 2010, 39(7):2970-2989.
[16] GHODSI A, ZAHARIA M, HINDMAN B, et al. Dominant resource fairness:fair allocation of multiple resource types[C]//Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX Association, 2011:323-336.
[17] 卢笛, 马建峰, 王一川, 等.云计算下保障公平性的多资源分配算法[J]. 西安电子科技大学学报(自然科学版), 2014, 41(3):162-168. (LU D, MA J F, WANG Y C, et al. Enhanced fairness-based multi-resource allocation algorithm for cloud computing[J]. Journal of Xidian University (Natural Science), 2014, 41(3):162-168.)
[18] Apache Mesos Documentation[EB/OL].[2015-10-03]. http://mesos.apache.org/documentation/latest/index.html.
[19] HINDMAN B, KONWINSKI A, ZAHARIA M, et al. Mesos:a platform for fine-grained resource sharing in the data center[C]//Proceedings of the 8th USENIX Conference on Networked Systems Design and Implementation. Berkeley, CA:USENIX, 2011:429-483.
[20] 霍菁, 石京燕, 孙功星, 等.一种改进的DRF算法对BESⅢ集群资源管理的优化[J]. 核电子学与探测技术, 2014, 34(10):1153-1158. (HUO J, SHI J Y, SUN G X, et al. The optimization of BESⅢ cluster resource management by using the improved DRF algorithm[J]. Nuclear Electronics & Detection Technology, 2014, 34(10):1153-1158.)
[21] BOUVERET S, LANG J. Efficiency and envy-freeness in fair division of indivisible goods:logical representation and complexity[J]. Journal of Artificial Intelligence Research, 2008, 32(1):525-564.
[22] BOLDI P, SANTINI M, VIGNA S. PageRank:functional dependencies[J]. ACM Transactions on Information Systems, 2009, 27(4):1139-1141.

Adaptive multi-resource scheduling dominant resource fairness algorithm for Mesos in heterogeneous clusters

适应异构集群的Mesos多资源调度DRF增强算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	Junna ZHANG, Xinxin WANG, Tianze LI, Xiaoyan ZHAO, Peiyan YUAN. Task offloading method based on dynamic service cache assistance [J]. Journal of Computer Applications, 2024, 44(5): 1493-1500.
[2]	Rui TANG, Chuanlin PANG, Ruizhi ZHANG, Chuan LIU, Shibo YUE. DDPG-based resource allocation in D2D communication-empowered cellular network [J]. Journal of Computer Applications, 2024, 44(5): 1562-1569.
[3]	Hualiang LUO, Quanzhong LI, Qi ZHANG. Robust resource allocation optimization in cognitive wireless network integrating information communication and over-the-air computation [J]. Journal of Computer Applications, 2024, 44(4): 1195-1202.
[4]	Fatang CHEN, Miao HUANG, Yufeng JIN. Resource allocation algorithm for low earth orbit satellites oriented to user demand [J]. Journal of Computer Applications, 2024, 44(4): 1242-1247.
[5]	Rui TANG, Shibo YUE, Ruizhi ZHANG, Chuan LIU, Chuanlin PANG. Energy efficiency optimization mechanism for UAV-assisted and non-orthogonal multiple access-enabled data collection system [J]. Journal of Computer Applications, 2024, 44(4): 1209-1218.
[6]	Xintong QIN, Zhengyu SONG, Tianwei HOU, Feiyue WANG, Xin SUN, Wei LI. Channel access and resource allocation algorithm for adaptive p-persistent mobile ad hoc network [J]. Journal of Computer Applications, 2024, 44(3): 863-868.
[7]	Yongjian MA, Xuhua SHI, Peiyao WANG. Constrained multi-objective evolutionary algorithm based on two-stage search and dynamic resource allocation [J]. Journal of Computer Applications, 2024, 44(1): 269-277.
[8]	Lei LI, Guofu ZHANG, Zhaopin SU, Feng YUE. Software testing resource allocation algorithm for dynamic changes in architecture [J]. Journal of Computer Applications, 2023, 43(7): 2261-2270.
[9]	Jieqin WANG, Shihyang LIN, Shiming PENG, Shuo JIA, Miaohui YANG. Hierarchical resource allocation mechanism of cooperative mobile edge computing [J]. Journal of Computer Applications, 2022, 42(8): 2501-2510.
[10]	Yu LI, Xiping HE, Lianggui TANG. Multi-user computation offloading and resource optimization policy based on device-to-device communication [J]. Journal of Computer Applications, 2022, 42(5): 1538-1546.
[11]	Junjie ZHANG, Runhe QIU. Joint optimization of user association and resource allocation in cognitive radio ultra-dense networks to improve genetic algorithm [J]. Journal of Computer Applications, 2022, 42(12): 3856-3862.
[12]	Zhi LI, Jianbin XUE. Task offloading and resource allocation based on simulated annealing algorithm in C-V2X internet of vehicles [J]. Journal of Computer Applications, 2022, 42(10): 3140-3147.
[13]	Zhisheng SHAO, Guofu ZHANG, Zhaopin SU, Lei LI. Dynamic testing resource allocation algorithm based on software architecture and generalized differential evolution [J]. Journal of Computer Applications, 2021, 41(12): 3692-3701.
[14]	HUANG Dongyan, FU Zhongwei, WANG Bo. Revenue maximization strategy for mobile-edge computing server with limited computing resources [J]. Journal of Computer Applications, 2020, 40(3): 765-769.
[15]	HUANG Xiuli, HUANG Jin, YU Pengfei, MIAO Weiwei, YANG Ruxia, LI Yijing, YU Peng. Security-risk-oriented distributed resource allocation method in power wireless private network [J]. Journal of Computer Applications, 2020, 40(12): 3586-3593.