Performance interference analysis and prediction for distributed machine learning jobs

doi:10.11772/j.issn.1001-9081.2021061404

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (6): 1649-1655.DOI: 10.11772/j.issn.1001-9081.2021061404

Special Issue: 2021年全国开放式分布与并行计算学术年会(DPCS 2021)论文

• National Open Distributed and Parallel Computing Conference 2021 (DPCS 2021） • Next Articles

Performance interference analysis and prediction for distributed machine learning jobs

Hongliang LI¹^,², Nong ZHANG¹, Ting SUN¹, Xiang LI¹()

^1.College of Computer Science and Technology，Jilin University，Changchun Jilin 130012，China
^2.Key Laboratory of Symbolic Computation and Knowledge Engineering of the Ministry of Education （Jilin University），Changchun Jilin 130012，China

Received:2021-08-05 Revised:2021-10-14 Accepted:2021-10-20 Online:2022-06-22 Published:2022-06-10
Contact: Xiang LI
About author:LI Hongliang，born in 1983，Ph. D.，associate professor. His research interests include distributed computing， high performance computing.
ZHANG Nong， born in 1999， M. S.candidate. His research interests include distributed computing， high performance computing.
SUN Ting， born in 1996， M. S. Her research interests include cloud computing， high performance computing.
Supported by:
National Key Research and Development Program of China(2017YFC1502306);National Natural Science Foundation of China(61602205)

分布式机器学习作业性能干扰分析与预测

李洪亮¹^,², 张弄¹, 孙婷¹, 李想¹()

^1.吉林大学计算机科学与技术学院，长春 130012
^2.符号计算与知识工程教育部重点实验室（吉林大学），长春 130012

通讯作者: 李想
作者简介:李洪亮（1983—），男，吉林长春人，副教授，博士，CCF会员，主要研究方向：分布式计算、高性能计算
张弄（1999—），男，四川南充人，硕士研究生，主要研究方向：分布式计算、高性能计算
孙婷（1996—），女，吉林洮南人，硕士，主要研究方向：云计算、高性能计算
基金资助:
国家重点研发计划项目(2017YFC1502306);国家自然科学基金资助项目(61602205)

Abstract

Abstract:

By analyzing the problem of job performance interference in distributed machine learning， it is found that performance interference is caused by the uneven allocation of GPU resources such as memory overload and bandwidth competition， and to this end， a mechanism for quickly predicting performance interference between jobs was designed and implemented， which can adaptively predict the degree of job interference according to the given GPU parameters and job types. First， the GPU parameters and interference rates during the operation of distributed machine learning jobs were obtained through experiments， and the influences of various parameters on performance interference were analyzed. Second， some GPU parameter-interference rate models were established by using multiple prediction technologies to analyze the job interference rate errors. Finally， an adaptive job interference rate prediction algorithm was proposed to automatically select the prediction model with the smallest error for a given equipment environment and job set to predict the job interference rates quickly and accurately. By selecting five commonly used neural network tasks， experiments were designed on two GPU devices and the results were analyzed. The results show that the proposed Adaptive Interference Prediction （AIP） mechanism can quickly complete the selection of prediction model and the performance interference prediction without providing any pre-assumed information， it has comsumption time less than 300 s and achieves prediction error rate in the range of 2% to 13%， which can be applied to scenarios such as job scheduling and load balancing.

Key words: distributed machine learning, performance interference, cluster scheduling, resource sharing, interference prediction

摘要：

通过分析分布式机器学习中作业性能干扰的问题，发现性能干扰是由于内存过载、带宽竞争等GPU资源分配不均导致的，为此设计并实现了快速预测作业间性能干扰的机制，该预测机制能够根据给定的GPU参数和作业类型自适应地预测作业干扰程度。首先，通过实验获取分布式机器学习作业运行时的GPU参数和干扰率，并分析出各类参数对性能干扰的影响；其次，依托多种预测技术建立GPU参数-干扰率模型进行作业干扰率误差分析；最后，建立自适应的作业干扰率预测算法，面向给定的设备环境和作业集合自动选择误差最小的预测模型，快速、准确地预测作业干扰率。选取5种常用的神经网络作业，在两种GPU设备上设计实验并进行结果分析。结果显示，所提出的自适应干扰预测（AIP）机制能够在不提供任何预先假设信息的前提下快速完成预测模型的选择和性能干扰预测，耗时在300 s以内，预测干扰率误差在2%~13%，可应用于作业调度和负载均衡等场景。

关键词: 分布式机器学习, 性能干扰, 集群调度, 资源共享, 干扰预测

CLC Number:

TP338.8

Hongliang LI, Nong ZHANG, Ting SUN, Xiang LI. Performance interference analysis and prediction for distributed machine learning jobs[J]. Journal of Computer Applications, 2022, 42(6): 1649-1655.

李洪亮, 张弄, 孙婷, 李想. 分布式机器学习作业性能干扰分析与预测[J]. 《计算机应用》唯一官方网站, 2022, 42(6): 1649-1655.

Figures/Tables 10

References 22

1	JORDAN M I， MITCHELL T M. Machine learning： trends， perspectives， and prospects［J］. Science， 2015， 349（6245）： 255-260. 10.1126/science.aaa8415
2	YU J H， HUANG T. Universally slimmable networks and improved training techniques［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 1803-1811. 10.1109/iccv.2019.00189
3	DENG J， DONG W， SOCHER R， et al. ImageNet： a large-scale hierarchical image database［C］// Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2009： 248-255. 10.1109/cvpr.2009.5206848
4	李青坪. 面向分布式深度学习的集群资源调度优化技术研究［D］. 南京：南京大学， 2020：1-3. 10.29026/oea.2022.210021
	LI Q P. Research on cluster resource scheduling optimization for distributed deep learning［D］. Nanjing： Nanjing University， 2020： 1-3. 10.29026/oea.2022.210021
5	PETEIRO-BARRAL D， GUIJARRO-BERDIÑAS B. A survey of methods for distributed machine learning［J］. Progress in Artificial Intelligence， 2013， 2（1）： 1-11. 10.1007/s13748-012-0035-5
6	LI M， ANDERSEN D G， PARK J W， et al. Scaling distributed machine learning with the parameter server［C］// Proceedings of the 11th USENIX Symposium on Operating Systems Design and Implementation. Berkeley： USENIX Association， 2014： 583-598. 10.1145/2640087.2644155
7	王丹. 分布式机器学习集群的资源调度机制研究［D］. 成都：电子科技大学， 2020：6-15.
	WANG D. Research on resource scheduling mechanism of distributed machine learning cluster［D］. Chengdu： University of Electronic Science and Technology of China， 2020： 6-15.
8	WEN L， LI X Y， GAO L. A transfer convolutional neural network for fault diagnosis based on ResNet-50［J］. Neural Computing and Applications， 2020， 32（10）： 6111-6124. 10.1007/s00521-019-04097-w
9	ALIPPI C， DISABATO S， ROVERI M. Moving convolutional neural networks to embedded systems： the AlexNet and VGG-16 case［C］// Proceedings of the 17th ACM/IEEE International Conference on Information Processing in Sensor Networks. Piscataway： IEEE， 2018： 212-223. 10.1109/ipsn.2018.00049
10	ABADI M， BARHAM P， CHEN J M， et al. TensorFlow： a system for large-scale machine learning［C］// Proceedings of the 12th USENIX Symposium on Operating Systems Design and Implementation. Berkeley： USENIX Association， 2016： 265-283.
11	PASZKE A， GROSS S， MASSA F， et al. PyTorch： an imperative style， high-performance deep learning library［C/OL］// Proceedings of the 33rd Conference on Neural Information Processing Systems. ［2021-05-11］.. 10.48550/arXiv.1912.01703
12	GU J C， CHOWDHURY M， SHIN K G， et al. Tiresias： a GPU cluster manager for distributed deep learning［C］// Proceedings of the 16th USENIX Symposium on Networked Systems Design and Implementation. Berkeley： USENIX Association， 2019： 485-500.
13	PENG Y H， BAO Y X， CHEN Y R， et al. Optimus： an efficient dynamic resource scheduler for deep learning clusters［C］// Proceedings of the 13th European Conference on Computer Systems. New York： ACM， 2018： No.3. 10.1145/3190508.3190517
14	MAHAJAN K， BALASUBRAMANIAN A， SINGHVI A， et al. THEMIS： fair and efficient GPU cluster scheduling［C］// Proceedings of the 17th USENIX Symposium on Networked Systems Design and Implementation. Berkeley： USENIX Association， 2020： 289-304.
15	LI H L， SUN T， LI X， et al. Job placement strategy with opportunistic resource sharing for distributed deep learning clusters［C］// Proceedings of the IEEE 22nd International Conference on High Performance Computing and Communications/ IEEE 18th International Conference on Smart City/ IEEE 6th International Conference on Data Science and Systems. Piscataway： IEEE， 2020： 620-627. 10.1109/hpcc-smartcity-dss50907.2020.00079
16	TRANMER M， ELLIOT M. Multiple linear regression［R/OL］. ［2021-05-11］.. 10.1007/978-1-4020-5614-7_2254
17	WAUTERS M， VANHOUCKE M. Support vector machine regression for project control forecasting［J］. Automation in Construction， 2014， 47： 92-106. 10.1016/j.autcon.2014.07.014
18	LOH W Y. Classification and regression tree methods［M］// RUGGERI F， KENETT R S， FALTIN F W. Encyclopedia of Statistics in Quality and Reliability. Hoboken， NJ： John Wiley & Sons， Inc.， 2008： 315-323.
19	HAMERLY G， ELKAN C. Learning the k in k-means［C］// Proceedings of the 16th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2003： 281-288.
20	JACKSON D. Employability skill development in work-integrated learning： barriers and best practice［J］. Studies in Higher Education， 2015， 40（2）： 350-367. 10.1080/03075079.2013.842221
21	LeCUN Y， BENGIO Y， HINTON G. Deep learning［J］. Nature， 2015， 521（7553）： 436-444. 10.1038/nature14539
22	GREFF K， SRIVASTAVA R K， KOUTNÍK J， et al. LSTM： a search space odyssey［J］. IEEE Transactions on Neural Networks and Learning Systems， 2017， 28（10）： 2222-2232. 10.1109/tnnls.2016.2582924

预测技术	调用函数和参数
多元线性回归^［16］	linear_model.LinearRegression
支持向量回归^［17］	SVR（kernel="linear"，"poly"）
回归树^［18］	DecisionTreeRegressor
K-最近邻^［19］	KNeighborsRegressor
集成学习^［20］	GradientBoostingRegressor RandomForestRegressor

预测技术	调用函数和参数
多元线性回归^［16］	linear_model.LinearRegression
支持向量回归^［17］	SVR（kernel="linear"，"poly"）
回归树^［18］	DecisionTreeRegressor
K-最近邻^［19］	KNeighborsRegressor
集成学习^［20］	GradientBoostingRegressor RandomForestRegressor

GPU参数		具体描述
类别	符号	具体描述
1	E	能耗（单位：W）
1	T	温度（单位：℃）
2	S	流处理器使用率（单位：%）
2	V	显存利用率（单位：%）
3	R	读带宽（单位：Mb/s）
3	W	写带宽（单位：Mb/s）

GPU参数		具体描述
类别	符号	具体描述
1	E	能耗（单位：W）
1	T	温度（单位：℃）
2	S	流处理器使用率（单位：%）
2	V	显存利用率（单位：%）
3	R	读带宽（单位：Mb/s）
3	W	写带宽（单位：Mb/s）

作业组合	TeslaP40（卡1）环境下数据集							作业组合	RTX3090（卡2）环境下数据集
	不同GPU参数变化百分比/%						干扰率/%		不同GPU参数变化百分比/%						干扰率/%
	E	T	S	V	R	W	干扰率/%		E	T	S	V	R	W	干扰率/%
1₄	82.8	35.0	-42.3	-20.60	36.7	161.0	113.2	2₁	58.6	13.9	23.40	0.74	192.2	212.8	212.8
2₄	211.0	44.4	-18.1	0.27	164.4	556.0	90.2	2₄	55.7	12.3	24.65	-3.48	126.8	131.0	131.0
5₄	21.4	7.7	-67.0	-41.50	185.2	218.7	116.3	5₃	0.6	2.7	1.54	-7.92	85.8	96.2	96.2
3₅	146.2	62.2	0.2	7.50	393.0	468.0	40.1	2₄₅	18.6	409.0	21.90	6.55	229.2	318.0	318.0

Performance interference analysis and prediction for distributed machine learning jobs

分布式机器学习作业性能干扰分析与预测

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 22

Related Articles 13

Recommended Articles

Metrics

模型	TeslaP40				RTX3090
模型	类别 1	类别 2	类别 3	全部类别	类别 1	类别 2	类别 3	全部类别
多元线性回归	0.44	0.16	0.31	0.19	0.06	0.06	0.09	0.14
SVR	0.45	0.13	0.28	0.20	0.09	0.06	0.03	0.05
回归树	0.62	0.45	0.29	0.29	0.07	0.11	0.02	0.06
KNN	0.57	0.25	0.37	0.27	0.06	0.07	0.04	0.04
集成学习	0.61	0.21	0.35	0.37	0.06	0.09	0.03	0.06

[1]	Chunyong YIN, Yongcheng ZHOU. Automatically adjusted clustered federated learning for double-ended clustering [J]. Journal of Computer Applications, 2024, 44(10): 3011-3020.
[2]	Mengjie LAN, Jianping CAI, Lan SUN. Self-regularization optimization methods for Non-IID data in federated learning [J]. Journal of Computer Applications, 2023, 43(7): 2073-2081.
[3]	CAI Yueping, ZHANG Wenpeng, LUO Sen. Minimum access guaranteed bandwidth allocation mechanism in data center network [J]. Journal of Computer Applications, 2017, 37(7): 1825-1829.
[4]	ZHAN Jinzhen, GUO Dawei, HUA Weixin. Device to device time division scheduling algorithm based on fairness [J]. Journal of Computer Applications, 2017, 37(3): 711-716.
[5]	SUN Lihua, CHEN Shipin. Cloud resource sharing design supporting multi-attribute range query based on SkipNet [J]. Journal of Computer Applications, 2016, 36(1): 72-76.
[6]	DU Xiongjie, WANG Min, TANG Xuehai, ZHANG Zhang. Scheduling mechanism based on service level objective in multi-tenant cluster [J]. Journal of Computer Applications, 2015, 35(4): 944-949.
[7]	Lǚ Xin GAO Feng. Privacy protection method in E-government information resource sharing [J]. Journal of Computer Applications, 2012, 32(01): 82-85.
[8]	XIE Yan-hong. General computing resource sharing environments based on .NET over the Internet [J]. Journal of Computer Applications, 2011, 31(09): 2563-2566.
[9]	ZHANG Xue-feng XU Sheng-chao. RB-CRSP: Node role-based computing resources sharing platform over Internet [J]. Journal of Computer Applications, 2011, 31(03): 834-838.
[10]	hua yanjiang. Decentralized approach for metadata management in computing resource sharing platform [J]. Journal of Computer Applications, 2011, 31(02): 462-465.
[11]	Zhang JianYin. Credit mechanism oriented resource sharing platform in structured P2P networks [J]. Journal of Computer Applications, 2010, 30(10): 2641-2644.
[12]	Jia-xin WU Jian PENG Da-ping ZHANG. Research and implementation of JXTA P2P protocol on Symbian platform [J]. Journal of Computer Applications, 2009, 29(4): 1082-1086.
[13]	. Research on seamless resource discovery architecture based on ASP pattern of manufacturing grid [J]. Journal of Computer Applications, 2009, 29(07): 1966-1969.

GPU参数类别	TeslaP40	RTX3090
1	27.00	33.00
2	38.00	33.00
3	35.00	34.00

GPU参数类别	TeslaP40	RTX3090
1	27.00	33.00
2	38.00	33.00
3	35.00	34.00

显卡名称	最大能耗/W	显存/MB	带宽/（Gb·s^-1）
TeslaP40	250	22 919	346
RTX3090	350	24 628	936

显卡名称	最大能耗/W	显存/MB	带宽/（Gb·s^-1）
TeslaP40	250	22 919	346
RTX3090	350	24 628	936