Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (6): 1884-1892.DOI: 10.11772/j.issn.1001-9081.2022050722
Special Issue: 先进计算
• Advanced computing • Previous Articles Next Articles
Heping FANG1,2, Shuguang LIU1(), Yongyi RAN3, Kunhua ZHONG1
Received:
2022-05-20
Revised:
2022-08-04
Accepted:
2022-08-08
Online:
2023-06-08
Published:
2023-06-10
Contact:
Shuguang LIU
About author:
FANG Heping, born in 1997, M. S. candidate. His research interests include deep reinforcement learning, green data center.Supported by:
通讯作者:
刘曙光
作者简介:
方和平(1997—),男,重庆人,硕士研究生,CCF会员,主要研究方向:深度强化学习、绿色数据中心基金资助:
CLC Number:
Heping FANG, Shuguang LIU, Yongyi RAN, Kunhua ZHONG. Integrated scheduling optimization of multiple data centers based on deep reinforcement learning[J]. Journal of Computer Applications, 2023, 43(6): 1884-1892.
方和平, 刘曙光, 冉泳屹, 钟坤华. 基于深度强化学习的多数据中心一体化调度优化[J]. 《计算机应用》唯一官方网站, 2023, 43(6): 1884-1892.
Add to citation manager EndNote|Ris|BibTeX
URL: https://www.joca.cn/EN/10.11772/j.issn.1001-9081.2022050722
资源 | 费用 |
---|---|
每个CPU核(core) | |
每GB内存 | |
每GB磁盘 | |
每Mb通信带宽 |
Tab.1 Resource price examples
资源 | 费用 |
---|---|
每个CPU核(core) | |
每GB内存 | |
每GB磁盘 | |
每Mb通信带宽 |
参数 | 设定值 |
---|---|
训练情节数 | 80 000 |
折扣因子 | 0.99 |
内存容量 | 5 000 |
学习率 | 0.001 |
1 | |
0.001 | |
0.000 166 5 | |
批量尺寸 | 128 |
目标网络更新周期 | 70 |
隐藏层数 | 2 |
Tab.2 Multiple data center task scheduling model parameters setting
参数 | 设定值 |
---|---|
训练情节数 | 80 000 |
折扣因子 | 0.99 |
内存容量 | 5 000 |
学习率 | 0.001 |
1 | |
0.001 | |
0.000 166 5 | |
批量尺寸 | 128 |
目标网络更新周期 | 70 |
隐藏层数 | 2 |
数据中心 | 每个CPU核数 单位时间价格 | RAM/GB | Disk/GB | 带宽/Mb |
---|---|---|---|---|
1 | 0.116 4 | 1.581 | 0.058 | 0.334 |
2 | 0.112 2 | 1.585 | 0.061 | 0.456 |
3 | 0.106 3 | 1.565 | 0.069 | 0.385 |
4 | 0.110 2 | 1.567 | 0.063 | 0.489 |
5 | 0.119 0 | 1.605 | 0.060 | 0.399 |
6 | 0.101 0 | 1.705 | 0.072 | 0.367 |
7 | 0.125 0 | 1.495 | 0.052 | 0.347 |
8 | 0.105 0 | 1.475 | 0.035 | 0.397 |
9 | 0.111 0 | 1.464 | 0.051 | 0.343 |
10 | 0.122 0 | 1.450 | 0.045 | 0.385 |
11 | 0.117 0 | 1.563 | 0.055 | 0.356 |
Tab.3 Prices of data center resources in unit time
数据中心 | 每个CPU核数 单位时间价格 | RAM/GB | Disk/GB | 带宽/Mb |
---|---|---|---|---|
1 | 0.116 4 | 1.581 | 0.058 | 0.334 |
2 | 0.112 2 | 1.585 | 0.061 | 0.456 |
3 | 0.106 3 | 1.565 | 0.069 | 0.385 |
4 | 0.110 2 | 1.567 | 0.063 | 0.489 |
5 | 0.119 0 | 1.605 | 0.060 | 0.399 |
6 | 0.101 0 | 1.705 | 0.072 | 0.367 |
7 | 0.125 0 | 1.495 | 0.052 | 0.347 |
8 | 0.105 0 | 1.475 | 0.035 | 0.397 |
9 | 0.111 0 | 1.464 | 0.051 | 0.343 |
10 | 0.122 0 | 1.450 | 0.045 | 0.385 |
11 | 0.117 0 | 1.563 | 0.055 | 0.356 |
参数 | 设定值 |
---|---|
训练情节数 | 9 000 |
折扣因子 | 0.99 |
内存容量 | 5 000 |
学习率 | 0.001 |
0.2 | |
0.001 | |
0.000 019 9 | |
批量尺寸 | 128 |
目标网络更新周期 | 30 |
隐藏层数 | 3 |
Tab.4 Data center internal task deployment model parameters settings
参数 | 设定值 |
---|---|
训练情节数 | 9 000 |
折扣因子 | 0.99 |
内存容量 | 5 000 |
学习率 | 0.001 |
0.2 | |
0.001 | |
0.000 019 9 | |
批量尺寸 | 128 |
目标网络更新周期 | 30 |
隐藏层数 | 3 |
算法 | 平均奖励 | 平均PUE | 平均过载数 |
---|---|---|---|
RR | 15.03±2.84 | 2.36±0.015 | 57±16.63 |
Greedy | 15.95±2.82 | 2.34±2.360 | 38±20.88 |
Double DQN | 17.45±2.77 | 2.30±0.017 | 33±16.63 |
Tab.5 Experimental results data
算法 | 平均奖励 | 平均PUE | 平均过载数 |
---|---|---|---|
RR | 15.03±2.84 | 2.36±0.015 | 57±16.63 |
Greedy | 15.95±2.82 | 2.34±2.360 | 38±20.88 |
Double DQN | 17.45±2.77 | 2.30±0.017 | 33±16.63 |
1 | LI Q R, PENG Z P, CUI D L, et al. Two-stage selection of distributed data centers based on deep reinforcement learning [J]. Cluster Computing, 2022, 25(4): 2699-2714. 10.1007/s10586-021-03525-8 |
2 | JIN Y B, GAO Y, QIAN Z Z, et al. Workload-aware scheduling across geo-distributed data centers[C]// Proceedings of the 15th IEEE International Conference on Trust, Security and Privacy in Computing and Communications/ 10th IEEE International Conference on Big Data Science and Engineering/ 14th IEEE International Symposium on Parallel and Distributed Processing with Applications. Piscataway: IEEE, 2016: 1455-1462. 10.1109/trustcom.2016.0228 |
3 | CAO Z W, ZHOU X, HU H, et al. Towards a systematic survey for carbon neutral data centers [J]. IEEE Communications Surveys and Tutorials, 2022, 24(2): 895-936. 10.1109/comst.2022.3161275 |
4 | PAKBAZNIA E, GHASEMAZAR M, PEDRAM M. Temperature-aware dynamic resource provisioning in a power-optimized datacenter [C]// Proceedings of the 2010 Design, Automation and Test in Europe Conference and Exhibition. Piscataway: IEEE, 2010: 124-129. 10.1109/date.2010.5457223 |
5 | XU J L, PALANISAMY B. Cost-aware resource management for federated clouds using resource sharing contracts [C]// Proceedings of the IEEE 10th International Conference on Cloud Computing. Piscataway: IEEE, 2017: 238-245. 10.1109/cloud.2017.38 |
6 | SHEN H Y, CHEN L H. A resource usage intensity aware load balancing method for virtual machine migration in cloud datacenters[J]. IEEE Transactions on Cloud Computing, 2020, 8(1): 17-31. 10.1109/tcc.2017.2737628 |
7 | GE Y J, WEI G Y. GA-based task scheduler for the cloud computing systems[C]// Proceedings of the 2010 International Conference on Web Information Systems and Mining. Piscataway: IEEE, 2010: 181-186. 10.1109/wism.2010.87 |
8 | ETTIKYALA K, LATHA Y V. Rank based efficient task scheduler for cloud computing[C]// Proceedings of the 2016 International Conference on Data Mining and Advanced Computing. Piscataway: IEEE, 2016: 343-346. 10.1109/sapience.2016.7684151 |
9 | YI D L, ZHOU X, WEN Y G, et al. Toward efficient compute-intensive job allocation for green data centers: a deep reinforcement learning approach [C]// Proceedings of the IEEE 39th International Conference on Distributed Computing Systems. Piscataway: IEEE, 2019: 634-644. 10.1109/icdcs.2019.00069 |
10 | RAN Y Y, HU H, ZHOU X, et al. DeepEE: joint optimization of job scheduling and cooling control for data center energy efficiency using deep reinforcement learning [C]// Proceedings of the IEEE 39th International Conference on Distributed Computing Systems. Piscataway: IEEE, 2019: 645-655. 10.1109/icdcs.2019.00070 |
11 | CHENG M X, LI J, BOGDAN P, et al. H₂O-Cloud: a resource and quality of service-aware task scheduling framework for warehouse-scale data centers[J]. IEEE Transactions on Computer-Aided Design of Integrated Circuits and Systems, 2020, 39(10): 2925-2937. 10.1109/tcad.2019.2930575 |
12 | 祝家钰,肖丹,王飞. 云计算下负载均衡的多维QoS约束任务调度机制[J]. 计算机工程与应用, 2013, 49(9):85-89. 10.3778/j.issn.1002-8331.1110-0676 |
ZHU J Y, XIAO D, WANG F. Multi-dimensional QoS constrained scheduling mechanism based on load balancing for cloud computing[J]. Computer Engineering and Applications, 2013, 49(9):85-89. 10.3778/j.issn.1002-8331.1110-0676 | |
13 | 左利云,曹志波. 云计算中调度问题研究综述[J]. 计算机应用研究, 2012, 29(11):4023-4027. 10.3969/j.issn.1001-3695.2012.11.005 |
ZUO L Y, CAO Z B. Review of scheduling research in cloud computing[J]. Application Research of Computers, 2012, 29(11): 4023-4027. 10.3969/j.issn.1001-3695.2012.11.005 | |
14 | ZHAO J, LI H X, WU C, et al. Dynamic pricing and profit maximization for the cloud with geo-distributed data centers [C]// Proceedings of the 2014 IEEE Conference on Computer Communications. Piscataway: IEEE, 2014: 118-126. 10.1109/infocom.2014.6847931 |
15 | ALZHOURI F, AGARWAL A, LIU Y. Maximizing cloud revenue using dynamic pricing of multiple class virtual machines [J]. IEEE Transactions on Cloud Computing, 2021, 9(2): 682-695. 10.1109/tcc.2018.2878023 |
16 | RAN Y Y, YANG J, ZHANG S B, et al. Dynamic IaaS computing resource provisioning strategy with QoS constraint[J]. IEEE Transactions on Services Computing, 2017, 10(2): 190-202. 10.1109/tsc.2015.2464212 |
17 | GU L, ZENG D Z, GUO S, et al. A general communication cost optimization framework for big data stream processing in geo-distributed data centers[J]. IEEE Transactions on Computers, 2016, 65(1): 19-29. 10.1109/tc.2015.2417566 |
18 | HILDA J, CHANDRASEKARAN S. Cost and time economical planning algorithm for scientific workflows in cloud computing [J]. Future Internet, 2021, 13(10): No.263. 10.3390/fi13100263 |
19 | WANG W, YAN Y, ZHANG L M, et al. Collaborative sparse coding for multiview action recognition[J]. IEEE MultiMedia, 2016, 23(4): 80-87. 10.1109/mmul.2016.69 |
20 | MNIH V, KAVUKCUOGLU K, SILVER D, et al. Playing Atari with deep reinforcement learning [EB/OL]. (2013-12-19) [2022-04-20].. 10.1038/nature14236 |
21 | van HASSELT H, GUEZ A, SILVER D. Deep reinforcement learning with double Q-learning[C]// Proceedings of the 30th AAAI Conference on Artificial Intelligence. Palo Alto, CA: AAAI Press, 2016: 2094-2100. 10.1609/aaai.v30i1.10295 |
22 | SCHAUL T, QUAN J, ANTONOGLOU I, et al. Prioritized experience replay [EB/OL]. (2016-02-25) [2022-04-20].. |
23 | GÓMEZ-MARTÍN C, VEGA-RODRÍGUEZ M A, GONZÁLEZ-SÁNCHEZ J L. Performance and energy aware scheduling simulator for HPC: evaluating different resource selection methods[J]. Concurrency and Computation: Practice and Experience, 2015, 27(17): 5436-5459. 10.1002/cpe.3607 |
24 | LI Q R, PENG Z P, CUI D L, et al. Data center selection based on reinforcement learning [C]// Proceedings of the 4th International Conference on Cloud Computing and Internet of Things. Piscataway: IEEE, 2019: 14-19. 10.1109/cciot48581.2019.8980333 |
25 | The Hebrew University — Experimental Systems Lab. Logs of real parallel workloads from production systems [EB/OL]. (2019-09-24) [2022-05-09].. |
26 | PATERIA S, SUBAGDJA B, TAN A H, et al. Hierarchical reinforcement learning: a comprehensive survey[J]. ACM Computing Surveys, 2021, 54(5): No.109. 10.1145/3453160 |
27 | BACON P L, HARB J, PRECUP D. The option-critic architecture[C]// Proceedings of the 31st AAAI Conference on Artificial Intelligence. Palo Alto, CA: AAAI Press, 2017: 1726-1734. 10.1609/aaai.v31i1.10916 |
28 | KULKARNI T D, NARASIMHAN K R, SAEEDI A, et al. Hierarchical deep reinforcement learning: integrating temporal abstraction and intrinsic motivation[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2016: 3682-3690. 10.48550/arXiv.1604.06057 |
[1] | Yi ZHOU, Hua GAO, Yongshen TIAN. Proximal policy optimization algorithm based on clipping optimization and policy guidance [J]. Journal of Computer Applications, 2024, 44(8): 2334-2341. |
[2] | Tian MA, Runtao XI, Jiahao LYU, Yijie ZENG, Jiayi YANG, Jiehui ZHANG. Mobile robot 3D space path planning method based on deep reinforcement learning [J]. Journal of Computer Applications, 2024, 44(7): 2055-2064. |
[3] | Xiaoyan ZHAO, Wei HAN, Junna ZHANG, Peiyan YUAN. Collaborative offloading strategy in internet of vehicles based on asynchronous deep reinforcement learning [J]. Journal of Computer Applications, 2024, 44(5): 1501-1510. |
[4] | Rui TANG, Chuanlin PANG, Ruizhi ZHANG, Chuan LIU, Shibo YUE. DDPG-based resource allocation in D2D communication-empowered cellular network [J]. Journal of Computer Applications, 2024, 44(5): 1562-1569. |
[5] | Xintong QIN, Zhengyu SONG, Tianwei HOU, Feiyue WANG, Xin SUN, Wei LI. Channel access and resource allocation algorithm for adaptive p-persistent mobile ad hoc network [J]. Journal of Computer Applications, 2024, 44(3): 863-868. |
[6] | Fuqin DENG, Huifeng GUAN, Chaoen TAN, Lanhui FU, Hongmin WANG, Tinlun LAM, Jianmin ZHANG. Multi-robot reinforcement learning path planning method based on request-response communication mechanism and local attention mechanism [J]. Journal of Computer Applications, 2024, 44(2): 432-438. |
[7] | Yuanchao LI, Chongben TAO, Chen WANG. Gait control method based on maximum entropy deep reinforcement learning for biped robot [J]. Journal of Computer Applications, 2024, 44(2): 445-451. |
[8] | Jiachen YU, Ye YANG. Irregular object grasping by soft robotic arm based on clipped proximal policy optimization algorithm [J]. Journal of Computer Applications, 2024, 44(11): 3629-3638. |
[9] | Jie LONG, Liang XIE, Haijiao XU. Integrated deep reinforcement learning portfolio model [J]. Journal of Computer Applications, 2024, 44(1): 300-310. |
[10] | Shaofa SHANG, Lin JIANG, Yuancheng LI, Yun ZHU. Adaptive partitioning and scheduling method of convolutional neural network inference model on heterogeneous platforms [J]. Journal of Computer Applications, 2023, 43(9): 2828-2835. |
[11] | Yu WANG, Tianjun REN, Zilin FAN. Air combat maneuver decision-making of unmanned aerial vehicle based on guided Minimax-DDQN [J]. Journal of Computer Applications, 2023, 43(8): 2636-2643. |
[12] | Ziteng WANG, Yaxin YU, Zifang XIA, Jiaqi QIAO. Sparse reward exploration mechanism fusing curiosity and policy distillation [J]. Journal of Computer Applications, 2023, 43(7): 2082-2090. |
[13] | Xiaolin LI, Yusang JIANG. Task offloading algorithm for UAV-assisted mobile edge computing [J]. Journal of Computer Applications, 2023, 43(6): 1893-1899. |
[14] | Xiaohui HUANG, Kaiming YANG, Jiahao LING. Order dispatching by multi-agent reinforcement learning based on shared attention [J]. Journal of Computer Applications, 2023, 43(5): 1620-1624. |
[15] | Tengfei CAO, Yanliang LIU, Xiaoying WANG. Edge computing and service offloading algorithm based on improved deep reinforcement learning [J]. Journal of Computer Applications, 2023, 43(5): 1543-1550. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||