Automatic tuning of Ceph parameters based on random forest and genetic algorithm

doi:10.11772/j.issn.1001-9081.2019081366

Abstract

Abstract:

The performance of Ceph system is significantly affected by the configuration parameters. In the optimization of configuration of Ceph cluster， there are many kinds of configuration parameters with complex meanings， which makes it difficult to achieve fast and accurate optimization. To solve the above problems， a parameter tuning method based on Random Forest （RF） and Genetic Algorithm （GA） was proposed to automatically adjust the Ceph parameter configuration in order to optimize the Ceph system performance. RF algorithm was used to construct a performance prediction model for the Ceph system， and the output of the prediction model was used as the input of GA， then the parameter configuration scheme was automatically and iteratively optimized by using GA. Simulation results show that compared with the system with default parameter configuration， the Ceph file system with optimized parameter configuration has the read and write performance improved by about 1.4 times， and the optimization time is much lower than that of the black box parameter tuning method.

Key words: Ceph, parameter configuration, Random Forest (RF), Genetic Algorithm (GA), automatic tuning

摘要：

Ceph系统性能受Ceph配置参数的显著影响，在Ceph集群的配置优化中，配置参数种类繁多、含义复杂，导致难以实现快速准确寻优。针对以上问题，提出一种基于随机森林（RF）和遗传算法（GA）的参数调优方法，用于自动调整Ceph参数配置以优化Ceph系统性能。该方法使用RF算法为Ceph系统构建性能预测模型，并将预测模型的输出作为GA的输入，通过GA对参数配置方案进行自动迭代优化。仿真结果表明，调优后的参数配置较默认的参数配置相比，使Ceph文件系统的读写性能提高了约1.4倍，并且寻优耗时远低于黑盒参数调优方法。

关键词: Ceph, 参数配置, 随机森林, 遗传算法, 自动调优

CLC Number:

TP393.06

Yu CHEN, Yingchi MAO. Automatic tuning of Ceph parameters based on random forest and genetic algorithm[J]. Journal of Computer Applications, 2020, 40(2): 347-351.

陈禹, 毛莺池. 基于随机森林和遗传算法的Ceph参数自动调优[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 347-351.

Figures/Tables 9

Tab. 1 Description of Ceph configuration parameters

参数名	数据类型	默认值	范围
filestore max sync interval	integer	5	5~16
filestore min sync interval	float	0.01	0.01~5
filestore op threads	integer	2	1~32
filestore queue max ops	integer	50	16~25 000
osd max write size	integer	90	30~2 000
osd op threads	integer	2	1~32
osd map cache size	integer	200	64~1 024

Fig. 1 Performance changes with different parameter configurations

Fig. 2 Overall architecture of the proposed model

Fig. 3 Framework of parameter optimization

Fig. 4 Influence of m and k values on model accuracy

Fig. 5 Random forest prediction effect map

Fig. 6 Performance comparison of several models

Fig. 7 Relationship between read and write performance and GA iteration step

Fig. 8 Comparison of optimization efficiency under different file sizes

References 15

1	HUANG M， LUO L， LI Y， et al. Research on data migration optimization of Ceph［C］// Proceedings of the 14th International Computer Conference on Wavelet Active Media Technology and Information Processing. Piscataway： IEEE， 2017： 83-88. 10.1109/iccwamtip.2017.8301454
2	李翔. Ceph分布式文件系统的研究及性能测试［D］. 西安：西安电子科技大学， 2014：15-20.
	LI X. Research and performance testing of the Ceph distributed file system［D］. Xi’an： Xidian University， 2014：15-20.
3	ZHANG X， GADDAM S， CHRONOPOULOS A T. Ceph distributed file system benchmarks on an Openstack cloud［C］// Proceedings of the 2015 IEEE International Conference on Cloud Computing in Emerging Markets. Piscataway： IEEE， 2015： 113-120. 10.1109/ccem.2015.12
4	CAO Z， TARASOV V， TIWARI S. Towards better understanding of black-box auto-tuning： a comparative analysis for storage systems［C］// Proceedings of the 2018 Annual USENIX Technical Conference. Berkeley： USENIX Association， 2018： 893-907.
5	YIGITBASI N， WILLKE T L， LIAO G， et al. Towards machine learning-based auto-tuning of MapReduce［C］// Proceedings of the IEEE 21st International Symposium on Modeling， Analysis and Simulation of Computer and Telecommunication Systems. Piscataway： IEEE， 2013：11-20. 10.1109/mascots.2013.9
6	曾林西. 基于性能预估的Hadoop参数自动调优系统［D］. 武汉：华中科技大学， 2013： 34-37.
	ZENG X L. A cost-based optimizer for configuration parameters of Hadoop candidate［D］. Wuhan： Huazhong University of Science and Technology， 2013： 34-37.
7	CAI L， QI Y， LI J. A Recommendation-based parameter tuning approach for Hadoop［C］// Proceedings of the 2017 IEEE International Symposium on Cloud and Service Computing. Piscataway： IEEE， 2017：223-230. 10.1109/sc2.2017.41
8	马跃，余骋远，于碧辉. 基于资源签名与遗传算法的Hadoop参数自动调优系统［J］. 计算机应用研究， 2017， 34（11）：3219-3222， 3228. 10.3969/j.issn.1001-3695.2017.11.004
	MA Y， YU C Y， YU B H. Hadoop parameter automatic tuning system based on resource signature and genetic algorithm［J］. Application Research of Computers， 2017， 34（11）： 3219-3222， 3228. 10.3969/j.issn.1001-3695.2017.11.004
9	WU D， WANG Y， FENG H， et al. Optimization design and realization of Ceph storage system based on software defined network［C］// Proceedings of the 13th International Conference on Computational Intelligence and Security. Piscataway： IEEE， 2017：277-281. 10.1109/cis.2017.00067
10	王皎，刘闫锋. Hadoop集群参数的自动调优［J］. 电脑知识与技术， 2012， 8（12）：2768-2772. 10.3969/j.issn.1009-3044.2012.12.036
	WANG J， LIU Y F. Parameter auto-tuning of Hadoop clusters［J］. Computer Knowledge and Technology， 2012， 8（12）： 2768-2772. 10.3969/j.issn.1009-3044.2012.12.036
11	刘辉勇，王勇，俸皓. Ceph云存储中基于混合文件系统的读写性能优化方法［J］. 微电子学与计算机， 2018， 35（5）：27-34.
	LIU H Y， WANG Y， FENG H. Optimization of Ceph reads and writes performance based on hybrid file system［J］. Microelectronics and Computer， 2018， 35（5）： 27-34.
12	BEI Z， YU Z， ZHANG H， et al. RFHOC： a random-forest approach to auto-tuning Hadoop’s configuration［J］. IEEE Transactions on Parallel and Distributed Systems， 2016， 27（5）： 1470-1483. 10.1109/tpds.2015.2449299
13	YU Z， BEI Z， QIAN X. Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing［C］// Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems. New York： ACM， 2018：564-577. 10.1145/3173162.3173187
14	YILDIRIM G， HALLAC İ R， AYDIM G， et al. Running genetic algorithms on Hadoop for solving high dimensional optimization problems［C］// Proceedings of the IEEE 9th International Conference on Application of Information and Communication Technologies. Piscataway： IEEE， 2015： 12-16. 10.1109/icaict.2015.7338506
15	BEI Z， YU Z， LUO N， et al. Configuring in-memory cluster computing using random forest［J］. Future Generation Computer Systems， 2018， 79（1）：1-15. 10.1016/j.future.2017.08.011

[1]	ZHANG Wenqiang, XING Zheng, YANG Weidong. Hybrid particle swarm optimization with multi-region sampling strategy to solve multi-objective flexible job-shop scheduling problem [J]. Journal of Computer Applications, 2021, 41(8): 2249-2257.
[2]	ZHANG Meng, GUO Jianquan. Channel structure choice of closed-loop supply chain under uncertain demand and recovery [J]. Journal of Computer Applications, 2021, 41(7): 2100-2107.
[3]	LI Jin, WANG Feng, YANG Shenyu. Freight routing optimization model and algorithm of battery-swapping electric vehicle [J]. Journal of Computer Applications, 2021, 41(6): 1792-1798.
[4]	HUO Shoujun, HAO Yan, SHI Huiyu, DONG Yanqing, CAO Rui. Pattern recognition of motor imagery EEG based on deep convolutional network [J]. Journal of Computer Applications, 2021, 41(4): 1042-1048.
[5]	ZHOU Meiling, CHEN Huaili. Fuzzy multi-objective charging scheduling algorithm for electric vehicle based on load balance [J]. Journal of Computer Applications, 2021, 41(4): 1192-1198.
[6]	MA Xiaomei, HE Fei. Label printing production scheduling technology based on improved genetic algorithm [J]. Journal of Computer Applications, 2021, 41(3): 860-866.
[7]	WANG Binrong, TAN Dailun, ZHENG Bochuan. Solving auto part spraying sequence by transforming to traveling salesman problem and genetic algorithm [J]. Journal of Computer Applications, 2021, 41(3): 881-886.
[8]	HUANG Xiaoxiang, HU Yongmei, WU Dan, REN Lijie. Early identification and prediction of abnormal carotid arteries based on variational autoencoder [J]. Journal of Computer Applications, 2021, 41(10): 3082-3088.
[9]	ZHANG Zenghui, JIANG Gaoxia, WANG Wenjian. Label noise filtering method based on local probability sampling [J]. Journal of Computer Applications, 2021, 41(1): 67-73.
[10]	ZHOU Xiang, ZHAI Junhai, HUANG Yajie, SHEN Ruicai, HOU Yingzhen. Instance selection algorithm for big data based on random forest and voting mechanism [J]. Journal of Computer Applications, 2021, 41(1): 74-80.
[11]	CAO Yu, WANG Cheng, WANG Xin, GAO Yueer. Urban road short-term traffic flow prediction based on spatio-temporal node selection and deep learning [J]. Journal of Computer Applications, 2020, 40(5): 1488-1493.
[12]	HUO Qingqing, GUO Jianquan. Multi-objective closed-loop logistics network model of fresh foods based on improved genetic algorithm [J]. Journal of Computer Applications, 2020, 40(5): 1494-1500.
[13]	MAO Minli, LIANG Chengji, HU Xiaoyuan. Optimization of quay crane assignment based on ship efficiency [J]. Journal of Computer Applications, 2020, 40(4): 1223-1230.
[14]	WANG Yan, HE Hongke. Improved fuzzy c-means MRI segmentation based on neighborhood information [J]. Journal of Computer Applications, 2020, 40(4): 1196-1201.
[15]	ZHOU Leilei, LIANG Chengji, HU Xiaoyuan. Delivery truck strategy under uncertain interference constraints [J]. Journal of Computer Applications, 2020, 40(3): 891-896.