Automatic tuning of Ceph parameters based on random forest and genetic algorithm

doi:10.11772/j.issn.1001-9081.2019081366

Abstract

Abstract:

The performance of Ceph system is significantly affected by the configuration parameters. In the optimization of configuration of Ceph cluster， there are many kinds of configuration parameters with complex meanings， which makes it difficult to achieve fast and accurate optimization. To solve the above problems， a parameter tuning method based on Random Forest （RF） and Genetic Algorithm （GA） was proposed to automatically adjust the Ceph parameter configuration in order to optimize the Ceph system performance. RF algorithm was used to construct a performance prediction model for the Ceph system， and the output of the prediction model was used as the input of GA， then the parameter configuration scheme was automatically and iteratively optimized by using GA. Simulation results show that compared with the system with default parameter configuration， the Ceph file system with optimized parameter configuration has the read and write performance improved by about 1.4 times， and the optimization time is much lower than that of the black box parameter tuning method.

Key words: Ceph, parameter configuration, Random Forest (RF), Genetic Algorithm (GA), automatic tuning

摘要：

Ceph系统性能受Ceph配置参数的显著影响，在Ceph集群的配置优化中，配置参数种类繁多、含义复杂，导致难以实现快速准确寻优。针对以上问题，提出一种基于随机森林（RF）和遗传算法（GA）的参数调优方法，用于自动调整Ceph参数配置以优化Ceph系统性能。该方法使用RF算法为Ceph系统构建性能预测模型，并将预测模型的输出作为GA的输入，通过GA对参数配置方案进行自动迭代优化。仿真结果表明，调优后的参数配置较默认的参数配置相比，使Ceph文件系统的读写性能提高了约1.4倍，并且寻优耗时远低于黑盒参数调优方法。

关键词: Ceph, 参数配置, 随机森林, 遗传算法, 自动调优

CLC Number:

TP393.06

Yu CHEN, Yingchi MAO. Automatic tuning of Ceph parameters based on random forest and genetic algorithm[J]. Journal of Computer Applications, 2020, 40(2): 347-351.

陈禹, 毛莺池. 基于随机森林和遗传算法的Ceph参数自动调优[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 347-351.

Figures/Tables 9

Tab. 1 Description of Ceph configuration parameters

参数名	数据类型	默认值	范围
filestore max sync interval	integer	5	5~16
filestore min sync interval	float	0.01	0.01~5
filestore op threads	integer	2	1~32
filestore queue max ops	integer	50	16~25 000
osd max write size	integer	90	30~2 000
osd op threads	integer	2	1~32
osd map cache size	integer	200	64~1 024

Fig. 1 Performance changes with different parameter configurations

Fig. 2 Overall architecture of the proposed model

Fig. 3 Framework of parameter optimization

Fig. 4 Influence of m and k values on model accuracy

Fig. 5 Random forest prediction effect map

Fig. 6 Performance comparison of several models

Fig. 7 Relationship between read and write performance and GA iteration step

Fig. 8 Comparison of optimization efficiency under different file sizes

References 15

1	HUANG M， LUO L， LI Y， et al. Research on data migration optimization of Ceph［C］// Proceedings of the 14th International Computer Conference on Wavelet Active Media Technology and Information Processing. Piscataway： IEEE， 2017： 83-88. 10.1109/iccwamtip.2017.8301454
2	李翔. Ceph分布式文件系统的研究及性能测试［D］. 西安：西安电子科技大学， 2014：15-20.
	LI X. Research and performance testing of the Ceph distributed file system［D］. Xi’an： Xidian University， 2014：15-20.
3	ZHANG X， GADDAM S， CHRONOPOULOS A T. Ceph distributed file system benchmarks on an Openstack cloud［C］// Proceedings of the 2015 IEEE International Conference on Cloud Computing in Emerging Markets. Piscataway： IEEE， 2015： 113-120. 10.1109/ccem.2015.12
4	CAO Z， TARASOV V， TIWARI S. Towards better understanding of black-box auto-tuning： a comparative analysis for storage systems［C］// Proceedings of the 2018 Annual USENIX Technical Conference. Berkeley： USENIX Association， 2018： 893-907.
5	YIGITBASI N， WILLKE T L， LIAO G， et al. Towards machine learning-based auto-tuning of MapReduce［C］// Proceedings of the IEEE 21st International Symposium on Modeling， Analysis and Simulation of Computer and Telecommunication Systems. Piscataway： IEEE， 2013：11-20. 10.1109/mascots.2013.9
6	曾林西. 基于性能预估的Hadoop参数自动调优系统［D］. 武汉：华中科技大学， 2013： 34-37.
	ZENG X L. A cost-based optimizer for configuration parameters of Hadoop candidate［D］. Wuhan： Huazhong University of Science and Technology， 2013： 34-37.
7	CAI L， QI Y， LI J. A Recommendation-based parameter tuning approach for Hadoop［C］// Proceedings of the 2017 IEEE International Symposium on Cloud and Service Computing. Piscataway： IEEE， 2017：223-230. 10.1109/sc2.2017.41
8	马跃，余骋远，于碧辉. 基于资源签名与遗传算法的Hadoop参数自动调优系统［J］. 计算机应用研究， 2017， 34（11）：3219-3222， 3228. 10.3969/j.issn.1001-3695.2017.11.004
	MA Y， YU C Y， YU B H. Hadoop parameter automatic tuning system based on resource signature and genetic algorithm［J］. Application Research of Computers， 2017， 34（11）： 3219-3222， 3228. 10.3969/j.issn.1001-3695.2017.11.004
9	WU D， WANG Y， FENG H， et al. Optimization design and realization of Ceph storage system based on software defined network［C］// Proceedings of the 13th International Conference on Computational Intelligence and Security. Piscataway： IEEE， 2017：277-281. 10.1109/cis.2017.00067
10	王皎，刘闫锋. Hadoop集群参数的自动调优［J］. 电脑知识与技术， 2012， 8（12）：2768-2772. 10.3969/j.issn.1009-3044.2012.12.036
	WANG J， LIU Y F. Parameter auto-tuning of Hadoop clusters［J］. Computer Knowledge and Technology， 2012， 8（12）： 2768-2772. 10.3969/j.issn.1009-3044.2012.12.036
11	刘辉勇，王勇，俸皓. Ceph云存储中基于混合文件系统的读写性能优化方法［J］. 微电子学与计算机， 2018， 35（5）：27-34.
	LIU H Y， WANG Y， FENG H. Optimization of Ceph reads and writes performance based on hybrid file system［J］. Microelectronics and Computer， 2018， 35（5）： 27-34.
12	BEI Z， YU Z， ZHANG H， et al. RFHOC： a random-forest approach to auto-tuning Hadoop’s configuration［J］. IEEE Transactions on Parallel and Distributed Systems， 2016， 27（5）： 1470-1483. 10.1109/tpds.2015.2449299
13	YU Z， BEI Z， QIAN X. Datasize-aware high dimensional configurations auto-tuning of in-memory cluster computing［C］// Proceedings of the 23rd International Conference on Architectural Support for Programming Languages and Operating Systems. New York： ACM， 2018：564-577. 10.1145/3173162.3173187
14	YILDIRIM G， HALLAC İ R， AYDIM G， et al. Running genetic algorithms on Hadoop for solving high dimensional optimization problems［C］// Proceedings of the IEEE 9th International Conference on Application of Information and Communication Technologies. Piscataway： IEEE， 2015： 12-16. 10.1109/icaict.2015.7338506
15	BEI Z， YU Z， LUO N， et al. Configuring in-memory cluster computing using random forest［J］. Future Generation Computer Systems， 2018， 79（1）：1-15. 10.1016/j.future.2017.08.011

[1]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[2]	Hao CHAO, Shuqi FENG, Yongli LIU. Convolutional recurrent neural network optimized by multiple context vectors in EEG-based emotion recognition [J]. Journal of Computer Applications, 2024, 44(7): 2041-2046.
[3]	Wangjun SHI, Jing WANG, Xiaojun NING, Youfang LIN. Sleep stage classification model by meta transfer learning in few-shot scenarios [J]. Journal of Computer Applications, 2024, 44(5): 1445-1451.
[4]	Lin GAO, Yu ZHOU, Tak Wu KWONG. Evolutionary bi-level adaptive local feature selection [J]. Journal of Computer Applications, 2024, 44(5): 1408-1414.
[5]	Xuanfeng LI, Shengcai LIU, Ke TANG. Novel genetic algorithm for solving chance-constrained multiple-choice Knapsack problems [J]. Journal of Computer Applications, 2024, 44(5): 1378-1385.
[6]	Yudan SONG, Jing WANG, Xuehui WANG, Zhaoyang MA, Youfang LIN. Sleep physiological time series classification method based on adaptive multi-task learning [J]. Journal of Computer Applications, 2024, 44(2): 654-662.
[7]	Xuan CAO, Tianjian LUO. Dynamic multi-domain adversarial learning method for cross-subject motor imagery EEG signals [J]. Journal of Computer Applications, 2024, 44(2): 645-653.
[8]	Tian CHEN, Conghu CAI, Xiaohui YUAN, Beibei LUO. Multimodal emotion recognition method based on multiscale convolution and self-attention feature fusion [J]. Journal of Computer Applications, 2024, 44(2): 369-376.
[9]	Siqi YANG, Tianjian LUO, Xuanhui YAN, Guangju YANG. Transfer kernel learning method based on spatial features for motor imagery EEG [J]. Journal of Computer Applications, 2024, 44(11): 3354-3363.
[10]	Ran ZHAI, Xuebin CHEN, Guopeng ZHANG, Langtao PEI, Zheng MA. Improved K-anonymity privacy protection algorithm based on different sensitivities [J]. Journal of Computer Applications, 2023, 43(5): 1497-1503.
[11]	Lubao LI, Tian CHEN, Fuji REN, Beibei LUO. Bimodal emotion recognition method based on graph neural network and attention [J]. Journal of Computer Applications, 2023, 43(3): 700-705.
[12]	Shuying YANG, Haiming GUO, Xin LI. EEG classification based on channel selection and multi-dimensional feature fusion [J]. Journal of Computer Applications, 2023, 43(11): 3418-3427.
[13]	Junpeng ZHANG, Yujie SHI, Rui JANG, Jingjing DONG, Changjian QIU. Review on advances in recognition and classification of cognitive impairment based on EEG signals [J]. Journal of Computer Applications, 2023, 43(10): 3297-3308.
[14]	Yanfei LIU, Zheng PENG, Yihui WANG, Zhong WANG. PID parameter tuning of brushed direct-current motor based on improved genetic algorithm [J]. Journal of Computer Applications, 2022, 42(5): 1634-1641.
[15]	Xiaohan LI, Huading JIA, Xue CHENG, Taiyong LI. Stock market volatility prediction method based on improved genetic algorithm and graph neural network [J]. Journal of Computer Applications, 2022, 42(5): 1624-1633.