Hyperparameter optimization for neural network based on improved real coding genetic algorithm

doi:10.11772/j.issn.1001-9081.2023040441

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (3): 671-676.DOI: 10.11772/j.issn.1001-9081.2023040441

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Hyperparameter optimization for neural network based on improved real coding genetic algorithm

Wei SHE¹^,², Yang LI³^,⁵, Lihong ZHONG²^,⁴^,⁵, Defeng KONG⁶, Zhao TIAN¹^,³()

^1.School of Cyber Science and Engineering，Zhengzhou University，Zhengzhou Henan 450001，China
^2.Songshan Laboratory，Zhengzhou Henan 450000，China
^3.Henan Provincial Collaborative Innovation Center for Internet Medical and Health Services，Zhengzhou Henan 450052，China
^4.Zhengzhou Key Laboratory of Blockchain and Data Intelligence （Zhengzhou University），Zhengzhou Henan 450001，China
^5.School of Computer and Artificial Intelligence，Zhengzhou University，Zhengzhou Henan 450001，China
^6.Institute of Engineering Protection，National Defense Engineering Research Institute of the Academy of Military Sciences，Luoyang Henan 471023，China

Received:2023-04-18 Revised:2023-05-25 Accepted:2023-05-31 Online:2023-12-04 Published:2024-03-10
Contact: Zhao TIAN
About author:SHE Wei， born in 1977， Ph. D.， professor. His research interests include complex system modeling and simulation， machine learning， blockchain， data intelligence.
LI Yang， born in 1999， M. S. candidate. His research interests include swarm intelligence， machine learning.
ZHONG Lihong， born in 1997， Ph. D. candidate. Her research interests include machine learning， recommendation system， swarm intelligence.
KONG Defeng， born in 1989， engineer. His research interests include military operations research， damage effectiveness evaluation.
Supported by:
Songshan Laboratory Pre-research Project(YYYY022022003);Key Research and Development and Promotion Project in Henan Province(212102310039)

基于改进实数编码遗传算法的神经网络超参数优化

佘维¹^,², 李阳³^,⁵, 钟李红²^,⁴^,⁵, 孔德锋⁶, 田钊¹^,³()

^1.郑州大学网络空间安全学院, 郑州 450001
^2.嵩山实验室, 郑州 450000
^3.互联网医疗与健康服务河南省协同创新中心(郑州大学), 郑州 450052
^4.郑州市区块链与数据智能重点实验室(郑州大学), 郑州 450001
^5.郑州大学计算机与人工智能学院, 郑州 450001
^6.军事科学院国防工程研究院工程防护研究所, 河南洛阳 471023

通讯作者: 田钊
作者简介:佘维（1977—），男，湖南常德人，教授，博士生导师，博士，CCF会员，主要研究方向：复杂系统建模与仿真、机器学习、区块链、数据智能
李阳（1999—），男，河南开封人，硕士研究生，主要研究方向：群体智能、机器学习
钟李红（1997—），女，四川射洪人，博士研究生，主要研究方向：机器学习、推荐系统、群体智能
孔德锋（1989—），男，河南郸城人，工程师，主要研究方向：军事运筹学、毁伤效能评估
田钊（1985—），男，河南荥阳人，副教授，博士，主要研究方向：群体智能、机器学习、区块链、数据智能。
基金资助:
嵩山实验室预研项目(YYYY022022003);河南省重点研发与推广专项(212102310039)

Abstract

Abstract:

To address the problems of poor effects， easily falling into suboptimal solutions， and inefficiency in neural network hyperparameter optimization， an Improved Real Coding Genetic Algorithm （IRCGA） based hyperparameter optimization algorithm for the neural network was proposed， which was named IRCGA-DNN （IRCGA for Deep Neural Network）. Firstly， a real-coded form was used to represent the values of hyperparameters， which made the search space of hyperparameters more flexible. Then， a hierarchical proportional selection operator was introduced to enhance the diversity of the solution set. Finally， improved single-point crossover and variational operators were designed to explore the hyperparameter space more thoroughly and improve the efficiency and quality of the optimization algorithm， respectively. Two simulation datasets were used to show IRCGA’s performance in damage effectiveness prediction and convergence efficiency. The experimental results on two datasets indicate that， compared to GA-DNN（Genetic Algorithm for Deep Neural Network）， the proposed algorithm reduces the convergence iterations by 8.7% and 13.6% individually， and the MSE （Mean Square Error） is not much different； compared to IGA-DNN（Improved Genetic Algorithm for Deep Neural Network）， IRCGA-DNN achieves reductions of 22.2% and 13.6% in convergence iterations respectively. Experimental results show that the proposed algorithm is better in both convergence speed and prediction performance， and is suitable for hyperparametric optimization of neural networks.

Key words: real coding, genetic algorithm, hyperparameter optimization, evolutionary neural network, machine learning

摘要：

针对神经网络超参数优化效果差、容易陷入次优解和优化效率低的问题，提出一种基于改进实数编码遗传算法（IRCGA）的深度神经网络超参数优化算法——IRCGA-DNN（IRCGA for Deep Neural Network）。首先，采用实数编码方式表示超参数的取值，使超参数的搜索空间更灵活；然后，引入分层比例选择算子增加解集多样性；最后，分别设计了改进的单点交叉和变异算子，以更全面地探索超参数空间，提高优化算法的效率和质量。基于两个仿真数据集，验证IRCGA-DNN的毁伤效果预测性能和收敛效率。实验结果表明，在两个数据集上，与GA-DNN（Genetic Algorithm for Deep Neural Network）相比，所提算法的收敛迭代次数分别减少了8.7%和13.6%，均方误差（MSE）相差不大；与IGA-DNN（Improved GA-DNN）相比，IRCGA-DNN的收敛迭代次数分别减少了22.2%和13.6%。实验结果表明，所提算法收敛速度和预测性能均更优，能有效处理神经网络超参数优化问题。

关键词: 实数编码, 遗传算法, 超参数优化, 进化神经网络, 机器学习

CLC Number:

TP301.6

Wei SHE, Yang LI, Lihong ZHONG, Defeng KONG, Zhao TIAN. Hyperparameter optimization for neural network based on improved real coding genetic algorithm[J]. Journal of Computer Applications, 2024, 44(3): 671-676.

佘维, 李阳, 钟李红, 孔德锋, 田钊. 基于改进实数编码遗传算法的神经网络超参数优化[J]. 《计算机应用》唯一官方网站, 2024, 44(3): 671-676.

Figures/Tables 10

References 15

1	孙永泽，陆忠华. 基于超限学习机与随机响应面方法的深度学习超参数优化算法［J］. 高技术通讯， 2019， 29（12）： 1165-1174. 10.3772/j.issn.1002-0470.2019.12.001
	SUN Y Z， LU Z H. A hyperparameter tuning algorithm based on extreme learning machine and stochastic response surface method for deep learning ［J］. Chinese High Technology Letters， 2019， 29（12）： 1165-1174. 10.3772/j.issn.1002-0470.2019.12.001
2	郭宇. 基于分布式粒子群算法的深度学习超参数优化研究［D］. 广州：华南理工大学， 2021： 25-45. 10.1080/01969722.2020.1827797
	GUO Y. Research on hyperparameter optimization of deep learning based on distributed particle swarm optimization ［D］. Guangzhou： South China University of Technology， 2021：25-45. 10.1080/01969722.2020.1827797
3	牛芗洁，王玉洁，唐剑. 基于遗传算法的PID控制器参数优化研究［J］. 计算机仿真， 2010， 27（11）： 180-182，230. 10.3969/j.issn.1006-9348.2010.11.047
	NIU X J， WANG Y J， TANG J. Optimization parameters of PID controller parameters based on genetic algorithm ［J］. Computer Simulation， 2010， 27（11）： 180-182，230. 10.3969/j.issn.1006-9348.2010.11.047
4	陈铁明，董航. 使用蚁群算法和深度强化学习的工业异常入侵检测［J］. 小型微型计算机系统， 2022， 43（4）： 779-784.
	CHEN T M， DONG H. Industrial anomaly intrusion detection using ant colony algorithm and deep reinforcement learning ［J］. Journal of Chinese Computer Systems， 2022， 43（4）： 779-784.
5	孟思涛. 基于改进灰狼算法优化模型的短时交通流预测研究［D］. 上海：东华大学， 2021： 22-49.
	MENG S T. Research on short-term traffic flow prediction based on improved grey wolf algorithm optimization model ［D］. Shanghai： Donghua University， 2021： 22-49.
6	张琳，汪廷华，周慧颖. 基于群智能算法的SVR参数优化研究进展［J］. 计算机工程与应用， 2021， 57（16）： 50-64.
	ZHANG L， WANG T H， ZHOU H Y. Research progress on parameter optimization of SVR based on swarm intelligence algorithm ［J］. Computer Engineering and Applications， 2021， 57（16）： 50-64.
7	LIANG X. Image-based pos-disaster inspection of reinforced concrete bridge systems using deep learning with Bayesian optimization ［J］. Computer-Aided Civil and Infrastructure Engineering， 2019， 34（5）： 415-430. 10.1111/mice.12425
8	LUO X J， OYEDELE L O， AJAYI A O， et al. Feature extraction and genetic algorithm enhanced adaptive deep neural network for energy consumption prediction in buildings ［J］. Renewable and Sustainable Energy Reviews， 2020，131：109980. 10.1016/j.rser.2020.109980
9	ZHANG Y， LI P， WANG X. Intrusion detection for IoT based on improved genetic algorithm and deep belief network ［J］. IEEE Access， 2019， 7： 31711-31722. 10.1109/access.2019.2903723
10	张宗腾，张琳，谢春燕，等. 基于改进GA-BP神经网络的目标毁伤效果评估［J］. 火力与指挥控制， 2021， 46（11）： 43-48. 10.3969/j.issn.1002-0640.2021.11.006
	ZHANG Z T， ZHANG L， XIE C Y， et al. Battle damage effect assessment based on improved GA-BP neural network［J］. Fire Control & Command Control， 2021， 46（11）： 43-48. 10.3969/j.issn.1002-0640.2021.11.006
11	汪维，刘光昆，赵强，等. 近爆作用下方形板表面爆炸载荷分布函数研究［J］. 中国科学（物理学力学天文学）， 2020， 50（2）： 144-152. 10.1360/sspma-2019-0188
	WANG W， LIU G K， ZHAO Q， et al. Study on load distributing function of square slab surface under close-in blast loading［J］. SCIENTIA SINICA Physica， Mechanica & Astronomica， 2020， 50（2）： 144-152. 10.1360/sspma-2019-0188
12	刘光昆，刘瑞朝，汪维，等. 地下钢筋混凝土拱形结构在顶爆条件下的抗爆试验［J］. 含能材料， 2021， 29（2）： 157-165. 10.11943/CJEM2020281
	LIU G K， LIU R C， WANG W， et al. Blast resistance experiment of underground reinforced concrete arch structure under top explosion ［J］. Chinese Journal of Energetic Materials， 2021， 29（2）： 157-165. 10.11943/CJEM2020281
13	LORENZO P R， NALEPA J， KAWULOK M， et al. Particle swarm optimization for hyper-parameter selection in deep neural networks ［C］// Proceedings of the 2017 Genetic and Evolutionary Computation Conference. New York： ACM， 2017： 481-488. 10.1145/3071178.3071208
14	NIKBAKHT S， ANITESCU C， RABCZUK T. Optimizing the neural network hyperparameters utilizing genetic algorithm［J］. Journal of Zhejiang University — SCIENCE A， 2021， 22（6）： 407-426. 10.1631/jzus.a2000384
15	SILVA SANTOS C E DA， SAMPAIO R C， DOS SANTOS COELHO L， et al. Multi-objective adaptive differential evolution for SVM/SVR hyperparameters selection ［J］. Pattern Recognition， 2021， 110： 107649. 10.1016/j.patcog.2020.107649

超参数	取值范围	超参数	取值范围
n₁	［2，2r］	a₂	［0，5］
n₂	［2，2r］	a₃	［0，5］
n₃	［2，2r］	batch	［0，3］
a₁	［0，5］	lr	［0，2］

超参数	取值范围	超参数	取值范围
n₁	［2，2r］	a₂	［0，5］
n₂	［2，2r］	a₃	［0，5］
n₃	［2，2r］	batch	［0，3］
a₁	［0，5］	lr	［0，2］

组别	不同选择比例下的MAE/10^-2
组别	α=50%	α=66%	α= 83%
平均值	0.677	0.656	0.767
第1组	0.672	0.632	0.735
第2组	0.713	0.683	0.814
第3组	0.645	0.652	0.751

组别	不同选择比例下的MAE/10^-2
组别	α=50%	α=66%	α= 83%
平均值	0.677	0.656	0.767
第1组	0.672	0.632	0.735
第2组	0.713	0.683	0.814
第3组	0.645	0.652	0.751

算法	MAE/10^-2			MSE/10^-2
算法	最优值	最差值	平均值	最优值	最差值	平均值
PSO-DNN	0.631	0.864	0.757	0.739	0.813	0.786
GA-DNN	0.608	0.713	0.654	0.714	0.786	0.754
DE-DNN	0.684	0.773	0.731	0.779	0.843	0.812
IGA-DNN	0.601	0.727	0.650	0.730	0.776	0.751
IRCGA-DNN	0.612	0.674	0.649	0.732	0.749	0.741

Hyperparameter optimization for neural network based on improved real coding genetic algorithm

基于改进实数编码遗传算法的神经网络超参数优化

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 15

Related Articles 15

Recommended Articles

Metrics

数据集	算法	收敛迭代次数	MSE/10^-2
数据集1	PSO-DNN	13	0.765
	GA-DNN	23	0.775
	DE-DNN	25	0.817
	IGA-DNN	27	0.756
	IRCGA-DNN	21	0.742
数据集2	PSO-DNN	12	0.362
	GA-DNN	22	0.357
	DE-DNN	19	0.387
	IGA-DNN	22	0.365
	IRCGA-DNN	19	0.353

[1]	Dong LIU, Chenhang LI, Changmao WU, Faxin RU, Yuanyuan XIA. Automatic design of optical systems based on correctable reinforced search genetic algorithm [J]. Journal of Computer Applications, 2024, 44(9): 2838-2847.
[2]	Xuebin CHEN, Zhiqiang REN, Hongyang ZHANG. Review on security threats and defense measures in federated learning [J]. Journal of Computer Applications, 2024, 44(6): 1663-1672.
[3]	Zihao YAO, Yuanming LI, Ziqiang MA, Yang LI, Lianggen WEI. Multi-object cache side-channel attack detection model based on machine learning [J]. Journal of Computer Applications, 2024, 44(6): 1862-1871.
[4]	Lin GAO, Yu ZHOU, Tak Wu KWONG. Evolutionary bi-level adaptive local feature selection [J]. Journal of Computer Applications, 2024, 44(5): 1408-1414.
[5]	Xuanfeng LI, Shengcai LIU, Ke TANG. Novel genetic algorithm for solving chance-constrained multiple-choice Knapsack problems [J]. Journal of Computer Applications, 2024, 44(5): 1378-1385.
[6]	Jianqiang LI, Zhou HE. Hybrid NSGA-Ⅱ for vehicle routing problem with multi-trip pickup and delivery [J]. Journal of Computer Applications, 2024, 44(4): 1187-1194.
[7]	Yi ZHENG, Cunyi LIAO, Tianqian ZHANG, Ji WANG, Shouyin LIU. Image denoising-based cell-level RSRP estimation method for urban areas [J]. Journal of Computer Applications, 2024, 44(3): 855-862.
[8]	Xuebin CHEN, Changsheng QU. Overview of backdoor attacks and defense in federated learning [J]. Journal of Computer Applications, 2024, 44(11): 3459-3469.
[9]	Rui ZHANG, Junming PAN, Xiaolu BAI, Jing HU, Rongguo ZHANG, Pengyun ZHANG. Agent model for hyperparameter self-optimization of deep classification model [J]. Journal of Computer Applications, 2024, 44(10): 3021-3031.
[10]	Renke SUN, Zhiyu HUANGFU, Hu CHEN, Zhongnian LI, Xinzheng XU. Survey of neural architecture search [J]. Journal of Computer Applications, 2024, 44(10): 2983-2994.
[11]	Wenze CHAI, Jing FAN, Shukui SUN, Yiming LIANG, Jingfeng LIU. Overview of deep metric learning [J]. Journal of Computer Applications, 2024, 44(10): 2995-3010.
[12]	Chunyong YIN, Yongcheng ZHOU. Automatically adjusted clustered federated learning for double-ended clustering [J]. Journal of Computer Applications, 2024, 44(10): 3011-3020.
[13]	Haoyang CUI, Hui ZHANG, Lei ZHOU, Chunming YANG, Bo LI, Xujian ZHAO. Multi-similarity K-nearest neighbor classification algorithm with ordered pairs of normalized real numbers [J]. Journal of Computer Applications, 2023, 43(9): 2673-2678.
[14]	Jing ZHONG, Chen LIN, Zhiwei SHENG, Shibin ZHANG. Quantum K-Means algorithm based on Hamming distance [J]. Journal of Computer Applications, 2023, 43(8): 2493-2498.
[15]	Mengjie LAN, Jianping CAI, Lan SUN. Self-regularization optimization methods for Non-IID data in federated learning [J]. Journal of Computer Applications, 2023, 43(7): 2073-2081.

算法	MAE/10^-1			MSE/10^-2
算法	最优值	最差值	平均值	最优值	最差值	平均值
PSO-DNN	0.415	0.509	0.461	0.360	0.410	0.393
GA-DNN	0.400	0.449	0.421	0.374	0.398	0.383
DE-DNN	0.463	0.556	0.499	0.387	0.431	0.402
IGA-DNN	0.392	0.458	0.415	0.354	0.387	0.378
IRCGA-DNN	0.383	0.430	0.404	0.345	0.387	0.375

算法	MAE/10^-1			MSE/10^-2
算法	最优值	最差值	平均值	最优值	最差值	平均值
PSO-DNN	0.415	0.509	0.461	0.360	0.410	0.393
GA-DNN	0.400	0.449	0.421	0.374	0.398	0.383
DE-DNN	0.463	0.556	0.499	0.387	0.431	0.402
IGA-DNN	0.392	0.458	0.415	0.354	0.387	0.378
IRCGA-DNN	0.383	0.430	0.404	0.345	0.387	0.375