Survey of neural architecture search

doi:10.11772/j.issn.1001-9081.2023101374

Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (10): 2983-2994.DOI: 10.11772/j.issn.1001-9081.2023101374

• Artificial intelligence • Next Articles

Survey of neural architecture search

Renke SUN(), Zhiyu HUANGFU, Hu CHEN, Zhongnian LI, Xinzheng XU

School of Compute Science and Technology，China University of Mining and Technology，Xuzhou Jiangsu 221116，China

Received:2023-10-13 Revised:2024-01-17 Accepted:2024-01-19 Online:2024-10-15 Published:2024-10-10
Contact: Renke SUN
About author:HUANGFU Zhiyu， born in 2000， M. S. candidate. His research interests include machine learning， medical image processing.
CHEN Hu， born in 1998， M. S. candidate. His research interests include machine learning， embedded artificial intelligence.
LI Zhongnian， born in 1991， Ph. D.， lecturer. His research interests include machine learning， medical image processing.
XU Xinzheng， born in 1980， Ph. D.， professor. His research interests include machine learning， embedded artificial intelligence.
Supported by:
National Natural Science Foundation of China(61976217);Xuzhou Science and Technology Planning Project(KC21193)

神经架构搜索综述

孙仁科(), 皇甫志宇, 陈虎, 李仲年, 许新征

中国矿业大学计算机科学与技术学院，江苏徐州 221116

通讯作者: 孙仁科
作者简介:孙仁科（1976—），男，江苏徐州人，讲师，博士，CCF会员，主要研究方向：机器学习、计算机视觉、嵌入式系统 srk@cumt.edu.cn
皇甫志宇（2000—），男，河南商丘人，硕士研究生，主要研究方向：机器学习、医学图像处理
陈虎（1998—），男，山东潍坊人，硕士研究生，主要研究方向：机器学习、嵌入式人工智能
李仲年（1991—），男，江苏邳州人，讲师，博士，CCF会员，主要研究方向：机器学习、医学图像处理
许新征（1980—），男，安徽宿州人，教授，博士，CCF高级会员，主要研究方向：机器学习、嵌入式人工智能。
基金资助:
国家自然科学基金资助项目(61976217);徐州市科技计划项目(KC21193)

Abstract

Abstract:

In recent years， deep learning has made breakthroughs in many fields due to its powerful representation capability， and the architecture of neural network is crucial to the final performance. However， the design of high-performance neural network architecture heavily relies on the priori knowledge and experience of the researchers. Because there are a lot of parameters for neural networks， it is difficult to design optimal neural network architecture. Therefore， automated Neural Architecture Search （NAS） gains significant attention. NAS is a technique that uses machine learning to automatically search for optimal network architecture without the need for a lot of human effort， and is an important means of future neural network design. NAS is essentially a search optimization problem， by designing search space， search strategy and performance evaluation strategy， NAS can automatically search the optimal network structure. Detailed and comprehensive analysis， comparison and summary for the latest research progress of NAS were provided from three aspects： search space， search strategy， and performance evaluation strategy， which facilitates readers to quickly understand the development process of NAS. And the future research directions of NAS were proposed.

Key words: Neural Architecture Search (NAS), deep learning, machine learning, neural network, search space, searching strategy, performance evaluation strategy

摘要：

近几年，深度学习因具有强大的表征能力，已经在许多领域中取得了突破性的进展，而神经网络的架构对它的性能至关重要。然而，高性能的神经网络架构设计严重依赖研究人员的先验知识和经验，神经网络参数量庞大，难以设计最优的神经网络架构，因此自动神经架构搜索（NAS）获得了极大的关注。NAS是一种使用机器学习的方法，可以在不需要大量人力的情况下，自动搜索最优网络架构的技术，是未来神经网络设计的重要手段之一。NAS本质上是一个搜索优化问题，通过对搜索空间、搜索策略和性能评估策略的设计，自动搜索最优的网络结构。从搜索空间、搜索策略和性能评估策略这3个方面详细且全面地分析、比较和总结目前NAS的研究进展，方便读者快速了解神经架构搜索的发展过程和各项技术的优缺点，并提出NAS未来可能的研究发展方向。

关键词: 神经架构搜索, 深度学习, 机器学习, 神经网络, 搜索空间, 搜索策略, 性能评估策略

CLC Number:

TP301.6

Renke SUN, Zhiyu HUANGFU, Hu CHEN, Zhongnian LI, Xinzheng XU. Survey of neural architecture search[J]. Journal of Computer Applications, 2024, 44(10): 2983-2994.

孙仁科, 皇甫志宇, 陈虎, 李仲年, 许新征. 神经架构搜索综述[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 2983-2994.

Figures/Tables 16

References 98

1	LIU Y， CHEN L， LI C， et al. Long-term and short-term memory networks based on forgetting memristors ［J］. Soft Computing， 2023， 27（23）： 18403-18418.
2	冯洋，邵晨泽.神经机器翻译前沿综述［J］.中文信息学报，2020，34（7）：1-18.
	FENG Y， SHAO C Z. Frontiers in neural machine translation： a literature review［J］. Journal of Chinese Information Processing， 2020， 34（7）：1-18.
3	李亚超，熊德意，张民. 神经机器翻译综述［J］. 计算机学报，2018，41（12）：2734-2755.
	LI Y C， XIONG D Y， ZHANG M. A survey of neural machine translation［J］. Chinese Journal of Computers， 2018， 41（12）：2734-2755.
4	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2014-09-04）［2023-10-01］. .
5	HOWARD A G， ZHU M， CHEN B， et al. MobileNets： efficient convolutional neural networks for mobile vision applications ［EB/OL］. （2017-04-17）［2023-10-01］. .
6	KRIZHEVSKY A， SUTSKEVER I， HINTON G E. ImageNet classification with deep convolutional neural networks［C］// Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates， 2012： 1097-1105.
7	REN S， HE K， GIRSHICK R， et al. Faster R-CNN： towards real-time object detection with region proposal networks［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2015： 91-99.
8	LIU W， ANGUELOV D， ERHAN D， et al. SSD： single shot MultiBox detector［C］// Proceedings of the 14th European Conference on Computer Vision. Cham： Springer， 2016： 21-37.
9	LIN T-Y， GOYAL P， GIRSHICK R， et al. Focal loss for dense object detection［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2999-3007.
10	LeCUN Y， BOTTOU L， BENGIO Y， et al. Gradient-based learning applied to document recognition ［J］. Proceedings of the IEEE， 1998， 86（11）： 2278-2324.
11	SZEGEDY C， LIU W， JIA Y， et al. Going deeper with convolutions［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015：1-9.
12	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778.
13	HUANG G， LIU Z， VAN DER MAATEN L， et al. Densely connected convolutional networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2261-2269.
14	ZHANG X， ZHOU X， LIN M， et al. ShuffleNet： an extremely efficient convolutional neural network for mobile devices［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 6848-6856.
15	HU J， SHEN L， SUN G. Squeeze-and-excitation networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7132-7141.
16	CHENG X， ZHONG Y， HARANDI M， et al. Hierarchical neural architecture search for deep stereo matching［C］// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates， 2020： 22158-22169.
17	ZHANG M， LI H， PAN S， et al. One-shot neural architecture search： maximising diversity to overcome catastrophic forgetting［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2021， 43（9）： 2921-2935.
18	ZOPH B， LE Q V. Neural architecture search with reinforcement learning［EB/OL］. （2017-02-15）［2023-10-01］. .
19	BAKER B， GUPTA O， NAIK N， et al. Designing neural network architectures using reinforcement learning ［EB/OL］. （2017-11-07）［2023-10-01］. .
20	REAL E， MOORE S， SELLE A， et al. Large-scale evolution of image classifiers［C］// Proceedings of the 34th International Conference on Machine Learning. New York： ACM， 2017： 2902-2911.
21	SHU Y， WANG. W， CAI S. Understanding architectures learnt by cell-based neural architecture search ［EB/OL］.（2019-09-20）［2023-10-01］. .
22	PHAM H， GUAN M Y， ZOPH B， et al. Efficient neural architecture search via parameter sharing ［EB/OL］. （2018-02-09）［2023-10-01］. .
23	BAKER B， GUPTA O， RASKAR R， et al. Accelerating neural architecture search using performance prediction［EB/OL］. （2017-05-30）［2023-10-01］. .
24	BASHIVAN P， TENSEN M， DICARLO J. Teacher guided architecture search［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019：5319-5328.
25	ZHENG X， JI R， TANG L， et al. Multinomial distribution learning for effective neural architecture search［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 1304-1313.
26	MEI J R， LI Y W， JIN X J， et al. AtomNAS： fine-grained end-to-end neural architecture search ［EB/OL］. （2020-02-23）［2023-10-01］. .
27	GUO Z， ZHANG X， MU H， et al. Single path One-shot neural architecture search with uniform sampling［EB/OL］. （2020-07-08）［2023-10-01］. .
28	CHEN Y， YANG T， ZHANG X， et al. DetNAS： neural architecture search on object detection［EB/OL］. （2019-10-30）［2023-10-01］. .
29	GHIASI G， LIN T-Y， LE Q V. NAS-FPN： learning scalable feature pyramid architecture for object detection［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 7029-7038.
30	PENG J， SUN M， ZHANG Z， et al. Efficient neural architecture transformation search in channel level for object detection［EB/OL］. （2019-09-05）［2023-10-01］. .
31	LIANG F， LIN C， GUO R， et al. Computation reallocation for object detection［EB/OL］.（2019-02-24）［2023-10-01］. .
32	DING S， CHEN T， GONG X， et al. AutoSpeech： neural architecture search for speaker recognition［EB/OL］. （2020-05-07）［2023-10-01］. .
33	LIU C， CHEN L-C， SCHROFF F， et al. Auto-DeepLab： hierarchical neural architecture search for semantic image segmentation［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE，2019： 82-92.
34	ZHANG Y， QIU Z， LIU J， et al. Customizable architecture search for semantic segmentation［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 11633-11642.
35	NEKRASOV V， CHEN H， SHEN C， et al. Fast neural architecture search of compact semantic segmentation models via auxiliary cells［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9118-9127.
36	LU Z， WHALEN I， BODDETI V， et al. NSGA-Net： s multi-objective genetic algorithm for neural architecture search［EB/OL］.（2019-04-18）［2023-10-01］. .
37	ELSKEN T， METZEN J H， HUTTER F. Efficient multi-objective neural architecture search via Lamarckian evolution［EB/OL］. （2018-04-24）［2023-10-01］. .
38	XIONG Y， MEHTA R， SINGH V. Resource constrained neural network architecture search： will a submodularity assumption help？［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 1901-1910.
39	LI W， WEN S， SHI K B， et al. Neural architecture search with a lightweight Transformer for text-to-image synthesis［J］. IEEE Transactions on Network Science and Engineering， 2022， 9（3）： 1567-1576.
40	XIE B， CHANG H， ZHANG Z， et al. Adversarially robust neural architecture search for graph neural networks［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 8143-8152.
41	CHENG A， WANG J， ZHANG X S， et al. DPNAS： neural architecture search for deep learning with differential privacy［C］// Proceedings of the 36th International Joint Conference on Artificial Intelligence. Menlo Park： AAAI Press， 2022： 6358-6366.
42	LIU C， LENG Z， SUN P， et al. LidarNAS： unifying and searching neural architectures for 3D point clouds［C］// Proceedings of the 17th European Conference on Computer Vision. Cham：Springer， 2022： 158-175.
43	FALANTI A， LOMURNO E， ARDAGNA D， et al. POPNASv3： a Pareto-optimal neural architecture search solution for image and time series classification［J］. Applied Soft Computing， 2023， 145： 110555.
44	KAIROUZ P， McMAHAN B， AVENT B， et al. Advances and open problems in federated learning［EB/OL］. （2019-10-10）［2023-10-01］. .
45	ZHU H， JIN Y. Multi-objective evolutionary federated learning［J］. IEEE Transactions on Neural Networks and Learning Systems， 2020， 31（4）： 1310-1322.
46	XU M， ZHAO Y， BIAN K， et al. Federated neural architecture search［EB/OL］. （2022-02-15）［2023-10-01］. .
47	LIU S， ZHANG H， JIN Y. A survey on computationally efficient neural architecture search［J］. Journal of Automation and Intelligence， 2022， 1（1）：100002.
48	ZOPH B， VASUDEVAN V， SHLENS J， et al. Learning transferable architectures for scalable image recognition［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8697-8710.
49	CAI H， YANG J， ZHANG W， et al. Path-level network transformation for efficient architecture search［C］// Proceedings of the 35th International Conference on Machine Learning. New York： JMLR.org， 2018： 677-686.
50	RAWAL A， MIIKKULAINEN R. From nodes to networks： evolving recurrent neural networks ［EB/OL］.（2018-07-07）［2023-10-01］. .
51	LIU H， SIMONYAN K， YANG Y. DARTS： differentiable architecture search ［EB/OL］. （2019-04-23）［2023-10-01］. .
52	HUANG H， SHEN L， HE C， et al. Differentiable neural architecture search for extremely lightweight image super-resolution［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2023， 33（6）： 2672-2682.
53	TAN M， CHEN B， PANG R， et al. MNASNet： platform-aware neural architecture search for mobile［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 2815-2823.
54	LI L， TALWALKAR A. Random search and reproducibility for neural architecture search ［EB/OL］. （2019-02-20）［2023-10-01］. .
55	BERGSTRA J， BENGIO Y. Random search for hyper-parameter optimization［J］. Journal of Machine Learning Research， 2012，13： 281-305.
56	BERGSTRA J， YAMINS D，COX D. Making a science of model search： hyperparameter optimization in hundreds of dimensions for vision architectures［C］// Proceedings of the 30th International Conference on Machine Learning. New York： JMLR.org， 2013： 115-123.
57	YANG J， LIU Y， XU H. HOTNAS： hierarchical optimal transport for neural architecture search［C］// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2023： 11990-12000.
58	SHEN Y， LI Y， ZHENG J， et al. ProxyBO： accelerating neural architecture search via Bayesian optimization with zero-cost proxies［C］// Proceedings of the 37th AAAI Conference on Artificial Intelligence. Menlo Park： AAAI Press， 2023： 9792-9801.
59	ZHONG Z， YAN J， WU W， et al. Practical block-wise neural network architecture generation［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2423-2432.
60	CAI H， CHEN T， ZHANG W， et al. Efficient architecture search by network transformation ［EB/OL］. （2017-11-21）［2023-10-01］. .
61	TAN M， LE Q V. EfficientNet： rethinking model scaling for convolutional neural networks ［C］// Proceedings of the 2019 International Conference on Machine Learning. New York： JMLR.org， 2019： 6105-6114.
62	SUGANUMA M， SHIRAKAWA S， NAGAO T. A genetic programming approach to designing convolutional neural network architectures［EB/OL］. （2017-08-17）［2023-10-01］. .
63	XIE L， YUILLE A. Genetic CNN ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 1388-1397.
64	CHEN T， GOODFELLOW I， SHLENS J. Net2Net： accelerating learning via knowledge transfer ［EB/OL］. （2015-11-18）［2023-10-01］. .
65	LIU H， SIMONYAN K， VINYALS O， et al. Hierarchical representations for efficient architecture search［EB/OL］. ［2023-10-01］. .
66	CUBUK E D， ZOPH B， SCHOENHOLZ S S， et al. Intriguing properties of adversarial examples［EB/OL］. ［2023-10-01］. .
67	LAWTON N， KUMAR A， THATTAI G， et al. Neural architecture search for parameter-efficient fine-tuning of large pre-trained language models ［EB/OL］. （2023-05-26）［2023-10-01］. .
68	REAL E， AGGARWAL A， HUANG Y P， et al. Regularized evolution for image classifier architecture search ［EB/OL］. （2019-02-16）［2023-10-01］. .
69	ELSKEN T， METZEN J H， HUTTER F. Simple and efficient architecture search for convolutional neural networks［EB/OL］. ［2023-10-01］. .
70	LI X， ZHOU Y， PAN Z， et al. Partial order pruning： for best speed/accuracy trade-off in neural architecture search ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 9137-9145.
71	WISTUBA M. Deep learning architecture search by neuro-cell-based evolution with function-preserving mutations［C］// Proceedings of the 2018 Joint European Conference on Machine Learning and Knowledge Discovery in Database. Cham：Springer， 2018：243-258.
72	CHU X， ZHANG B， XU R， et al. Multi-objective reinforce devolution in mobile neural architecture search ［EB/OL］. （2019-01-16）［2023-10-01］. .
73	CAI H， ZHU L， HAN S. ProxylessNAS： direct neural architecture search on target task and hardware［EB/OL］. （2019-02-23）［2023-10-01］. .
74	XU Y， XIE L， ZHANG X， et al. PC-DARTS： partial channel connections for memory-efficient differentiable architecture search ［EB/OL］.（2020-04-07）［2023-10-01］. .
75	CHEN X， XIE L， WU J， et al. Progressive differentiable architecture search： bridging the depth gap between search and evaluation ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 1294-1303.
76	HUNDT A， JAIN V， HAGER G. sharpDARTS： faster and more accurate differentiable architecture search ［EB/OL］.（2019-03-23）［2023-10-01］. .
77	XUE Y， QIN J. Partial connection based on channel attention for differentiable neural architecture search［J］. IEEE Transactions on Industrial Informatics， 2023，19（5）：6804-6813.
78	XIAO H， WANG Z， ZHU Z， et al. Shapley-NAS： discovering operation contribution for neural architecture search［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 11882-11891.
79	CHRABASZCZ P， LOSHCHILOV I， HUTTER F. A downsampled variant of ImageNet as an alternative to the CIFAR datasets ［EB/OL］. （2017-07-27）］［2023-10-01］. .
80	SNOEK J， RIPPEL O， SWERSKY K， et al. Scalable Bayesian optimization using deep neural networks［C］// Proceedings of the 32nd International Conference on Machine Learning. New York： JMLR.org， 2015： 2171-2180.
81	KLEIN A， FALKNER S， SPRINGENBERG J T， et al. Learning curve prediction with Bayesian neural networks ［EB/OL］. （2016-11-04）［2023-10-01］. .
82	DOMHAN T， SPRINGENBERG J T， HUTTER F. Speeding up automatic hyperparameter optimization of deep neural networks by extrapolation of learning curves［C］// Proceedings of the Twenty-Fourth International Conference on Artificial Intelligence. Palo Alto： AAAI Press， 2015：3460-3468.
83	MILLS K G， HAN F X， ZHANG J， et al. GENNAPE： towards generalized neural architecture performance estimators［C］// Proceedings of the 37th AAAI Conference on Artificial Intelligence. Menlo Park： AAAI Press， 2023： 9190-9199.
84	LIU C， ZOPH B， NEUMANN M， et al. Progressive neural architecture search ［C］// Proceedings of the 15th European Conference on Computer Vision. Cham： Springer， 2018： 19-35.
85	CAI H， GAN C， WANG T， et al. Once-for-All： train one network and specialize it for efficient deployment ［EB/OL］. ［2023-10-01］. .
86	DUAN Y， CHEN X， XU H， et al. TransNAS-Bench-101： improving transferability and generalizability of cross-task neural architecture search［C］// Proceedings of the 2021 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 5247-5256.
87	WEI T， WANG C， RUI Y， et al. Network morphism［C］// Proceedings of the 33rd International Conference on Machine Learning. New York： JMLR.org， 2016： 564-572.
88	LIU Y， SUN Y， XUE B， et al. A survey on evolutionary neural architecture search［J］. IEEE Transactions on Neural Networks and Learning Systems， 2023， 34（2）：550-570.
89	JIN H， SONG Q， HU X. Auto-Keras： an efficient neural architecture search system ［C］// Proceedings of the 25th ACM SIGKDD International Conference on Knowledge Discovery & Data mining. New York： ACM， 2019： 1946-1956.
90	GAIER A， HA D. Weight agnostic neural networks［C］// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook： Curran Associates， 2019： 5364-5378.
91	HUANG T， YOU S， WANG F， et al. GreedyNASv2： greedier search with a greedy path filter［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 11892-11901.
92	DONG P， NIU X， LI L， et al. Prior-guided One-shot neural architecture search［EB/OL］. （2022-07-27）［2023-10-01］. .
93	HE X， YAO J， WANG Y， et al. NAS-LID： efficient neural architecture search with local intrinsic dimension［C］// Proceedings of the 37th AAAI Conference on Artificial Intelligence. Menlo Park： AAAI Press， 2023： 7839-7847.
94	WANG H， GE C， CHEN H， et al. PreNAS： preferred One-shot learning towards efficient neural architecture search ［EB/OL］. （2013-04-28）［2023-10-01］. .
95	BI Y， XUE B， ZHANG M. An evolutionary deep learning approach using genetic programming with convolution operators for image classification［C］// Proceedings of the 2019 IEEE/CVF Congress on Evolutionary Computation. Piscataway： IEEE， 2019： 3197-3204.
96	EVANS B， AL-SAHAF H， XUE B， et al. Evolutionary deep learning： a genetic programming approach to image classification［C］// Proceedings of the 2018 IEEE Congress on Evolutionary Computation. Piscataway： IEEE， 2018： 1-6.
97	EVANS B P， AI-SAHAF H， XUE B， et al. Genetic programming and gradient descent： a memetic approach to binary image classification［EB/OL］. （2019-09-28）［2023-10-01］. .
98	KLEIN A， FALKNER S， BARTELS S， et al. Fast Bayesian hyperparameter optimization on large datasets ［J］. Electronic Journal of Statistics， 2017， 11（2）： 4945-4968.

网络	操作类型	参数	参数设置
CNN^［19］	卷积操作	滤波器大小	｛64，128，256，512｝
		卷积核大小	｛1，3，5｝
		步幅	｛1｝
	池化操作	池化层深度	小于12
		池化核大小	｛5，3，2｝
		步幅	｛3，2｝
	全连接操作	层深度	小于12
		层数	小于3
		神经元大小	｛512，256，128｝
RNN方正汇总行^［18］	循环神经网络操作	激活函数	tanh、ReLU、 Identity、sigmoid

网络	操作类型	参数	参数设置
CNN^［19］	卷积操作	滤波器大小	｛64，128，256，512｝
		卷积核大小	｛1，3，5｝
		步幅	｛1｝
	池化操作	池化层深度	小于12
		池化核大小	｛5，3，2｝
		步幅	｛3，2｝
	全连接操作	层深度	小于12
		层数	小于3
		神经元大小	｛512，256，128｝
RNN方正汇总行^［18］	循环神经网络操作	激活函数	tanh、ReLU、 Identity、sigmoid

搜索方法类别	算法	CIFAR-10			ImageNet
搜索方法类别	算法	Top-1/%	GPUDays	GPUs	Top-5/%	Top-1/%	GPUDays	GPUs
人工设计	ResNet^［12］	93.57	—	—	80.62	95.51	—	—
人工设计	DenseNet^［13］	96.54	—	—	78.54	94.46	—	—
神经进化	GeNet^［63］	92.90	17	—	72.13	90.26	17	—
神经进化	MNASNet^［53］	—	—	—	76.70	93.30	4.5	—
强化学习	MetaQNN^［19］	93.08	100	10	—	—	—	—
	BlockQNN^［61］	97.35	96	32	81.00	95.42	96	32（1080Ti）
	文献［18］算法	96.35	—	—	—	—	—	—
	ENAS^［22］	97.11	0.45	1	—	—	—	—
梯度	DARTS^［51］	97.23	4	4（1080Ti）	73.30	81.30	4	—
	P-DARTS^［75］	97.75	0.3	—	75.60	92.60	0.3	—
	sharpDARTS^［76］	98.07	0.8	1（2080Ti）	74.90	92.20	0.8	—
	PC-DARTS^［73］	97.43	0.1	4（1080Ti）	75.80	92.70	3.8	8（V100）
随机搜索	Random Search^［54］	96.09	8	—	—	—	—	—
贝叶斯优化	Bergstra^［56］	95.00	—	—	—	—	—	—
贝叶斯优化	ProxyBO^［58］	97.12	0.7	—	—	—	—	—

搜索方法类别	算法	CIFAR-10			ImageNet
搜索方法类别	算法	Top-1/%	GPUDays	GPUs	Top-5/%	Top-1/%	GPUDays	GPUs
人工设计	ResNet^［12］	93.57	—	—	80.62	95.51	—	—
人工设计	DenseNet^［13］	96.54	—	—	78.54	94.46	—	—
神经进化	GeNet^［63］	92.90	17	—	72.13	90.26	17	—
神经进化	MNASNet^［53］	—	—	—	76.70	93.30	4.5	—
强化学习	MetaQNN^［19］	93.08	100	10	—	—	—	—
	BlockQNN^［61］	97.35	96	32	81.00	95.42	96	32（1080Ti）
	文献［18］算法	96.35	—	—	—	—	—	—
	ENAS^［22］	97.11	0.45	1	—	—	—	—
梯度	DARTS^［51］	97.23	4	4（1080Ti）	73.30	81.30	4	—
	P-DARTS^［75］	97.75	0.3	—	75.60	92.60	0.3	—
	sharpDARTS^［76］	98.07	0.8	1（2080Ti）	74.90	92.20	0.8	—
	PC-DARTS^［73］	97.43	0.1	4（1080Ti）	75.80	92.70	3.8	8（V100）
随机搜索	Random Search^［54］	96.09	8	—	—	—	—	—
贝叶斯优化	Bergstra^［56］	95.00	—	—	—	—	—	—
贝叶斯优化	ProxyBO^［58］	97.12	0.7	—	—	—	—	—

[1]	Na WANG, Lin JIANG, Yuancheng LI, Yun ZHU. Optimization of tensor virtual machine operator fusion based on graph rewriting and fusion exploration [J]. Journal of Computer Applications, 2024, 44(9): 2802-2809.
[2]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[3]	Yun LI, Fuyou WANG, Peiguang JING, Su WANG, Ao XIAO. Uncertainty-based frame associated short video event detection method [J]. Journal of Computer Applications, 2024, 44(9): 2903-2910.
[4]	Tingjie TANG, Jiajin HUANG, Jin QIN. Session-based recommendation with graph auxiliary learning [J]. Journal of Computer Applications, 2024, 44(9): 2711-2718.
[5]	Rui ZHANG, Pengyun ZHANG, Meirong GAO. Self-optimized dual-modal multi-channel non-deep vestibular schwannoma recognition model [J]. Journal of Computer Applications, 2024, 44(9): 2975-2982.
[6]	Jinjin LI, Guoming SANG, Yijia ZHANG. Multi-domain fake news detection model enhanced by APK-CNN and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2674-2682.
[7]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[8]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[9]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[10]	Hang YANG, Wanggen LI, Gensheng ZHANG, Zhige WANG, Xin KAI. Multi-layer information interactive fusion algorithm based on graph neural network for session-based recommendation [J]. Journal of Computer Applications, 2024, 44(9): 2719-2725.
[11]	Guanglei YAO, Juxia XIONG, Guowu YANG. Flower pollination algorithm based on neural network optimization [J]. Journal of Computer Applications, 2024, 44(9): 2829-2837.
[12]	Ying HUANG, Jiayu YANG, Jiahao JIN, Bangrui WAN. Siamese mixed information fusion algorithm for RGBT tracking [J]. Journal of Computer Applications, 2024, 44(9): 2878-2885.
[13]	Yu DU, Yan ZHU. Constructing pre-trained dynamic graph neural network to predict disappearance of academic cooperation behavior [J]. Journal of Computer Applications, 2024, 44(9): 2726-2731.
[14]	Xingyao YANG, Yu CHEN, Jiong YU, Zulian ZHANG, Jiaying CHEN, Dongxiao WANG. Recommendation model combining self-features and contrastive learning [J]. Journal of Computer Applications, 2024, 44(9): 2704-2710.
[15]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.

Survey of neural architecture search

神经架构搜索综述

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 16

References 98

Related Articles 15

Recommended Articles

Metrics