优化搜索空间下带约束的可微分神经网络架构搜索

doi:10.11772/j.issn.1001-9081.2021010170

《计算机应用》唯一官方网站 ›› 2022, Vol. 42 ›› Issue (1): 44-49.DOI: 10.11772/j.issn.1001-9081.2021010170

优化搜索空间下带约束的可微分神经网络架构搜索

李建明¹^,²(), 陈斌²^,³, 江志伟⁴, 覃健⁴

^1.中国科学院成都计算机应用研究所, 成都 610041
^2.中国科学院大学, 北京 100049
^3.哈尔滨工业大学(深圳) 国际人工智能研究院, 广东深圳 518055
^4.中科院广州电子技术有限公司, 广州 510070

收稿日期:2021-01-29 修回日期:2021-03-08 接受日期:2021-03-30 发布日期:2021-04-15 出版日期:2022-01-10
通讯作者: 李建明
作者简介:李建明（1989—），男，四川南充人，博士研究生，主要研究方向：机器视觉、人工智能、神经网络架构设计
陈斌（1970—），男，四川广汉人，教授，博士生导师，博士，主要研究方向：机器视觉、人工智能
江志伟（1970—），男，福建永定人，高级工程师，博士，主要研究方向：图形图像处理、人工智能、3D打印
覃健（1992—），男，广西钟山人，主要研究方向：自动化控制、视觉检测。

Constrained differentiable neural architecture search in optimized search space

Jianming LI¹^,²(), Bin CHEN²^,³, Zhiwei JIANG⁴, Jian QIN⁴

^1.Chengdu Institute of Computer Applications，Chinese Academy of Sciences，Chengdu Sichuan 610041，China
^2.University of Chinese Academy of Sciences，Beijing 100049，China
^3.International Research Institute of Artificial Intelligence，Harbin Institute of Technology，Shenzhen，Shenzhen Guangdong 518055，China
^4.Guangzhou Electronic Technology Company Limited，Chinese Academy of Sciences，Guangzhou Guangdong 510070，China

Received:2021-01-29 Revised:2021-03-08 Accepted:2021-03-30 Online:2021-04-15 Published:2022-01-10
Contact: Jianming LI
About author:LI Jianming， born in 1989， Ph. D. candidate. His research interests include computer vision， artificial intelligence， neural network architecture design.
CHEN Bin， born in 1970， Ph. D.， professor. His research interests include computer vision， artificial intelligence.
JIANG Zhiwei， born in 1970， Ph. D.， senior engineer. His research interests include graphics and image processing， artificial intelligence， 3D printing.
QIN Jian， born in 1992. His research interests include automatic control， visual inspection.

摘要/Abstract

摘要：

可微分架构搜索（DARTS）可高效、自动地设计神经网络架构，但其超网络的构建方式与派生策略的设计之间存在性能“鸿沟”。针对上述问题，提出了优化搜索空间下带约束的可微分神经网络架构搜索算法。首先，以候选操作关联的架构参数为量化指标来分析超网络的训练过程，发现在派生架构中未生效的候选操作none占据了权重最大的架构参数，从而导致算法搜得的架构表现欠佳，针对该问题设计了优化的搜索空间；然后，分析了DARTS超网络与派生架构之间的差异后，以架构参数为基础定义了架构熵，并把架构熵作为DARTS超网络目标函数的约束项，从而促使超网络缩小与派生架构的差异；最后，在CIFAR-10数据集上进行了实验。实验结果表明，所提算法在其中搜得的架构取得了97.17%的分类准确率，综合准确率、参数量和搜索时间优于对比算法。所提出的算法是有效的，提升了搜得架构在CIFAR-10数据集上的准确率。

关键词: 卷积神经网络, 网络架构搜索, 可微分架构搜索, 架构确定性, 架构熵

Abstract:

Differentiable ARchiTecture Search （DARTS） can design neural network architectures efficiently and automatically. However， there is a performance “wide gap” between the construction method of super network and the design of derivation strategy in it. To solve the above problem， a differentiable neural architecture search algorithm with constraint in optimal search space was proposed. Firstly， the training process of the super network was analyzed by using the architecture parameters associated with the candidate operations as the quantitative indicators， and it was found that the invalid candidate operation none occupied the architecture parameter with the maximum weight in deviation architecture， which caused that architectures obtained by the algorithm had poor performance. Aiming at this problem， an optimized search space was proposed. Then， the difference between the super network of DARTS and derivation architecture was analyzed， the architecture entropy was defined based on architecture parameters， and this architecture entropy was used as the constraint of the objective function of DARTS， so as to promote the super network to narrow the difference with the derivation strategy. Finally， experiments were conducted on CIFAR-10 dataset. The experimental results show that the searched architecture by the proposed algorithm achieved 97.17% classification accuracy in these experiments， better than the comparison algorithms in accuracy， parameter quantity and search time comprehensively. The proposed algorithm is effective and improves classification accuracy of searched architecture on CIFAR-10 dataset.

Key words: convolutional neural network, network architecture search, Differentiable ARchiTecture Search （DARTS）, architecture determinacy, architecture entropy

中图分类号:

TP183

李建明, 陈斌, 江志伟, 覃健. 优化搜索空间下带约束的可微分神经网络架构搜索[J]. 计算机应用, 2022, 42(1): 44-49.

Jianming LI, Bin CHEN, Zhiwei JIANG, Jian QIN. Constrained differentiable neural architecture search in optimized search space[J]. Journal of Computer Applications, 2022, 42(1): 44-49.

图/表 5

图1 两种典型的架构参数变化曲线

Fig.1 Two typical curves of changing trends of architecture parameters

图2 约束项引入前后的架构熵对比

Fig.2 Comparison of architecture entropy before and after introducing constraint

表1 CIFAR-10数据集上β不同取值时的平均准确率

Tab.1 Average accuracy on CIFAR-10 dataset with different β

$β$	平均准确率/%
0	97.00
0.01	97.09
0.1	96.78
1	96.97

表1 CIFAR-10数据集上β不同取值时的平均准确率

Tab.1 Average accuracy on CIFAR-10 dataset with different β

$β$	平均准确率/%
0	97.00
0.01	97.09
0.1	96.78
1	96.97

图3 CIFAR-10数据集上搜索到的cells

Fig.3 Searched cells on CIFAR-10 dataset

表2 与相关方法在CIFAR-10数据集上的准确率对比

Tab.2 Accuracy comparison with related methods on CIFAR-10 dataset

方法	准确率/ %	参数量/ M	搜索时间/（GPU·d）	方法类别
DARTS_V1^［10］	97.00	3.3	1.5	gradient-based
R-DARTS^［12］	96.95	—	—	gradient-based
ENAS^［18］	97.11	4.6	0.5	RL
SNAS^［13］	97.15	2.8	1.5	gradient-based
GDAS^［19］	97.18	3.4	0.2	gradient-based
NASNet-A^［7］	97.35	3.3	2000	RL
P-DARTS^［15］	97.50	3.4	0.3	gradient-based
本文方法	97.17	2.8	0.3	gradient-based

参考文献 19

1	KRIZHEVSKY A， SUTSKEVER I， HINTON G E. ImageNet classification with deep convolutional neural networks［C］// Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2012： 1097-1105.
2	SZEGEDY C， LIU W， JIA Y Q， et al. Going deeper with convolutions［C］// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2015： 1-9. 10.1109/cvpr.2015.7298594
3	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
4	HUANG G， LIU Z， MAATEN L VAN DER， et al. Densely connected convolutional networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2261-2269. 10.1109/cvpr.2017.243
5	HU J， SHEN L， SUN G. Squeeze-and-excitation networks［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7132-7141. 10.1109/cvpr.2018.00745
6	ZOPH B， LE Q V. Neural architecture search with reinforcement learning［EB/OL］. （2017-02-15）［2020-12-17］..
7	ZOPH B， VASUDEVAN V， SHLENS J， et al. Learning transferable architectures for scalable image recognition［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 8697-8710. 10.1109/cvpr.2018.00907
8	REAL E， AGGARWAL A， HUANG Y P， et al. Regularized evolution for image classifier architecture search［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 4780-4789. 10.1609/aaai.v33i01.33014780
9	SONG D H， XU C， JIA X， et al. Efficient residual dense block search for image super-resolution［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 12007-12014. 10.1609/aaai.v34i07.6877
10	LIU H X， SIMONYAN K， YANG Y M. DARTS： differentiable architecture search［EB/OL］. （2019-04-23）［2020-12-17］..
11	WU B C， DAI X L， ZHANG P Z， et al. FBNet： hardware-aware efficient ConvNet design via differentiable neural architecture search［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 10726-10734. 10.1109/cvpr.2019.01099
12	ZELA A， ELSKEN T， SAIKIA T， et al. Understanding and robustifying differentiable architecture search［EB/OL］. （2020-01-28）［2020-12-17］..
13	XIE S R， ZHENG H H， LIU C X， et al. SNAS： stochastic neural architecture search［EB/OL］. （2020-04-01）［2020-12-17］.. 10.1109/cvpr42600.2020.01210
14	CHU X X， ZHOU T B， ZHANG B， et al. Fair DARTS： eliminating unfair advantages in differentiable architecture search［C］// Proceedings of the 2020 European Conference on Computer Vision， LNCS12360. Cham： Springer， 2020： 465-480.
15	CHEN X， XIE L X， WU J， et al. Progressive differentiable architecture search： bridging the depth gap between search and evaluation［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2019： 1294-1303. 10.1109/iccv.2019.00138
16	XU Y H， XIE L X， ZHANG X P， et al. PC-DARTS： partial channel connections for memory-efficient differentiable architecture search［EB/OL］. （2020-04-07）［2020-12-17］.. 10.1109/cvpr42600.2020.01298
17	SHU Y， WANG W， CAI S F. Understanding architectures learnt by cell-based neural architecture search［EB/OL］. （2020-01-01）［2020-12-17］..
18	PHAM H， GUAN M Y， ZOPH B， et al. Efficient neural architecture search via parameter sharing［C］// Proceedings of the 35th International Conference on Machine Learning. New York： JMLR.org， 2018： 4095-4104.
19	DONG X Y， YANG Y. Searching for a robust neural architecture in four GPU hours［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 1761-1770. 10.1109/cvpr.2019.00186

[1]	包银鑫, 曹阳, 施佺. 基于改进时空残差卷积神经网络的城市路网短时交通流预测[J]. 《计算机应用》唯一官方网站, 2022, 42(1): 258-264.
[2]	李恒鑫, 常侃, 谭宇飞, 凌铭阳, 覃团发. 应用通道间相关性及增强信息蒸馏的彩色图像去马赛克网络[J]. 《计算机应用》唯一官方网站, 2022, 42(1): 245-251.
[3]	许慧青, 陈斌, 王敬飞, 陈志毅, 覃健. 基于卷积神经网络的细长路面病害检测方法[J]. 《计算机应用》唯一官方网站, 2022, 42(1): 265-272.
[4]	王贺兵, 张春梅. 基于非对称卷积-压缩激发-次代残差网络的人脸关键点检测[J]. 计算机应用, 2021, 41(9): 2741-2747.
[5]	宋中山, 梁家锐, 郑禄, 刘振宇, 帖军. 基于双向门控尺度特征融合的遥感场景分类[J]. 计算机应用, 2021, 41(9): 2726-2735.
[6]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[7]	张永斌, 常文欣, 孙连山, 张航. 基于字典的域名生成算法生成域名的检测方法[J]. 计算机应用, 2021, 41(9): 2609-2614.
[8]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[9]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[10]	牟长宁, 王海鹏, 周丕宇, 侯鑫行. 基于图卷积神经网络的串联质谱从头测序[J]. 计算机应用, 2021, 41(9): 2773-2779.
[11]	曾祥银, 郑伯川, 刘丹. 基于深度卷积神经网络和聚类的左右轨道线检测[J]. 计算机应用, 2021, 41(8): 2324-2329.
[12]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 《计算机应用》唯一官方网站, 2021, 41(8): 2273-2287.
[13]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[14]	黄程程, 董霄霄, 李钊. 基于二维Winograd算法的深流水线5×5卷积方法[J]. 计算机应用, 2021, 41(8): 2258-2264.
[15]	吴则举, 焦翠娟, 陈亮. 基于改进Faster R-CNN的轮胎缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1939-1946.

优化搜索空间下带约束的可微分神经网络架构搜索

Constrained differentiable neural architecture search in optimized search space

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 5

参考文献 19

相关文章 15

编辑推荐

Metrics