Unlabeled network pruning algorithm based on Bayesian optimization

doi:10.11772/j.issn.1001-9081.2021112020

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (1): 30-36.DOI: 10.11772/j.issn.1001-9081.2021112020

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Unlabeled network pruning algorithm based on Bayesian optimization

GAO Yuanyuan¹, YU Zhenhua¹, DU Fang^1,2, SONG Lijuan^1,2

1.School of Information Engineering， Ningxia University， Yinchuan Ningxia 750021， China
2.Collaborative Innovation Center for Ningxia Big Data and Artificial Intelligence Co-founded by Ningxia Municipality and Ministry of Education（Ningxia University）， Yinchuan Ningxia 750021， China

Received:2021-11-29 Revised:2022-05-03 Online:2022-06-06
Contact: GAO Yuanyuan， born in 1996， M. S. Her research interests include deep learning， model compression.
About author:GAO Yuanyuan， born in 1996， M. S. Her research interests include deep learning， model compression；YU Zhenhua， born in 1989， Ph. D.， associate professor. His research interests include machine learning， computer vision；DU Fang， born in 1974， Ph. D.， professor. Her research interests include big data management， artificial intelligence；SONG Lijuan， born in 1978， Ph. D.， associate professor. Her research interests include image processing， computer vision；
Supported by:
This work is partially supported by Ningxia Natural Science Foundation （2018A0899）.

基于贝叶斯优化的无标签网络剪枝算法

高媛媛¹, 余振华¹, 杜方^1,2, 宋丽娟^1,2

1.宁夏大学信息工程学院，银川 750021
2.宁夏大数据与人工智能省部共建协同创新中心（宁夏大学），银川 750021

通讯作者: 高媛媛（1996—），女，陕西榆林人，硕士，主要研究方向：深度学习、模型压缩yy_Gao@nxu.edu.cn
作者简介:高媛媛（1996—），女，陕西榆林人，硕士，主要研究方向：深度学习、模型压缩；余振华（1989—），男，青海海东人，副教授，博士，主要研究方向：机器学习、计算机视觉；杜方（1974—），女，宁夏银川人，教授，博士，研究方向：大数据管理、人工智能；宋丽娟（1978—），女，宁夏石嘴山人，副教授，博士，研究方向：图像处理、计算机视觉；
基金资助:
宁夏自然科学基金资助项目（2018A0899）。

Abstract

Abstract: To deal with too many parameters and too much computation in Deep Neural Networks （DNNs）， an unlabeled neural network pruning algorithm based on Bayesian optimization was proposed. Firstly， based on a global pruning strategy， the sub-optimal compression ratio of the model caused by layer-by-layer pruning was avoided effectively. Secondly， the pruning process was independent on the labels of data samples， and the compression ratios of all layers were optimized by minimizing the distance between the output features of pruning and baseline networks. Finally， the Bayesian optimization algorithm was adopted to find the optimal compression ratio of each layer， thereby improving the efficiency and accuracy of sub-network search. Experimental results show that when compressing VGG-16 network by the proposed algorithm on CIFAR-10 dataset， the parameter compression ratio is 85.32%， and the Floating Point of Operations （FLOPS） compression ratio is 69.20% with only 0.43% accuracy loss. Therefore， the DNN model can be compressed effectively by the proposed algorithm， and the compressed model can still maintain good accuracy.

Key words: Deep Neural Network (DNN), model compression, network pruning, network structure search, Bayesian optimization

摘要： 针对深度神经网络（DNN）的参数和计算量过大问题，提出一种基于贝叶斯优化的无标签网络剪枝算法。首先，利用全局剪枝策略来有效避免以逐层方式修剪而导致的模型次优压缩率；其次，在网络剪枝过程中不依赖数据样本标签，并通过最小化剪枝网络与基线网络输出特征的距离对网络每层的压缩率进行优化；最后，利用贝叶斯优化算法寻找网络每一层的最优剪枝率，以提高子网搜索的效率和精度。实验结果表明，使用所提算法在CIFAR-10数据集上对VGG-16网络进行压缩，参数压缩率为85.32%，每秒浮点运算次数（FLOPS）压缩率为69.20%，而精度损失仅为0.43%。可见，所提算法可以有效地压缩DNN模型，且压缩后的模型仍能保持良好的精度。

关键词: 深度神经网络, 模型压缩, 网络剪枝, 网络结构搜索, 贝叶斯优化

CLC Number:

TP183

GAO Yuanyuan, YU Zhenhua, DU Fang, SONG Lijuan. Unlabeled network pruning algorithm based on Bayesian optimization[J]. Journal of Computer Applications, 2023, 43(1): 30-36.

高媛媛, 余振华, 杜方, 宋丽娟. 基于贝叶斯优化的无标签网络剪枝算法[J]. 《计算机应用》唯一官方网站, 2023, 43(1): 30-36.

References

1 GUO C X， FAN B， ZHANG Q， et al. AugFPN： improving multi-scale feature learning for object detection［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 12592-12601. 10.1109/cvpr42600.2020.01261
2 KONG X， TU Z P， SHI S M， et al. Neural machine translation with adequacy-oriented learning［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 6618-6625. 10.1609/aaai.v33i01.33016618
3 DONGARE A D， KHARDE R R， KACHARE A D. Introduction to artificial neural network［J］. International Journal of Engineering and Innovative Technology， 2012， 2（1）： 189-194.
4 江颉，李一飞.一种轻量化的CNN活动识别方法研究［J］.小型微型计算机系统， 2019， 40（9）： 1962-1967. 10.3969/j.issn.1000-1220.2019.09.029 JIANG J， LI Y F. Research on a lightweight CNN activity recognition method［J］. Journal of Chinese Computer Systems， 2019， 40（9）： 1962-1967. 10.3969/j.issn.1000-1220.2019.09.029
5 KRIZHEVSKY A， SUTSKEVER I， HINTON G E. ImageNet classification with deep convolutional neural networks［J］. Communications of the ACM， 2017， 60（6）： 84-90. 10.1145/3065386
6 靳丽蕾，杨文柱，王思乐，等.一种用于卷积神经网络压缩的混合剪枝方法［J］.小型微型计算机系统， 2018， 39（12）： 2596-2601. 10.3969/j.issn.1000-1220.2018.12.007 JIN L L， YANG W Z， WANG S L， et al. Mixed pruning method for convolutional neural network compression［J］. Journal of Chinese Computer Systems， 2018， 39（12）： 2596-2601. 10.3969/j.issn.1000-1220.2018.12.007
7 韩云飞，蒋同海，马玉鹏，等.深度神经网络的压缩研究［J］.计算机应用研究，2018，35（10）：2894-2897，2903. 10.3969/j.issn.1001-3695.2018.10.003 HAN Y F， JIANG T H， MA Y P， et al. Compression of deep neural networks［J］. Application Research of Computers， 2018， 35（10）：2894-2897， 2903. 10.3969/j.issn.1001-3695.2018.10.003
8 HAN S， MAO H Z， DALLY W J. Deep compression： compressing deep neural networks with pruning， trained quantization and Huffman coding［EB/OL］. （2016-02-15）［2021-10-11］.https：//arxiv.org/pdf/1510.00149.pdf. 10.1609/aaai.v35i9.16950
9 GUO Y W， YAO A B， CHEN Y R. Dynamic network surgery for efficient DNNs［C］// Proceedings of the 30th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2016： 1387-1395.
10 LI H， KADAV A， DURDANOVIC I， et al. Pruning filters for efficient ConvNets［EB/OL］. （2017-03-10）［2021-09-22］.https：//arxiv.org/pdf/1608.08710.pdf.
11 HE Y， LIU P， WANG Z W， et al. Filter pruning via geometric median for deep convolutional neural networks acceleration［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 4335-4344. 10.1109/cvpr.2019.00447
12 LIN M B， JI R R， WANG Y， et al. HRank： filter pruning using high-rank feature map［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 1526-1535. 10.1109/cvpr42600.2020.00160
13 WANG W X， CHEN M H， ZHAO S， et al. Accelerate CNNs from three dimensions： a comprehensive pruning framework［C］// Proceedings of the 38th International Conference on Machine Learning. New York： JMLR.org， 2021： 10717-10726.
14 ZOPH B， LE Q V. Neural architecture search with reinforcement learning［EB/OL］. （2017-02-15）［2021-02-15］.https：//arxiv.org/pdf/1611.01578.pdf.
15 REAL E， MOORE S， SELLE A， et al. Large-scale evolution of image classifiers［C］// Proceedings of the 34th International Conference on Machine Learning. New York： JMLR.org， 2017： 2902-2911.
16 REAL E， AGGARWAL A， HUANG Y P， et al. Regularized evolution for image classifier architecture search［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019： 4780-4789. 10.1609/aaai.v33i01.33014780
17 CAO S C， WANG X F， KITANI K M. Learnable embedding space for efficient neural architecture compression［EB/OL］. （2019-04-25）［2021-06-28］.https：//arxiv.org/pdf/1902.00383.pdf.
18 LIU N， MA X L， XU Z Y， et al. AutoCompress： an automatic DNN structured pruning framework for ultra-high compression rates［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 4876-4883. 10.1609/aaai.v34i04.5924
19 刘会东，杜方，余振华，等.基于强化学习的无标签网络剪枝［J］.模式识别与人工智能，2021，34（3）：214-222. 10.16451/j.cnki.issn1003-6059.202103003 LIU H D， DU F， YU Z H， et al. Label-free network pruning via reinforcement learning［J］. Pattern Recognition and Artificial Intelligence， 2021， 34（3）： 214-222. 10.16451/j.cnki.issn1003-6059.202103003
20 PELIKAN M， GOLDBERG D E， CANTú-PAZ E. BOA： the Bayesian optimization algorithm［C］// Proceedings of the 1st Annual Conference on Genetic and Evolutionary Computation - Volume 1. San Francisco： Morgan Kaufmann Publishers Inc.，1999： 525-532. 10.1162/106365600750078808
21 DENTON E， ZAREMBA W， BRUNA J， et al. Exploiting linear structure within convolutional networks for efficient evaluation［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 1. Cambridge： MIT Press， 2014： 1269-1277.
22 MINNEHAN B， SAVAKIS A. Cascaded projection： end-to-end network compression and acceleration［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 10707-10716. 10.1109/cvpr.2019.01097
23 GUPTA S， AGRAWAL A， GOPALAKRISHNAN K， et al. Deep learning with limited numerical precision［C］// Proceedings of the 32nd International Conference on Machine Learning. New York： JMLR.org， 2015： 1737-1746.
24 BA L J， CARUANA R. Do deep nets really need to be deep？［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2014，2： 2654-2662.
25 HINTON G， VINYALS O， DEAN J. Distilling the knowledge in a neural network ［EB/OL］. （2015-03-09）［2021-01-13］.https：//arxiv.org/pdf/1503.02531.pdf.
26 WANG W H， BAO H B， HUANG S H， et al. MiniLMv2： multi-head self-attention relation distillation for compressing pretrained Transformers［C］// Findings of the Association for Computational Linguistics： ACL-IJCNLP 2021. Stroudsburg， PA： ACL， 2021： 2140-2151. 10.18653/v1/2021.findings-acl.188
27 ZHAO H， WANG H B， LI X， et al. Progressive blockwise knowledge distillation for neural network acceleration［C］// Proceedings of the 27th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2018： 2769-2775. 10.24963/ijcai.2018/384
28 COURBARIAUX M， HUBARA I， SOUDRY D， et al. Binarized neural networks： training deep neural networks with weights and activations constrained to +1 or -1［EB/OL］. （2016-03-17）［2021-05-27］.https：//arxiv.org/pdf/1602.02830.pdf.
29 LI F F， ZHANG B， LIU B. Ternary weight networks［EB/OL］. （2016-11-19）［2021-10-11］.https：//arxiv.org/pdf/1605.04711.pdf.
30 HUBARA I， NAHSHAN Y， HANANI Y， et al. Accurate post training quantization with small calibration sets［C］// Proceedings of the 38th International Conference on Machine Learning. New York： JMLR.org， 2021： 4466-4475. 10.1007/s10994-021-06053-z
31 HUANG Z H， WANG N Y. Data-driven sparse structure selection for deep neural networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11220. Cham： Springer， 2018： 317-334.
32 HE Y， KANG G L， DONG X Y， et al. Soft filter pruning for accelerating deep convolutional neural networks［C］// Proceedings of the 27th International Joint Conference on Artificial Intelligence. California： ijcai.org， 2018： 2234-2240. 10.24963/ijcai.2018/309
33 DONG X Y， YANG Y. Network pruning via transformable architecture search［C/OL］// Proceedings of the 33rd Conference on Neural Information Processing Systems. ［2021-10-11］.https：//proceedings.neurips.cc/paper/2019/file/a01a0380ca3c61428c26a231f0e49a09-Paper.pdf. 10.1109/iccv.2019.00378
34 HE Y， DING Y H， LIU P， et al. Learning filter pruning criteria for deep convolutional neural networks acceleration［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2006-2015. 10.1109/cvpr42600.2020.00208

[1]	Rui SHI, Yong LI, Yanhan ZHU. Adversarial sample attack algorithm of modulation signal based on equalization of feature gradient [J]. Journal of Computer Applications, 2024, 44(8): 2521-2527.
[2]	Dongwei WANG, Baichen LIU, Zhi HAN, Yanmei WANG, Yandong TANG. Deep network compression method based on low-rank decomposition and vector quantization [J]. Journal of Computer Applications, 2024, 44(7): 1987-1994.
[3]	Mengmei YAN, Dongping YANG. Review of mean field theory for deep neural network [J]. Journal of Computer Applications, 2024, 44(2): 331-343.
[4]	Xujian ZHAO, Hanglin LI. Deep neural network compression algorithm based on hybrid mechanism [J]. Journal of Computer Applications, 2023, 43(9): 2686-2691.
[5]	Yunfei SHEN, Fei SHEN, Fang LI, Jun ZHANG. Deep neural network model acceleration method based on tensor virtual machine [J]. Journal of Computer Applications, 2023, 43(9): 2836-2844.
[6]	Xiaolin LI, Songjia YANG. Hybrid beamforming for multi-user mmWave relay networks using deep learning [J]. Journal of Computer Applications, 2023, 43(8): 2511-2516.
[7]	Haiyu YANG, Wenpu GUO, Kai KANG. Signal modulation recognition method based on convolutional long short-term deep neural network [J]. Journal of Computer Applications, 2023, 43(4): 1318-1322.
[8]	LIU Xiaoyu, CHEN Huaixin, LIU Biyuan, LIN Ying, MA Teng. License plate detection algorithm in unrestricted scenes based on adaptive confidence threshold [J]. Journal of Computer Applications, 2023, 43(1): 67-73.
[9]	Wentao MAO, Guifang WU, Chao WU, Zhi DOU. Animation video generation model based on Chinese impressionistic style transfer [J]. Journal of Computer Applications, 2022, 42(7): 2162-2169.
[10]	Meng YU, Wentao HE, Xuchuan ZHOU, Mengtian CUI, Keqi WU, Wenjie ZHOU. Review of recommendation system [J]. Journal of Computer Applications, 2022, 42(6): 1898-1913.
[11]	Xiao QIN, Miao CHENG, Shaobing ZHANG, Lian HE, Xiangwen SHI, Pinxue WANG, Shang ZENG. Pruning of YOLOv4 based on rank information in industrial scenes [J]. Journal of Computer Applications, 2022, 42(5): 1417-1423.
[12]	Quan CHEN, Li LI, Yongle CHEN, Yuexing DUAN. Adversarial attack algorithm for deep learning interpretability [J]. Journal of Computer Applications, 2022, 42(2): 510-518.
[13]	Xinyuan QIU, Zecong YE, Xiaolong CUI, Zhiqiang GAO. Survey of communication overhead of federated learning [J]. Journal of Computer Applications, 2022, 42(2): 333-342.
[14]	Anyi WANG, Heng ZHANG. Multi-input multi-output intelligent receiver model based on multi-label classification algorithm [J]. Journal of Computer Applications, 2022, 42(10): 3124-3129.
[15]	HUANG Jishuang, ZHANG Hua, LI Yonglong, ZHAO Hao, WANG Haoran, FENG Chuncheng. Hydraulic tunnel defect recognition method based on dynamic feature distillation [J]. Journal of Computer Applications, 2021, 41(8): 2358-2365.

Unlabeled network pruning algorithm based on Bayesian optimization

基于贝叶斯优化的无标签网络剪枝算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics