Gradient-based deep network pruning algorithm

doi:10.11772/j.issn.1001-9081.2019081374

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (5): 1253-1259.DOI: 10.11772/j.issn.1001-9081.2019081374

• Artificial intelligence • Previous Articles Next Articles

Gradient-based deep network pruning algorithm

WANG Zhongfeng^1,2,3, XU Zhiyuan^1,2,3,4, SONG Chunhe^1,2,3, ZHANG Hongyu⁵, CAI Yingkai⁵

1.State Key Laboratory of Robotics (Shenyang Institute of Automation， Chinese Academy of Sciences), ShenyangLiaoning 110016, China
2.Key Laboratory of Networked Control System(Shenyang Institute of Automation， Chinese Academy of Sciences),ShenyangLiaoning 110016,China
3.Institute of Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, ShenyangLiaoning 110169, China
4.University of Chinese Academy of Sciences, Beijing 100049, China
5.State Grid Liaoning Electric Power Company Limited, ShenyangLiaoning 110016, China

Received:2019-08-07 Revised:2019-10-15 Online:2020-05-10 Published:2020-05-15
Contact: XU Zhiyuan, born in 1993, M. S. candidate. His research interests include deep learning compression.
About author:WANG Zhongfeng, born in 1976, Ph. D., research fellow. His research interests include ubiquitous electric power internet of things.XU Zhiyuan, born in 1993, M. S. candidate. His research interests include deep learning compression.SONG Chunhe, born in 1981, Ph. D., research fellow. His research interests include edge computing.ZHANG Hongyu， born in 1976, M. S.， senior engineer. His research interests include power system and its automation.CAI Yingkai， born in 1979, M. S.， senior engineer. His research interests include power marketing informationization.

基于梯度的深度网络剪枝算法

王忠锋^1,2,3, 徐志远^1,2,3,4, 宋纯贺^1,2,3, 张宏宇⁵, 蔡颖凯⁵

1.机器人学国家重点实验室(中国科学院沈阳自动化研究所)，沈阳 110016
2.网络化控制系统重点实验室(中国科学院沈阳自动化研究所)，沈阳 110016
3.中国科学院机器人与智能制造创新研究院，沈阳 110169
4.中国科学院大学，北京 100049
5.国网辽宁省电力有限公司，沈阳 110016

通讯作者: 徐志远(1993—)
作者简介:王忠锋（1976—），男，辽宁沈阳人，研究员，博士，主要研究方向：泛在电力物联网；徐志远(1993—)，男，辽宁抚顺人，硕士研究生，主要研究方向：深度学习压缩；宋纯贺(1981—)，男，辽宁鞍山人，研究员，博士，主要研究方向：边缘计算；张宏宇（1976—），男，辽宁义县人，高级工程师，硕士，主要研究方向：电力系统及其自动化；蔡颖凯（1979—），男，辽宁鞍山人，高级工程师，硕士，主要研究方向：电力营销信息化。

Abstract

Abstract:

Deep neural network models usually have a large number of redundant weight parameters. Calculating the deep network model requires a large amount of computing resources and storage pace, which makes the deep network model difficult to be deployed on some edge devices and embedded devices. To resolve this issue, a Gradient-based Deep network Pruning (GDP) algorithm was proposed. The core idea of GDP algorithm was to use the gradient as the basis for judging the importance of each weight. To eliminate the weights corresponding to the gradients smaller than the threshold, an adaptive method was used to find the threshold to screen the weights. The deep network model was retrained after pruning to restore the network performance. The experimental results show that the GDP algorithm reduces the computational cost by 35.3 percentage points with a precision loss of only 0.14 percentage points on the CIFAR-10 dataset. Compared with the state-of-the-art PFEC （Pruning Filters for Efficient ConvNets） algorithm, the GDP algorithm increases the network model accuracy by 0.13 percentage points, and reduces the computational cost by 1.1 percentage points, indicating that the proposed algorithm has superior performance of deep network in terms of both compression and acceleration.

Key words: deep network, compression and acceleration, pruning, adaptive threshold, neural network

摘要：

深度神经网络模型通常存在大量冗余的权重参数，计算深度网络模型需要占用大量的计算资源和存储空间，导致深度网络模型难以部署在一些边缘设备和嵌入式设备上。针对这一问题，提出了一种基于梯度的深度网络剪枝(GDP)算法。GDP算法核心思想是以梯度作为评判权值重要性的依据。首先，通过自适应的方法找出阈值进行权值参数的筛选；然后，剔除那些小于阈值的梯度所对应的权值；最后，重新训练剪枝后的深度网络模型来恢复网络精度。实验结果表明：在CIFAR-10数据集上，GDP算法在精度仅下降0.14个百分点的情况下，计算量减少了35.3个百分点；与当前流行的PFEC算法相比，GDP算法使网络模型精度提高了0.13个百分点，计算量下降了1.1个百分点，具有更优越的深度网络压缩与加速性能。

关键词: 深度网络, 压缩与加速, 剪枝, 自适应阈值, 神经网络

CLC Number:

TP391.4

WANG Zhongfeng, XU Zhiyuan, SONG Chunhe, ZHANG Hongyu, CAI Yingkai. Gradient-based deep network pruning algorithm[J]. Journal of Computer Applications, 2020, 40(5): 1253-1259.

王忠锋, 徐志远, 宋纯贺, 张宏宇, 蔡颖凯. 基于梯度的深度网络剪枝算法[J]. 计算机应用, 2020, 40(5): 1253-1259.

References

1 KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[C]// Proceedings of the 2012 International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2012: 1097-1105.
2 WANG M , DENG W . Deep face recognition: a survey[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1804.06655.pdf.
3 HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 770-778.
4 SIMONYAN K , ZISSERMAN A . Two-stream convolutional networks for action recognition in videos[C]// Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2014: 568-576.
5 DENG L , YU D . Deep learning: methods and applications[J]. Foundations and Trends in Signal Processing, 2014, 7(3/4): 197-387.
6 SZEGEDY C , LIU W , JIA Y , et al . Going deeper with convolutions[EB/OL]. [2019-07-11].http://www.arxiv.org/pdf/1409.4842.pdf.
7 SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1409.1556.pdf.
8 XUE J , LI J , YU D , et al . Singular value decomposition based low-footprint speaker adaptation and personalization for deep neural network[C]// Proceedings of the 2014 IEEE International Conference on Acoustics, Speech and Signal Processing. Piscataway: IEEE, 2014: 6359-6363.
9 KIM Y D , PARK E , YOO S, et al . Compression of deep convolutional neural networks for fast and low power mobile applications[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1511.06530.pdf.
10 TAI C , XIAO T , ZHANG Y , et al . Convolutional neural networks with low-rank regularization[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1511.06067.pdf.
11 ZHU C , HAN S , MAO H , et al . Trained ternary quantization[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1612.01064.pdf.
12 ZHOU A , YAO A , GUO Y , et al . Incremental network quantization: Towards lossless CNNs with low-precision weights[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1702.03044.pdf.
13 ZHANG X , ZOU J , HE K , et al . Accelerating very deep convolutional networks for classification and detection[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(10):1943-1955.
14 KIM J , PARK S , KWAK N . Paraphrasing complex network: network compression via factor transfer[C]// Proceedings of the 32nd International Conference on Neural Information Processing Systems.Moutreal,Canada: Neural Information Processing Systems Foundation, Inc., 2018: 2760-2769.
15 ANIL R , PEREYRA G , PASSOS A , et al . Large scale distributed neural network training through online distillation[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1804.03235.pdf.
16 HE Y , KANG G , DONG X , et al . Soft filter pruning for accelerating deep convolutional neural networks[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1808.06866.pdf.
17 靳丽蕾, 杨文柱, 王思乐, 等 . 一种用于卷积神经网络压缩的混合剪枝方法[J]. 小型微型计算机系统, 2018, 39(12): 38-43. JIN L L ， YANG W Z ， WANG S L , et al .Mixed pruning method for convolutional neural network compression[J]. Journal of Chinese Computer Systems, 2018, 39(12): 38-43.
18 CHANDAKKAR P S , LI Y , DING P L K , et al . Strategies for re-training a pruned neural network in an edge computing paradigm[C]// Proceedings of the 2017 IEEE International Conference on Edge Computing. Piscataway: IEEE, 2017: 244-247.
19 HAN S , LIU X , MAO H , et al . EIE: efficient inference engine on compressed deep neural network[J]. ACM SIGARCH Computer Architecture News, 2016, 44(3): 243-254.
20 HAN S , KANG J , MAO H , et al . ESE: efficient speech recognition engine with sparse LSTM on FPGA[C]// Proceedings of the 2017 ACM/SIGDA International Symposium on Field-Programmable Gate Arrays. New York: ACM, 2017: 75-84.
21 WEN W , XU C , WU C , et al . Coordinating filters for faster deep neural networks[C]// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 658-666.
22 YU X , LIU T , WANG X , et al . On compressing deep models by low rank and sparse decomposition[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 67-76.
23 COURBARIAUX M , HUBARA I , SOUDRY D , et al . Binarized neural networks: training deep neural networks with weights and activations constrained to +1 or - 1 [EB/OL]. [2019-07-11].https://arxiv.org/pdf/1602.02830.pdf.
24 RASTEGARI M , ORDONEZ V , REDMON J , et al . XNOR-Net: ImageNet classification using binary convolutional neural networks[C]// Proceedings of the 2016 European Conference on Computer Vision, LNCS 9908. Cham: Springer, 2016: 525-542.
25 LI F , ZHANG B , LIU B . Ternary weight networks[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1605.04711.pdf.
26 HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network[EB/OL]. [2019-07-11].http://serve.3ezy.com/arxiv.org/pdf/1503.02531.
27 ROMERO A , BALLAS N , KAHOU S E , et al . FitNets: hints for thin deep nets[EB/OL]. [2019-07-11].http://www.algomagic.com/readings?reading=1412.6550.pdf.
28 KORATTIKARA A , RATHOD V , MURPHY K P , et al . Bayesian dark knowledge[C]// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2015: 3438-3446.
29 HAN S , POOL J , TRAN J , et al . Learning both weights and connections for efficient neural network[C]// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge: MIT Press, 2015: 1135-1143.
30 SUN Y , WANG X , TANG X . Sparsifying neural network connections for face recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 4856-4864.
31 HAN S , MAO H , DALLY W J . Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1510.00149.pdf.
32 IANDOLA F N , HAN S , MOSKEWICZ M W , et al . SqueezeNet: AlexNet-level accuracy with 50x fewer parameters and< 0.5 MB model size[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1602.07360.pdf.
33 SRIVASTAVA N , HINTON G , KRIZHEVSKY A , et al . Dropout: a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research, 2014, 15: 1929-1958.
34 IOFFE S , SZEGEDY C . Batch normalization: accelerating deep network training by reducing internal covariate shift[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1502.03167.pdf.
35 LI H , KADAV A , DURDANOVIC I , et al . Pruning filters for efficient ConvNets[EB/OL]. [2019-07-11].https://arxiv.org/pdf/1608.08710.pdf.
36 HE Y , LIU P , WANG Z , et al . Filter pruning via geometric median for deep convolutional neural networks acceleration[C]// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2019: 4335-4344.

[1]	SONG Zhongshan, LIANG Jiarui, ZHENG Lu, LIU Zhenyu, TIE Jun. Remote sensing scene classification based on bidirectional gated scale feature fusion [J]. Journal of Computer Applications, 2021, 41(9): 2726-2735.
[2]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[3]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[4]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[5]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[6]	MOU Changning, WANG Haipeng, ZHOU Piyu, HOU Xinhang. De novo peptide sequencing by tandem mass spectrometry based on graph convolutional neural network [J]. Journal of Computer Applications, 2021, 41(9): 2773-2779.
[7]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[8]	LIU Zichen, LI Xiaojuan, WEI Wei. Automatic patent price evaluation based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(9): 2532-2538.
[9]	DING Yin, SANG Nan, LI Xiaoyu, WU Feizhou. Prediction method of capacity data in telecom industry based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(8): 2373-2378.
[10]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[11]	HUANG Chengcheng, DONG Xiaoxiao, LI Zhao. Deep pipeline 5×5 convolution method based on two-dimensional Winograd algorithm [J]. Journal of Computer Applications, 2021, 41(8): 2258-2264.
[12]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[13]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[14]	LIU Huan, LI Xiaoge, HU Likun, HU Feixiong, WANG Penghua. Knowledge graph driven recommendation model of graph neural network [J]. Journal of Computer Applications, 2021, 41(7): 1865-1870.
[15]	WU Zeju, JIAO Cuijuan, CHEN Liang. Tire defect detection method based on improved Faster R-CNN [J]. Journal of Computer Applications, 2021, 41(7): 1939-1946.

Gradient-based deep network pruning algorithm

基于梯度的深度网络剪枝算法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics