基于联合动态剪枝的深度神经网络压缩算法

doi:10.11772/j.issn.1001-9081.2020121914

计算机应用 ›› 2021, Vol. 41 ›› Issue (6): 1589-1596.DOI: 10.11772/j.issn.1001-9081.2020121914

所属专题： 2020年全国开放式分布与并行计算学术年会（DPCS 2020）

• 2020年全国开放式分布与并行计算学术年会（DPCS 2020） • 上一篇下一篇

基于联合动态剪枝的深度神经网络压缩算法

张明明¹, 卢庆宁², 李文中², 宋浒¹

1. 国网江苏省电力有限公司信息通信分公司, 南京 210024;
2. 计算机软件新技术国家重点实验室(南京大学), 南京 210023

收稿日期:2020-11-04 修回日期:2021-03-29 出版日期:2021-06-10 发布日期:2021-06-21
通讯作者: 卢庆宁
作者简介:张明明(1974-),男,江苏常州人,高级工程师,硕士,主要研究方向:深度学习、卷积神经网络模型加速;卢庆宁(1997-),男,江苏南京人,硕士研究生,主要研究方向:机器学习、模型压缩、异常检测;李文中(1979-),男,广西平南人,教授,博士生导师,博士,CCF会员,主要研究方向:分布式计算、深度学习;宋浒(1986-),男,安徽合肥人,高级工程师,博士,主要研究方向:机器学习、大数据分析。
基金资助:
国网江苏省电力有限公司科技项目（J2020069）。

Deep neural network compression algorithm based on combined dynamic pruning

ZHANG Mingming¹, LU Qingning², LI Wenzhong², SONG Hu¹

1. Information and Communication Branch, State Grid Jiangsu Electric Power Company Limited, Nanjing Jiangsu 210024, China;
2. State Key Laboratory for Novel Software Technology(Nanjing University), Nanjing Jiangsu 210023, China

Received:2020-11-04 Revised:2021-03-29 Online:2021-06-10 Published:2021-06-21
Supported by:
This work is partially supported by the Science and Technology Project of State Grid Jiangsu Electric Power Company Limited (J2020069).

摘要/Abstract

摘要： 作为模型压缩的一个分支，网络剪枝算法通过移除深度神经网络中不重要的参数来降低计算消耗；然而，永久性的剪枝会导致模型容量不可逆转的损失。针对该问题，提出了一种联合动态剪枝的算法来综合分析卷积核与输入图像的特征。一方面，将部分卷积核置零，并允许其在训练过程中更新，直到网络收敛之后再永久性移除被置零的卷积核。另一方面，采样输入图像的特征，然后利用通道重要性预测网络对这些特征进行分析，从而确定卷积运算中可以跳过的通道。基于M-CifarNet与VGG16的实验结果表明，联合动态剪枝分别取得了2.11和1.99的浮点运算压缩比，而与基准模型（M-CifarNet、VGG16）相比准确率仅分别下降不到0.8个百分点和1.2个百分点。相较于现有的网络剪枝算法，联合动态剪枝有效地减少了模型的浮点运算次数（FLOPs）以及参数规模，在同样的压缩比下获得了更高的准确率。

关键词: 模型压缩, 网络剪枝, 动态剪枝, 深度神经网络, 卷积核

Abstract: As a branch of model compression, network pruning algorithm reduces the computational cost by removing unimportant parameters in the deep neural network. However, permanent pruning will cause irreversible loss of the model capacity. Focusing on this issue, a combined dynamic pruning algorithm was proposed to comprehensively analyze the characteristics of the convolution kernel and the input image. Part of the convolution kernels were zeroized and allowed to be updated during the training process until the network converged, thereafter the zeroized kernels would be permanently removed. At the same time, the input images were sampled to extract their features, then a channel importance prediction network was used to analyze these features to determine the channels able to be skipped during the convolution operation. Experimental results based on M-CifarNet and VGG16 show that the combined dynamic pruning can respectively provide 2.11 and 1.99 floating-point operation compression ratios, with less than 0.8 percentage points and 1.2 percentage points accuracy loss respectively compared to the benchmark model (M-CifarNet、VGG16). Compared with the existing network pruning algorithms, the combined dynamic pruning algorithm effectively reduces the Floating-Point Operations Per second (FLOPs) and the parameter scale of the model, and achieves the higher accuracy under the same compression ratio.

Key words: model compression, network pruning, dynamic pruning, Deep Neural Network (DNN), convolution kernel

中图分类号:

TP183

张明明, 卢庆宁, 李文中, 宋浒. 基于联合动态剪枝的深度神经网络压缩算法[J]. 计算机应用, 2021, 41(6): 1589-1596.

ZHANG Mingming, LU Qingning, LI Wenzhong, SONG Hu. Deep neural network compression algorithm based on combined dynamic pruning[J]. Journal of Computer Applications, 2021, 41(6): 1589-1596.

参考文献

[1] SIMONYAN K,ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2020-10-24]. https://arxiv.org/pdf/1409.1556.pdf.
[2] DENIL M,SHAKIBI B,DINH L,et al. Predicting parameters in deep learning[C]//Proceedings of the 2013 26th International Conference on Neural Information Processing Systems. Red Hook:Curran Associates Inc.,2013:2148-2156.
[3] 雷杰, 高鑫, 宋杰, 等. 深度网络模型压缩综述[J]. 软件学报, 2018, 29(2):251-266.(LEI J,GAO X,SONG J,et al. Survey of deep neural network model compression[J]. Journal of Software,2018,29(2):251-266.)
[4] 李江昀, 赵义凯, 薛卓尔, 等. 深度神经网络模型压缩综述[J]. 工程科学学报, 2019, 41(10):1229-1239.(LI J Y,ZHAO Y K, XUE Z E,et al. A survey of model compression for deep neural networks[J]. Chinese Journal of Engineering,2019,41(10):1229-1239.)
[5] 林景栋, 吴欣怡, 柴毅, 等. 卷积神经网络结构优化综述[J]. 自动化学报, 2020, 46(1):24-37.(LIN J D,WU X Y,CHAI Y,et al. Structure optimization of convolutional neural networks:a survey[J]. Acta Automatica Sinica,2020,46(1):24-37.)
[6] HAN S,MAO H,DALLY W J. Deep compression:compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL].[2020-10-19]. https://arxiv.org/pdf/1510.00149.pdf.
[7] 巩凯强, 张春梅, 曾光华. 卷积神经网络模型剪枝结合张量分解压缩方法[J]. 计算机应用, 2020, 40(11):3146-3151.(GONG K Q,ZHANG C M,ZENG G H. Convolution neural network model compression method based on pruning and tensor decomposition[J]. Journal of Computer Applications,2020,40(11):3146-3151.)
[8] 王忠锋, 徐志远, 宋纯贺, 等. 基于梯度的深度网络剪枝算法[J]. 计算机应用, 2020, 40(5):1253-1259.(WANG Z F,XU Z Y, SONG C H,et al. Gradient-based deep network pruning algorithm[J]. Journal of Computer Applications,2020,40(5):1253-1259.)
[9] LI H,KADAV A,DURDANOVIC I,et al. Pruning filters for efficient ConvNets[EB/OL].[2020-10-21]. https://arxiv.org/pdf/1608.08710.pdf.
[10] LIU Z,LI J,SHEN Z,et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2755-2763.
[11] HE Y, KANG G, DONG X, et al. Soft filter pruning for accelerating deep convolutional neural networks[C]//Proceedings of the 2018 27th International Joint Conference on Artificial Intelligence. Menlo Park:AAAI Press,2018:2234-2240.
[12] GUO Y W,YAO A B,CHEN Y R. Dynamic network surgery for efficient DNNs[EB/OL].[2020-11-01]. https://arxiv.org/pdf/1608.04493.pdf.
[13] SRIVASTAVA N, HINTON G, KRIZHEVSKY A, et al. Dropout:a simple way to prevent neural networks from overfitting[J]. Journal of Machine Learning Research,2014,15:1929-1958.
[14] GAO X T,ZHAO Y R,DUDZIAK Ł,et al. Dynamic channel pruning:feature boosting and suppression[EB/OL].[2020-10-03]. https://arxiv.org/pdf/1810.05331.pdf.
[15] HUA W Z,ZHOU Y,DE SA C,et al. Channel gating neural networks[EB/OL].[2020-10-07]. https://arxiv.org/pdf/1805.12549.pdf.
[16] IOFFE S,SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[EB/OL].[2020-10-23]. https://arxiv.org/pdf/1502.03167.pdf.
[17] GLOROT X,BORDES A,BENGIO Y. Deep sparse rectifier neural networks[C]//Proceedings of the 2011 14th International Conference on Artificial Intelligence and Statistics. New York:JMLR. org,2011:315-323.
[18] ZHOU B L,KHOSLA A,LAPEDRIZA A,et al. Learning deep features for discriminative localization[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:2921-2929.
[19] KRIZHEVSKY A,NAIR V,HINTON G. CIFAR-10 and CIFAR-100 datasets[DS/OL].[2020-10-28]. http://www.cs.toronto.edu/~kriz/cifar.html.
[20] MOLCHANOV P, TYREE S, KARRAS T, et al. Pruning convolutional neural networks for resource efficient inference[EB/OL].[2020-11-02]. https://arxiv.org/pdf/1611.06440.pdf.
[21] ZHAO Y R,GAO X T,MULLINS R,et al. Mayo:a framework for auto-generating hardware friendly deep neural networks[C]//Proceedings of the 20182nd International Workshop on Embedded and Mobile Deep Learning. New York:ACM,2018:25-30.

基于联合动态剪枝的深度神经网络压缩算法

Deep neural network compression algorithm based on combined dynamic pruning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	黄继爽, 张华, 李永龙, 赵皓, 王皓冉, 冯春成. 基于动态特征蒸馏的水工隧洞缺陷识别方法[J]. 计算机应用, 2021, 41(8): 2358-2365.
[2]	王曙燕, 侯则昱, 孙家泽. 面向深度学习的对抗样本差异性检测方法[J]. 计算机应用, 2021, 41(7): 1849-1856.
[3]	张文烨, 尚方信, 郭浩. 基于Octave卷积的混合精度神经网络量化方法[J]. 计算机应用, 2021, 41(5): 1299-1304.
[4]	郭文旭, 苏远歧, 刘跃虎. 基于ZYNQ平台的YOLOv3压缩和加速[J]. 计算机应用, 2021, 41(3): 669-676.
[5]	杨丽, 王时绘, 朱博. 基于动态和静态偏好的兴趣点推荐算法[J]. 计算机应用, 2021, 41(2): 398-406.
[6]	陈彦如, 张涂静娃, 杜千, 冉茂亮, 王红军. 基于深度森林的高铁站室内热舒适度等级预测[J]. 计算机应用, 2021, 41(1): 258-264.
[7]	李鸣, 郭晨皓, 陈星. 视觉类深度神经网络的自动标注[J]. 计算机应用, 2020, 40(6): 1593-1600.
[8]	陈程军, 毛莺池, 王绎超. 基于激活-熵的分层迭代剪枝策略的CNN模型压缩[J]. 计算机应用, 2020, 40(5): 1260-1265.
[9]	韦伟, 李小娟. 基于相似论文增广的深度学习专利质量评估[J]. 计算机应用, 2020, 40(4): 966-971.
[10]	谢斌红, 钟日新, 潘理虎, 张英俊. 结合剪枝与流合并的卷积神经网络加速压缩方法[J]. 计算机应用, 2020, 40(3): 621-625.
[11]	樊琦, 李卓, 陈昕. 基于边缘计算的分支神经网络模型推断延迟优化[J]. 计算机应用, 2020, 40(2): 342-346.
[12]	邓凯, 黄佳进, 秦进. 基于物品的统一推荐模型[J]. 计算机应用, 2020, 40(2): 530-534.
[13]	杨坚伟, 严群, 姚剑敏, 林志贤. 基于深度神经网络的移动端人像分割[J]. 计算机应用, 2020, 40(12): 3644-3650.
[14]	杨磊, 赵红东. 基于轻量级深度神经网络的环境声音识别[J]. 计算机应用, 2020, 40(11): 3172-3177.
[15]	朱倩倩, 刘渊, 李甫. 深度神经网络的仿生矩阵约简与量化方法[J]. 计算机应用, 2020, 40(10): 2817-2821.