卷积神经网络模型剪枝结合张量分解压缩方法

doi:10.11772/j.issn.1001-9081.2020030362

计算机应用 ›› 2020, Vol. 40 ›› Issue (11): 3146-3151.DOI: 10.11772/j.issn.1001-9081.2020030362

卷积神经网络模型剪枝结合张量分解压缩方法

巩凯强, 张春梅, 曾光华

北方民族大学计算机科学与工程学院, 银川 750021

收稿日期:2020-03-26 修回日期:2020-06-01 出版日期:2020-11-10 发布日期:2020-07-17
通讯作者: 巩凯强(1994-),男,甘肃天水人,硕士研究生,CCF会员,主要研究方向:计算机视觉、模型压缩;kq192011@sina.com
作者简介:张春梅(1964-),女,宁夏银川人,教授,硕士,CCF会员,主要研究方向:计算机视觉、模式识别;曾光华(1992-),男,贵州铜仁人,硕士,主要研究方向:图像处理、嵌入式系统
基金资助:
北方民族大学研究生创新项目（YCX19063）。

Convolution neural network model compression method based on pruning and tensor decomposition

GONG Kaiqiang, ZHANG Chunmei, ZENG Guanghua

College of Computer Science and Engineering, North Minzu University, Yinchuan Ningxia 750021, China

Received:2020-03-26 Revised:2020-06-01 Online:2020-11-10 Published:2020-07-17
Supported by:
This work is partially supported by North Minzu University Graduate Innovation Project (YCX19063).

摘要/Abstract

摘要： 针对卷积神经网络（CNN）拥有巨大的参数量及计算量，限制了其在嵌入式系统等资源受限设备上应用的问题，提出了基于统计量的网络剪枝结合张量分解的神经网络压缩方法，其核心思想是以均值和方差作为评判权值贡献度的依据。首先，以Lenet5为剪枝模型，网络各卷积层的均值和方差分布以聚类方式分离出提取特征较弱的滤波器，而使用保留的滤波器重构下一层卷积层；然后，将剪枝方法结合张量分解对更快的区域卷积神经网络（Faster RCNN）进行压缩，低维卷积层采取剪枝方法，而高维卷积层被分解为三个级联卷积层；最后，将压缩后的模型进行微调，使其在训练集上重新达到收敛状态。在PASCAL VOC测试集上的实验结果表明，所提方法降低了Faster RCNN模型54%的存储空间而精确率仅下降了0.58%，同时在树莓派4B系统上达到1.4倍的前向计算加速，有助于深度CNN模型在资源受限的嵌入式设备上的部署。

关键词: 卷积神经网络, 目标检测, 更快的区域卷积神经网络, 剪枝, 张量分解

Abstract: Focused on the problem that the huge number of parameters and calculations of Convolutional Neural Network (CNN) limit the application of CNN on resource-constrained devices such as embedded systems, a neural network compression method of statistics based network pruning and tensor decomposition was proposed. The core idea was to use the mean and variance as the basis for evaluating the weight contribution. Firstly, Lenet5 was used as a pruning model, the mean and variance distribution of each convolutional layer of the network were clustered to separate filters with weaker extracted features, and the retained filters were used to reconstruct the next convolutional layer. Secondly, the pruning method was combined with tensor decomposition to compress the Faster Region with Convolutional Neural Network (Faster RCNN). The pruning method was adopted for the low-dimensional convolution layers, and the high-dimensional convolutional layers were decomposed into three cascaded convolutional layers. Finally, the compressed model was fine-tuned, making the model be at the convergence state once again on the training set. Experimental results on the PASCAL VOC test set show that the proposed method reduces the storage space of the Faster RCNN model by 54% while the decrease of the accuracy is only 0.58%, at the same time, the method can reach 1.4 times acceleration of forward computing on the Raspberry Pi 4B system, which helpful for the deployment of deep CNN models on resource-constrained embedded devices.

Key words: Convolutional Neural Network (CNN), object detection, Faster Region with Convolutional Neural Network (Faster RCNN), pruning, tensor decomposition

中图分类号:

TP183

巩凯强, 张春梅, 曾光华. 卷积神经网络模型剪枝结合张量分解压缩方法[J]. 计算机应用, 2020, 40(11): 3146-3151.

GONG Kaiqiang, ZHANG Chunmei, ZENG Guanghua. Convolution neural network model compression method based on pruning and tensor decomposition[J]. Journal of Computer Applications, 2020, 40(11): 3146-3151.

参考文献

[1] RUMELHART D E,HINTON G E,WILLIAMS R J. Learning representations by back-propagating errors[J]. Nature,1986,323(6088):533-536.
[2] REED R. Pruning algorithms-a survey[J]. IEEE Transactions on Neural Networks,1993,4(5):740-747.
[3] MOZER M C,SMOLENSKY P. Skeletonization:a technique for trimming the fat from a network via relevance assessment[C]//Proceedings of the 1st International Conference on Neural Information Processing Systems. San Francisco, CA:Morgan Kaufmann Publishers Inc.,1988:107-115.
[4] LECUN Y,DENKER J S,SOLLA S A. Optimal brain damage[C]//Proceedings of the 2nd International Conference on Neural Information Processing Systems. San Francisco, CA:Morgan Kaufmann Publishers Inc.,1989:598-605.
[5] JADERBERG M, VEDALDI A, ZISSERMAN A. Speeding-up convolutional neural networks with low rank expansions[EB/OL].[2019-12-04]. https://arxiv.org/pdf/1405.3866.pdf.
[6] ZHANG X, ZOU J, MING X, et al. Efficient and accurate approximations of nonlinear convolutional networks[C]//Proceedings of the 2015 IEEE Conference on Computer Vison and Pattern Recognition. Piscataway:IEEE,2015:1984-1992.
[7] LIU B,WANG M,FOROOSH H,et al. Sparse convolutional neural networks[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2015:806-814.
[8] HAN S,POOL J,TRAN J,et al. Learning both weights and connections for efficient neural network[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2015:1135-1143.
[9] LEBEDEV V,LEMPITSKY V. Fast ConvNets using group-wise brain damage[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:2554-2564.
[10] YUAN M,LIN Y. Model selection and estimation in regression with grouped variables[J]. Journal of the Royal Statistical Society Series B(Statistical Methodology),2006,68(1):49-67.
[11] MOLCHANOV P, TYREE S, KARRAS T, et al. Pruning convolutional neural networks for resource efficient inference[EB/OL].[2019-06-30]. https://arxiv.org/pdf/1611.06440.pdf.
[12] LI H,KADAV A,DURDANOVIC I,et al. Pruning filters for efficient ConvNets[EB/OL].[2019-06-30]. https://arxiv.org/pdf/1608.08710.pdf.
[13] HE Y,LIN J,LIU Z,et al. AWC:autoML for modelcompression and acceleration on mobile devices[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11211. Cham:Springer,2018:815-832.
[14] DENTON E,ZAREMBA W,BRUNA J,et al. Exploiting linear structure within convolutional networks for efficient evaluation[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:1269-1277.
[15] LEBEDEV V,GANIN Y,RAKHUBA M,et al. Speeding-up convolutional neural networks using fine-tuned CP-decomposition[EB/OL].[2019-07-11]. https://arxiv.org/pdf/1412.6553.pdf.
[16] TAI C,XIAO T,ZHANG Y,et al. Convolutional neural networks with low-rank regularization[EB/OL].[2019-07-11]. https://arxiv.org/pdf/1511.06067.pdf.
[17] LIU Z,LI J,SHEN Z,et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2755-2763.
[18] 李小伟. 轻量级深度学习目标检测算法研究及系统设计[D]. 合肥:安徽大学,2019:20-58.(LI X W. Algorithm research and system design of lightweight deep learning object detection[D]. Hefei:Anhui University,2019:20-58.)
[19] MITTAL D,BHARDWAJ S,KHAPRA M M,et al. Recovering from random pruning:on the plasticity of deep convolutional neural networks[EB/OL].[2019-09-12]. https://arxiv.org/pdf/1801.10447.pdf.
[20] 吴进, 吴汉宁, 刘安, 等. 一种基于Lasso回归与SVD融合的深度学习模型压缩方法[J]. 电讯技术,2019,59(5):495-500. (WU J, WU H N, LIU A, et al. A deep learning modelcompression method based on Lasso regression and SVD fusion[J]. Telecommunication Engineering,2019,59(5):495-500.)
[21] DENIL M,SHAKIBI B,DINH L,et al. Predicting parameters in deep learning[C]//Proceedings of the 26th International Conference on Neural Information Proceedings Systems. Red Hook,NY:Curran Associates Inc.,2013:2148-2156.

卷积神经网络模型剪枝结合张量分解压缩方法

Convolution neural network model compression method based on pruning and tensor decomposition

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	马佳良, 陈斌, 孙晓飞. 基于改进的Faster R-CNN的通用目标检测框架[J]. 计算机应用, 2021, 41(9): 2712-2719.
[2]	宋中山, 梁家锐, 郑禄, 刘振宇, 帖军. 基于双向门控尺度特征融合的遥感场景分类[J]. 计算机应用, 2021, 41(9): 2726-2735.
[3]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[4]	张永斌, 常文欣, 孙连山, 张航. 基于字典的域名生成算法生成域名的检测方法[J]. 计算机应用, 2021, 41(9): 2609-2614.
[5]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[6]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[7]	牟长宁, 王海鹏, 周丕宇, 侯鑫行. 基于图卷积神经网络的串联质谱从头测序[J]. 计算机应用, 2021, 41(9): 2773-2779.
[8]	王贺兵, 张春梅. 基于非对称卷积-压缩激发-次代残差网络的人脸关键点检测[J]. 计算机应用, 2021, 41(9): 2741-2747.
[9]	陈静, 毛莺池, 陈豪, 王龙宝, 王子成. 基于改进单点多盒检测器的大坝缺陷目标检测方法[J]. 计算机应用, 2021, 41(8): 2366-2372.
[10]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 计算机应用, 2021, 41(8): 2273-2287.
[11]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[12]	黄程程, 董霄霄, 李钊. 基于二维Winograd算法的深流水线5×5卷积方法[J]. 计算机应用, 2021, 41(8): 2258-2264.
[13]	樊玮, 李晨炫, 邢艳, 黄睿, 彭洪健. 航空发动机损伤图像的二分类到多分类递进式检测网络[J]. 计算机应用, 2021, 41(8): 2352-2357.
[14]	曾祥银, 郑伯川, 刘丹. 基于深度卷积神经网络和聚类的左右轨道线检测[J]. 计算机应用, 2021, 41(8): 2324-2329.
[15]	吴则举, 焦翠娟, 陈亮. 基于改进Faster R-CNN的轮胎缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1939-1946.