结合剪枝与流合并的卷积神经网络加速压缩方法

doi:10.11772/j.issn.1001-9081.2019081363

计算机应用 ›› 2020, Vol. 40 ›› Issue (3): 621-625.DOI: 10.11772/j.issn.1001-9081.2019081363

• 人工智能 • 下一篇

结合剪枝与流合并的卷积神经网络加速压缩方法

谢斌红¹, 钟日新¹, 潘理虎^1,2, 张英俊¹

1. 太原科技大学计算机科学与技术学院, 太原 030024;
2. 中国科学院地理科学与资源研究所, 北京 100101

收稿日期:2019-08-06 修回日期:2019-10-10 出版日期:2020-03-10 发布日期:2019-10-31
通讯作者: 钟日新
作者简介:谢斌红(1972-),男,山西万荣人,副教授,硕士,主要研究方向:软件体系结构、服务计算;钟日新(1995-),男,山西朔州人,硕士研究生,主要研究方向:软件体系架构、深度学习;潘理虎(1974-),男,河南驻马店人,副教授,博士,主要研究方向:人工智能、软件工程;张英俊(1969-),男,山西河津人,高级工程师,硕士,主要研究方向:软件体系结构、智能软件。
基金资助:
山西省科技重大专项（20141101001）；山西省重点计划研发项目（201803D121048）。

Accelerated compression method for convolutional neural network combining with pruning and stream merging

XIE Binhong¹, ZHONG Rixin¹, PAN Lihu^1,2, ZHANG Yingjun¹

1. Department of Computer Science and Technology, Taiyuan University of Science and Technology, Taiyuan Shanxi 030024, China;
2. Institute of Geographic Sciences and Natural Resources Research, Chinese Academy of Sciences, Beijing 100101 China

Received:2019-08-06 Revised:2019-10-10 Online:2020-03-10 Published:2019-10-31
Supported by:
This work is partially supported by the Science and Technology Major Project of Shanxi Province (20141101001), the Key Research and Development Project of Shanxi Province (201803D121048).

摘要/Abstract

摘要： 深度卷积神经网络因规模庞大、计算复杂而限制了其在实时要求高和资源受限环境下的应用，因此有必要对卷积神经网络现有的结构进行优化压缩和加速。为了解决这一问题，提出了一种结合剪枝、流合并的混合压缩方法。该方法通过不同角度去压缩模型，进一步降低了参数冗余和结构冗余所带来的内存消耗和时间消耗。首先，从模型的内部将每层中冗余的参数剪去；然后，从模型的结构上将非必要的层与重要的层进行流合并；最后，通过重新训练来恢复模型的精度。在MNIST数据集上的实验结果表明，提出的混合压缩方法在不降低模型精度前提下，将LeNet-5压缩到原来的1/20，运行速度提升了8倍。

关键词: 卷积神经网络, 模型压缩, 网络剪枝, 流合并, 冗余

Abstract: Deep convolutional neural networks are generally large in scale and complex in computation, which limits their application in high real-time and resource-constrained environments. Therefore, it is necessary to optimize the compression and acceleration of the existing structures of convolutional neural networks. In order to solve this problem, a hybrid compression method combining pruning and stream merging was proposed. In the method， the model was decompressed through different angles, further reducing the memory consumption and time consumption caused by parameter redundancy and structural redundancy. Firstly, the redundant parameters in each layer were cut off from the inside of the model. Then the non-essential layers were merged with the important layers from the structure of the model. Finally, the accuracy of the model was restored by retraining. The experimental results on the MNIST dataset show that the proposed hybrid compression method compresses LeNet-5 to 1/20 and improves its running speed by 8 times without reducing the accuracy of the model.

Key words: Convolutional Neural Network (CNN), model compression, network pruning, stream merging, redundancy

中图分类号:

TP183

谢斌红, 钟日新, 潘理虎, 张英俊. 结合剪枝与流合并的卷积神经网络加速压缩方法[J]. 计算机应用, 2020, 40(3): 621-625.

XIE Binhong, ZHONG Rixin, PAN Lihu, ZHANG Yingjun. Accelerated compression method for convolutional neural network combining with pruning and stream merging[J]. Journal of Computer Applications, 2020, 40(3): 621-625.

参考文献

[1] LECUN Y A,BOTTOU L,ORR G B,et al. Efficient backprop[M]//ORR G B,MÜLLER K R. Neural Networks:Tricks of the Trade,LNCS 7700. Berlin:Springer,2012:9-48.
[2] HASSIBI B,STORK D G. Second order derivatives for network pruning:optimal brain surgeon[C]//Proceedings of the 5th International Conference on Neural Information Processing Systems. San Francisco:Morgan Kaufmann Publishers Inc.,1992:164-171.
[3] SRINIVAS S,BADU R V. Learning neural network architectures using backpropagation[C]//Proceedings of the 2016 British Machine Vision Conference. Durham:BMVA Press,2016:No. 104.
[4] WEN W,WU C,WANG Y,et al. Learning structured sparsity in deep neural networks[C]//Proceedings of the 30th International Conference on Neural Information Processing System. New York:Curran Associates Inc.,2016:2082-2090.
[5] LI H,KADAV A,DURDANOVIC I,et al. Pruning filters for efficient convnets[EB/OL].[2019-06-20]. https://arxiv.org/pdf/1608.08710.pdf.
[6] GUPTA S,AGRAWAL A,GOPALAKRISHNAN K,et al. Deep learning with limited numerical precision[C]//Proceedings of the 32nd International Conference on Machine Learning. New York:JMLR.org,2015:1737-1746.
[7] GYSEL P,MOTAMEDI M,GHIASI S. Hardware-oriented approximation of convolutional neural networks[EB/OL].[2019-06-20]. https://arxiv.org/pdf/1604.03168.pdf.
[8] DENIL M,SHAKIBI B,DINH L,et al. Predicting parameters in deep learning[C]//Proceedings of the 26th International Conference on Neural Information Processing Systems. New York:Curran Associates Inc.,2013:2148-2516.
[9] JADERBERG M,VEDALI A,ZISSERMAN A. Speeding up convolutional neural networks with low rank expansions[C]//Proceedings of the 2014 British Machine Vision Conference. Durham:BMVA Press,2014:No. 73.
[10] DENTON E,ZAZREMBA W,BRUNA J,et al. Exploiting linear structure within convolutional networks for efficient evaluation[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2014:1269-1277.
[11] BUCILUǍ C,CARUANA R,NICULESCU-MIZIL A. Model compression[C]//Proceedings of the 12th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM,2006:535-541.
[12] HINTON G,VINYALS O,DEAN J. Distilling the knowledge in a neural network[EB/OL].[2019-06-20]. https://arxiv.org/pdf/1503.02531.pdf.
[13] HUANG Z,WANG N. Like what you like:knowledge distill via neuron selectivity transfer[EB/OL].[2019-06-20]. https://arxiv.org/pdf/1707.01219.pdf.
[14] LI D,WANG X,KONG D. DeepRebirth:accelerating deep neural network execution on mobile devices[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto,CA:AAAI Press,2018:2322-2330.
[15] LECUN Y,BOTTOU L,BENGIO Y,et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998,86(11):2278-2324.
[16] 黄文坚, 唐源. TesnsorFlow实战[M]. 北京:电子工业出版社, 2017:233-242. (HUANG W J,TANG Y. Actual Combat of TensorFlow[M]. Beijing:Publishing House of Electronics Industry, 2017:233-242.)
[17] HAN S,POOL J,TRAN J,et al. Learning both weights and connections for efficient neural networks[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2015:1135-1143.
[18] LIU Z,LI J,SHEN Z,et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of the 2017 International Conference on Computer Vision. Piscataway:IEEE, 2017:2755-2763.
[19] 靳丽蕾, 杨文柱, 王思乐, 等. 一种用于卷积神经网络压缩的混合剪枝方法[J]. 小型微型计算机系统,2018,39(12):2596-2601. (JIN L L,YANG W Z,WANG S L. et al. Mixed pruning method for convolution neural network compression[J]. Journal of Chinese Computer Systems,2018,39(12):2596-2601.
[20] FRANKLE J,CARBIN M. The lottery ticket hypothesis:finding sparse trainable neural networks[EB/OL].[2019-06-20]. https://arxiv.org/pdf/1803.03635.pdf.

结合剪枝与流合并的卷积神经网络加速压缩方法

Accelerated compression method for convolutional neural network combining with pruning and stream merging

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	王贺兵, 张春梅. 基于非对称卷积-压缩激发-次代残差网络的人脸关键点检测[J]. 计算机应用, 2021, 41(9): 2741-2747.
[2]	宋中山, 梁家锐, 郑禄, 刘振宇, 帖军. 基于双向门控尺度特征融合的遥感场景分类[J]. 计算机应用, 2021, 41(9): 2726-2735.
[3]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[4]	张永斌, 常文欣, 孙连山, 张航. 基于字典的域名生成算法生成域名的检测方法[J]. 计算机应用, 2021, 41(9): 2609-2614.
[5]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[6]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[7]	牟长宁, 王海鹏, 周丕宇, 侯鑫行. 基于图卷积神经网络的串联质谱从头测序[J]. 计算机应用, 2021, 41(9): 2773-2779.
[8]	张师鹏, 李永忠, 杜祥通. 基于半监督学习和三支决策的入侵检测模型[J]. 计算机应用, 2021, 41(9): 2602-2608.
[9]	曾祥银, 郑伯川, 刘丹. 基于深度卷积神经网络和聚类的左右轨道线检测[J]. 计算机应用, 2021, 41(8): 2324-2329.
[10]	黄继爽, 张华, 李永龙, 赵皓, 王皓冉, 冯春成. 基于动态特征蒸馏的水工隧洞缺陷识别方法[J]. 计算机应用, 2021, 41(8): 2358-2365.
[11]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 计算机应用, 2021, 41(8): 2273-2287.
[12]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[13]	黄程程, 董霄霄, 李钊. 基于二维Winograd算法的深流水线5×5卷积方法[J]. 计算机应用, 2021, 41(8): 2258-2264.
[14]	武光利, 李雷霆, 郭振洲, 王成祥. 基于改进的双向长短期记忆网络的视频摘要生成模型[J]. 计算机应用, 2021, 41(7): 1908-1914.
[15]	卿欣艺, 陈玉玲, 周正强, 涂园超, 李涛. 基于中国剩余定理的区块链存储扩展模型[J]. 计算机应用, 2021, 41(7): 1977-1982.