基于激活-熵的分层迭代剪枝策略的CNN模型压缩

doi:10.11772/j.issn.1001-9081.2019111977

摘要
图/表
参考文献(31)
相关文章 (15)

全文: PDF (718 KB) HTML
输出: BibTeX | EndNote (RIS)

摘要

针对卷积神经网络(CNN)模型现有剪枝策略各尽不同和效果一般的情况，提出了基于激活-熵的分层迭代剪枝(AE-LIP)策略，保证模型精度在可控范围内的同时缩减模型的参数量。首先，结合神经元激活值和信息熵，构建基于激活-熵的权重评判准则，计算权值重要性得分；然后，逐层剪枝，根据重要性得分对权值排序，并结合各层剪枝数量筛选出待剪枝权重并将其设置为0；最后，微调模型，重复上述过程，直至迭代结束。实验结果表明，采用基于激活-熵的分层迭代剪枝策略:AlexNet模型压缩了87.5%;相应的准确率下降了2.12个百分点，比采用基于幅度的权重剪枝策略提高了1.54个百分点，比采用基于相关性的权重剪枝策略提高0.91个百分点。VGG-16模型压缩了84.1%；相应的准确率下降了2.62个百分点，比采用上述两个对比策略分别提高了0.62个百分点和0.27个百分点。说明所提策略在保证模型精确度下有效缩减了CNN模型的大小，有助于CNN模型在存储受限的移动设备上的部署。

	服务

	把本文推荐给朋友
	加入我的书架
	加入引用管理器
	E-mail Alert
	RSS
	作者相关文章
	陈程军
	毛莺池
	王绎超

关键词 ：移动云计算, 神经元激活值, 信息熵, 迭代剪枝, 模型压缩

Abstract：

Since the existing pruning strategies of the Convolutional Neural Network （CNN） model are different and have general effects, an Activation-Entropy based Layer-wise Iterative Pruning (AE-LIP) strategy was proposed to reduce the parameter amount of the model while ensuring the accuracy of the model within a controllable range. Firstly, combined with the neuronal activation value and information entropy, a weight evaluation criteria based on activation-entropy was constructed, and the weight importance score was calculated. Secondly, the pruning was performed layer by layer, the weights were sorted according to the importance score, and the pruning number in each layer was combined to filter out the weights to be pruned and set them to zero. Finally, the model was fine-tuned, and the above process was repeated until the iteration ended. The experimental results show that the activation-entropy based layer-wise iterative pruning strategy makes the AlexNet model compressed 87.5%, and the corresponding accuracy is reduced by 2.12 percentage points, which is 1.54 percentage points higher than that of the magnitude-based weight pruning strategy and 0.91 percentage points higher than that of the correlation-based weight pruning strategy; the strategy makes VGG-16 model compressed 84.1%, and the corresponding accuracy is reduced by 2.62 percentage points, which is 0.62 and 0.27 percentage points higher than those of the two above strategies. It can be seen that the proposed strategy reduces the size of the CNN model effectively while ensuring the accuracy of the model, and is helpful for the deployment of CNN model on mobile devices with limited storage.

Key words： mobile cloud computing neuronal activation value information entropy iterative pruning model compression

收稿日期: 2019-11-21

中图分类号:

TP389.1

基金资助:

“十三五”国家重点研发计划项目（2018YFC0407105）；华能集团重点研发项目（HNKJ17-21）。

通讯作者: 毛莺池(1976—) E-mail: yingchimao@hhu.edu.cn

作者简介: 陈程军(1996—)，男，江苏南通人，硕士研究生，CCF会员，主要研究方向：人工智能、分布式数据处理；毛莺池(1976—)，女，上海人，教授，博士，CCF会员，主要研究方向：物联网、分布式数据处理；王绎超(1994—)，男，山西介休人，硕士，主要研究方向：分布式数据处理。

引用本文:

陈程军, 毛莺池, 王绎超. 基于激活-熵的分层迭代剪枝策略的CNN模型压缩[J]. 计算机应用, 2020, 40(5): 1260-1265. CHEN Chengjun, MAO Yingchi, WANG Yichao. CNN model compression based on activation-entropy based layer-wise iterative pruning strategy. Journal of Computer Applications, 2020, 40(5): 1260-1265.

链接本文:

http://www.joca.cn/CN/10.11772/j.issn.1001-9081.2019111977 或 http://www.joca.cn/CN/Y2020/V40/I5/1260

1 KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[C]// Proceedings of the 25th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2012: 1097-1105.
2 SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1409.1556.pdf.
3 SZEGEDY C , LIU W , JIA Y , et al . Going deeper with convolutions[C]// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 1-9.
4 彭冬亮,王天兴 . 基于GoogLeNet模型的剪枝算法[J]. 控制与决策, 2019, 34(6):1259-1264. (PENG D L, WANG T X. Pruning algorithm based on GoogLeNet model[J]. Control and Decision, 2019, 34(6):1259-1264.)
5 HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016:770-778.
6 HUANG G , LIU Z , MAATEN L VAN DER , et al . Densely connected convolutional networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261-2269.
7 WANG J , ZHANG J , BAO W , et al . Not just privacy: improving performance of private deep learning in mobile cloud[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2018: 2407-2416.
8 ZHU M H , GUPTA S . To prune, or not to prune : exploring the efficacy of pruning for model compression[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1710.01878.pdf.
9 WANG H , ZHANG Q , WANG Y , et al . Structured deep neural network pruning by varying regularization parameters[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1804.09461.pdf.
10 JéGOU H , DOUZE M , SCHMID C . Product quantization for nearest neighbor search[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(1):117-128.
11 HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1503.02531.pdf.
12 HOWARD A G , ZHU M , CHEN B , et al . MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1704.04861.pdf.
13 ZHANG X , ZHOU X , LIN M , et al . ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 6848-6856.
14 DENIL M , SHAKIBI B , DINH L , et al . Predicting parameters in deep learning[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2013: 2148-2156.
15 HASSIBI B , STORK D G , WOLFF G J . Optimal brain surgeon and general network pruning[C]// Proceedings of the 1993 IEEE International Conference on Neural Networks. Piscataway: IEEE, 1993: 293-299.
16 HAN S , MAO H , DALLY W J . Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1510.00149.pdf.
17 MOLCHANOV P , TYREE S , KARRAS T , et al . Pruning convolutional neural networks for resource efficient inference[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1611.06440.pdf.
18 SUN Y , WANG X , TANG X . Sparsifying neural network connections for face recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 4856-4864.
19 LUO J , WU J . An entropy-based pruning method for CNN compression[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1706.05791.pdf.
20 DONG J , ZHENG H , LIAN L . Activation-based weight significance criterion for pruning deep neural networks[C]// Proceedings of the 2017 International Conference on Image and Graphics, LNCS 10667. Cham: Springer, 2017: 62-73.
21 HE Y , LIU P , WANG Z , et al . Pruning filter via geometric median for deep convolutional neural networks acceleration[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1811.00250.pdf.
22 靳丽蕾,杨文柱,王思乐,等 . 一种用于卷积神经网络压缩的混合剪枝方法[J]. 小型微型计算机系统, 2018, 39(12): 2596-2601. JIN L L , YANG W Z , WANG S L , et al . Mixed pruning method for convolutional neural network compression[J]. Journal of Chinese Computer Systems, 2018, 39(12): 2596-2601.
23 靳丽蕾 . 基于剪枝的卷积神经网络压缩方法研究[D]. 保定:河北大学, 2019:10-63.(JIN L L. Research on convolution neural network compression method based on pruning[D]. Baoding: Hebei University, 2019:10-63.)
24 WANG Z , ZHU C , XIA Z , et al . Towards thinner convolutional neural networks through gradually global pruning[C]// Proceedings of the 2017 IEEE International Conference on Image Processing. Piscataway: IEEE, 2017: 3939-3943.
25 LEE N, AJANTHAN T , TORR P H S . SNIP: single-shot network pruning based on connection sensitivity[EB/OL]. [2019-06-30].http://www.robots.ox.ac.uk/~tvg/publications/2019/SNIP-ICLR-camera-ready.pdf.
26 GUO Y , YAO A , CHEN Y . Dynamic network surgery for efficient DNNs[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2016: 1387-1395.
27 SRINIVAS S , BABU R V . Data-free parameter pruning for deep neural networks[C]// Proceedings of the 2015 British Machine Vision Conference. Durham: BMVA, 2015: No.31.
28 HU H ， PENG R ， TAI Y . Network trimming: a data-driven neuron pruning approach towards efficient deep architectures [EB/OL].https://arxiv.org/pdf/1607.03250.pdf.
29 LEE W, XIANG D . Information-theoretic measures for anomaly detection[C]// Proceedings of the 2001 IEEE Symposium on Security and Privacy. Piscataway: IEEE, 2001: 130-143.
30 LIU Y , SCHMIDT B . LightSpMV: Faster CSR-based sparse matrix-vector multiplication on CUDA-enabled GPUs[C]// 2015 IEEE 26th International Conference on Application-specific Systems, Architectures and Processors.Piscataway: IEEE, 2015: 82-89.
31 KRIZHEVSKY A . Learning multiple layers of features from tiny images[D]. Toronto, ON: University of Toronto, 2009:3-60.