CNN model compression based on activation-entropy based layer-wise iterative pruning strategy

doi:10.11772/j.issn.1001-9081.2019111977

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (5): 1260-1265.DOI: 10.11772/j.issn.1001-9081.2019111977

• Artificial intelligence • Previous Articles Next Articles

CNN model compression based on activation-entropy based layer-wise iterative pruning strategy

CHEN Chengjun, MAO Yingchi, WANG Yichao

College of Computer and Information, Hohai University, Nanjing Jiangsu 211100, China

Received:2019-11-21 Revised:2020-02-12 Online:2020-05-10 Published:2020-05-15
Contact: MAO Yingchi, born in 1976, Ph. D., professor. Her research interests include internet of things, distributed data processing.
About author:CHEN Chengjun, born in 1996, M. S. candidate. His research interests include artificial intelligence, distributed data processing.MAO Yingchi, born in 1976, Ph. D., professor. Her research interests include internet of things, distributed data processing.WANG Yichao, born in 1994, M. S. His research interests include distributed data processing.
Supported by:
This work is partially supported by the “13th Five Year Plan” National Key Research and Development Program of China (2018YFC0407105), the Key Research and Development Program of Huaneng Group (HNKJ17-21).

基于激活-熵的分层迭代剪枝策略的CNN模型压缩

陈程军, 毛莺池, 王绎超

河海大学计算机与信息学院，南京 211100

通讯作者: 毛莺池(1976—)
作者简介:陈程军(1996—)，男，江苏南通人，硕士研究生，CCF会员，主要研究方向：人工智能、分布式数据处理；毛莺池(1976—)，女，上海人，教授，博士，CCF会员，主要研究方向：物联网、分布式数据处理；王绎超(1994—)，男，山西介休人，硕士，主要研究方向：分布式数据处理。
基金资助:
“十三五”国家重点研发计划项目（2018YFC0407105）；华能集团重点研发项目（HNKJ17-21）。

Abstract

Abstract:

Since the existing pruning strategies of the Convolutional Neural Network （CNN） model are different and have general effects, an Activation-Entropy based Layer-wise Iterative Pruning (AE-LIP) strategy was proposed to reduce the parameter amount of the model while ensuring the accuracy of the model within a controllable range. Firstly, combined with the neuronal activation value and information entropy, a weight evaluation criteria based on activation-entropy was constructed, and the weight importance score was calculated. Secondly, the pruning was performed layer by layer, the weights were sorted according to the importance score, and the pruning number in each layer was combined to filter out the weights to be pruned and set them to zero. Finally, the model was fine-tuned, and the above process was repeated until the iteration ended. The experimental results show that the activation-entropy based layer-wise iterative pruning strategy makes the AlexNet model compressed 87.5%, and the corresponding accuracy is reduced by 2.12 percentage points, which is 1.54 percentage points higher than that of the magnitude-based weight pruning strategy and 0.91 percentage points higher than that of the correlation-based weight pruning strategy; the strategy makes VGG-16 model compressed 84.1%, and the corresponding accuracy is reduced by 2.62 percentage points, which is 0.62 and 0.27 percentage points higher than those of the two above strategies. It can be seen that the proposed strategy reduces the size of the CNN model effectively while ensuring the accuracy of the model, and is helpful for the deployment of CNN model on mobile devices with limited storage.

Key words: mobile cloud computing, neuronal activation value, information entropy, iterative pruning, model compression

摘要：

针对卷积神经网络(CNN)模型现有剪枝策略各尽不同和效果一般的情况，提出了基于激活-熵的分层迭代剪枝(AE-LIP)策略，保证模型精度在可控范围内的同时缩减模型的参数量。首先，结合神经元激活值和信息熵，构建基于激活-熵的权重评判准则，计算权值重要性得分；然后，逐层剪枝，根据重要性得分对权值排序，并结合各层剪枝数量筛选出待剪枝权重并将其设置为0；最后，微调模型，重复上述过程，直至迭代结束。实验结果表明，采用基于激活-熵的分层迭代剪枝策略:AlexNet模型压缩了87.5%;相应的准确率下降了2.12个百分点，比采用基于幅度的权重剪枝策略提高了1.54个百分点，比采用基于相关性的权重剪枝策略提高0.91个百分点。VGG-16模型压缩了84.1%；相应的准确率下降了2.62个百分点，比采用上述两个对比策略分别提高了0.62个百分点和0.27个百分点。说明所提策略在保证模型精确度下有效缩减了CNN模型的大小，有助于CNN模型在存储受限的移动设备上的部署。

关键词: 移动云计算, 神经元激活值, 信息熵, 迭代剪枝, 模型压缩

CLC Number:

TP389.1

CHEN Chengjun, MAO Yingchi, WANG Yichao. CNN model compression based on activation-entropy based layer-wise iterative pruning strategy[J]. Journal of Computer Applications, 2020, 40(5): 1260-1265.

陈程军, 毛莺池, 王绎超. 基于激活-熵的分层迭代剪枝策略的CNN模型压缩[J]. 计算机应用, 2020, 40(5): 1260-1265.

References

1 KRIZHEVSKY A , SUTSKEVER I , HINTON G E . ImageNet classification with deep convolutional neural networks[C]// Proceedings of the 25th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2012: 1097-1105.
2 SIMONYAN K , ZISSERMAN A . Very deep convolutional networks for large-scale image recognition[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1409.1556.pdf.
3 SZEGEDY C , LIU W , JIA Y , et al . Going deeper with convolutions[C]// Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2015: 1-9.
4 彭冬亮,王天兴 . 基于GoogLeNet模型的剪枝算法[J]. 控制与决策, 2019, 34(6):1259-1264. (PENG D L, WANG T X. Pruning algorithm based on GoogLeNet model[J]. Control and Decision, 2019, 34(6):1259-1264.)
5 HE K , ZHANG X , REN S , et al . Deep residual learning for image recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016:770-778.
6 HUANG G , LIU Z , MAATEN L VAN DER , et al . Densely connected convolutional networks[C]// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2017: 2261-2269.
7 WANG J , ZHANG J , BAO W , et al . Not just privacy: improving performance of private deep learning in mobile cloud[C]// Proceedings of the 24th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2018: 2407-2416.
8 ZHU M H , GUPTA S . To prune, or not to prune : exploring the efficacy of pruning for model compression[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1710.01878.pdf.
9 WANG H , ZHANG Q , WANG Y , et al . Structured deep neural network pruning by varying regularization parameters[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1804.09461.pdf.
10 JéGOU H , DOUZE M , SCHMID C . Product quantization for nearest neighbor search[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(1):117-128.
11 HINTON G , VINYALS O , DEAN J . Distilling the knowledge in a neural network[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1503.02531.pdf.
12 HOWARD A G , ZHU M , CHEN B , et al . MobileNets: efficient convolutional neural networks for mobile vision applications[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1704.04861.pdf.
13 ZHANG X , ZHOU X , LIN M , et al . ShuffleNet: an extremely efficient convolutional neural network for mobile devices[C]// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2018: 6848-6856.
14 DENIL M , SHAKIBI B , DINH L , et al . Predicting parameters in deep learning[C]// Proceedings of the 26th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2013: 2148-2156.
15 HASSIBI B , STORK D G , WOLFF G J . Optimal brain surgeon and general network pruning[C]// Proceedings of the 1993 IEEE International Conference on Neural Networks. Piscataway: IEEE, 1993: 293-299.
16 HAN S , MAO H , DALLY W J . Deep compression: compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1510.00149.pdf.
17 MOLCHANOV P , TYREE S , KARRAS T , et al . Pruning convolutional neural networks for resource efficient inference[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1611.06440.pdf.
18 SUN Y , WANG X , TANG X . Sparsifying neural network connections for face recognition[C]// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2016: 4856-4864.
19 LUO J , WU J . An entropy-based pruning method for CNN compression[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1706.05791.pdf.
20 DONG J , ZHENG H , LIAN L . Activation-based weight significance criterion for pruning deep neural networks[C]// Proceedings of the 2017 International Conference on Image and Graphics, LNCS 10667. Cham: Springer, 2017: 62-73.
21 HE Y , LIU P , WANG Z , et al . Pruning filter via geometric median for deep convolutional neural networks acceleration[EB/OL]. [2019-06-30].https://arxiv.org/pdf/1811.00250.pdf.
22 靳丽蕾,杨文柱,王思乐,等 . 一种用于卷积神经网络压缩的混合剪枝方法[J]. 小型微型计算机系统, 2018, 39(12): 2596-2601. JIN L L , YANG W Z , WANG S L , et al . Mixed pruning method for convolutional neural network compression[J]. Journal of Chinese Computer Systems, 2018, 39(12): 2596-2601.
23 靳丽蕾 . 基于剪枝的卷积神经网络压缩方法研究[D]. 保定:河北大学, 2019:10-63.(JIN L L. Research on convolution neural network compression method based on pruning[D]. Baoding: Hebei University, 2019:10-63.)
24 WANG Z , ZHU C , XIA Z , et al . Towards thinner convolutional neural networks through gradually global pruning[C]// Proceedings of the 2017 IEEE International Conference on Image Processing. Piscataway: IEEE, 2017: 3939-3943.
25 LEE N, AJANTHAN T , TORR P H S . SNIP: single-shot network pruning based on connection sensitivity[EB/OL]. [2019-06-30].http://www.robots.ox.ac.uk/~tvg/publications/2019/SNIP-ICLR-camera-ready.pdf.
26 GUO Y , YAO A , CHEN Y . Dynamic network surgery for efficient DNNs[C]// Proceedings of the 30th International Conference on Neural Information Processing Systems. New York: Curran Associates Inc., 2016: 1387-1395.
27 SRINIVAS S , BABU R V . Data-free parameter pruning for deep neural networks[C]// Proceedings of the 2015 British Machine Vision Conference. Durham: BMVA, 2015: No.31.
28 HU H ， PENG R ， TAI Y . Network trimming: a data-driven neuron pruning approach towards efficient deep architectures [EB/OL].https://arxiv.org/pdf/1607.03250.pdf.
29 LEE W, XIANG D . Information-theoretic measures for anomaly detection[C]// Proceedings of the 2001 IEEE Symposium on Security and Privacy. Piscataway: IEEE, 2001: 130-143.
30 LIU Y , SCHMIDT B . LightSpMV: Faster CSR-based sparse matrix-vector multiplication on CUDA-enabled GPUs[C]// 2015 IEEE 26th International Conference on Application-specific Systems, Architectures and Processors.Piscataway: IEEE, 2015: 82-89.
31 KRIZHEVSKY A . Learning multiple layers of features from tiny images[D]. Toronto, ON: University of Toronto, 2009:3-60.

[1]	HUANG Jishuang, ZHANG Hua, LI Yonglong, ZHAO Hao, WANG Haoran, FENG Chuncheng. Hydraulic tunnel defect recognition method based on dynamic feature distillation [J]. Journal of Computer Applications, 2021, 41(8): 2358-2365.
[2]	ZHANG Mingming, LU Qingning, LI Wenzhong, SONG Hu. Deep neural network compression algorithm based on combined dynamic pruning [J]. Journal of Computer Applications, 2021, 41(6): 1589-1596.
[3]	ZHANG Wenye, SHANG Fangxin, GUO Hao. Mixed precision neural network quantization method based on Octave convolution [J]. Journal of Computer Applications, 2021, 41(5): 1299-1304.
[4]	WANG Zhihe, CHANG Xiaoqing, DU Hui. Adaptive affinity propagation clustering algorithm based on universal gravitation [J]. Journal of Computer Applications, 2021, 41(5): 1337-1342.
[5]	YUAN Yuan, WU Wen, WAN Yi. Single image shadow detection method based on entropy driven domain adaptive learning [J]. Journal of Computer Applications, 2020, 40(7): 2131-2136.
[6]	ZHANG Wu, CHEN Hongmei. Hyperspectral band selection based on multi-kernelized fuzzy rough set and grasshopper optimization algorithm [J]. Journal of Computer Applications, 2020, 40(5): 1425-1430.
[7]	XIE Binhong, ZHONG Rixin, PAN Lihu, ZHANG Yingjun. Accelerated compression method for convolutional neural network combining with pruning and stream merging [J]. Journal of Computer Applications, 2020, 40(3): 621-625.
[8]	WANG Qingyong, MAO Yingchi, WANG Yichao, WANG Longbao. Computing task offloading based on multi-cloudlet collaboration [J]. Journal of Computer Applications, 2020, 40(2): 328-334.
[9]	TONG Yuzhen, WANG Yingming. Multi-attribute group decision making method for probabilistic linguistic term set based on regret theory and distance from average solution method [J]. Journal of Computer Applications, 2020, 40(11): 3152-3158.
[10]	LEI Xiaokang, YIN Zhigang, ZHAO Ruilian. FPGA-based convolutional neural network fixed-point acceleration [J]. Journal of Computer Applications, 2020, 40(10): 2811-2816.
[11]	ZHU Qianqian, LIU Yuan, LI Fu. Bio-inspired matrix reduction and quantization method for deep neural network [J]. Journal of Computer Applications, 2020, 40(10): 2817-2821.
[12]	ZHANG Wu, CHEN Hongmei. Hyperspectral band selection algorithm based on kernelized fuzzy rough set [J]. Journal of Computer Applications, 2020, 40(1): 258-263.
[13]	DING Lianjing, LIU Guangshuai, LI Xurui, CHEN Xiaowen. Face recognition combining weighted information entropy with enhanced local binary pattern [J]. Journal of Computer Applications, 2019, 39(8): 2210-2216.
[14]	JI Shuwei, YANG Xiwang, HUANG Jinying, YIN Ning. Model compression method of convolution neural network based on feature-reuse [J]. Journal of Computer Applications, 2019, 39(6): 1607-1613.
[15]	MAO Yingchi, CAO Hai, PING Ping, LI Xiaofang. Feature selection based on maximum conditional and joint mutual information [J]. Journal of Computer Applications, 2019, 39(3): 734-741.

CNN model compression based on activation-entropy based layer-wise iterative pruning strategy

基于激活-熵的分层迭代剪枝策略的CNN模型压缩

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics