Bio-inspired matrix reduction and quantization method for deep neural network

doi:10.11772/j.issn.1001-9081.2020020222

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (10): 2817-2821.DOI: 10.11772/j.issn.1001-9081.2020020222

• Artificial intelligence • Previous Articles Next Articles

Bio-inspired matrix reduction and quantization method for deep neural network

ZHU Qianqian^1,2, LIU Yuan^1,2, LI Fu³

1. School of Artificial Intelligence and Computer, Jiangnan University, Wuxi Jiangsu 214122, China;
2. Jiangsu Key Laboratory of Media Design and Software Technology(Jiangnan University), Wuxi Jiangsu 214122, China;
3. Quantum Cloud New Media Technology Company Limited, Wuxi Jiangsu 214122, China

Received:2020-03-05 Revised:2020-05-11 Online:2020-10-10 Published:2020-05-21
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61972182)

深度神经网络的仿生矩阵约简与量化方法

朱倩倩^1,2, 刘渊^1,2, 李甫³

1. 江南大学人工智能与计算机学院, 江苏无锡 214122;
2. 江苏省媒体设计与软件技术重点实验室(江南大学), 江苏无锡 214122;
3. 无锡量子云数字新媒体科技有限公司, 江苏无锡 214122

通讯作者: 朱倩倩
作者简介:朱倩倩(1995-),女,江苏连云港人,硕士研究生,CCF会员,主要研究方向:计算机视觉、人工智能;刘渊(1967-),男,江苏无锡人,教授,博士,CCF高级会员,主要研究方向:网络应用、数字媒体;李甫(1984-),男,湖北武汉人,教授,博士,主要研究方向:量子动力学、量子统计、未来网络、高性能计算。
基金资助:
国家自然科学基金资助项目（61972182）。

Abstract

Abstract: Bio-inspired Deep Neural Network (DNN) is a revolutionary breakthrough in artificial intelligent field. However, the lack of storage space as well as computing capacity caused by the explosive increase of the model weights not only keeps DNN apart from its original inspiration, but also makes it difficult to deploy DNN on embedded/mobile devices. In order to solve this problem, the biological selection principle in the evolution was studied, and a novel neural network algorithm based on "evolution" + "randomness" + "selection" was proposed. In this method, the size of the existing models were greatly simplified on the premise of maintaining the basic framework of the existing neural network models. First, the weight parameters were clustered. Then, based on the cluster centroid values of the parameters, the random perturbation was added to reconstruct the parameters. Finally, the image classification and object detection were performed on the reconstructed model to realize the accuracy test and model stability analysis. Experimental results on ImageNet dataset and COCO dataset show that the proposed model reconstruction method can compress the sizes of four models, including Darknet19, ResNet18, ResNet50 and YOLOv3, to 1/4-1/3 of the original ones, and under the condition of 1%-3% performance improvement in the test accuracy of image classification and object detection, there is the possibility of further simplification.

Key words: model compression, Deep Neural Network (DNN), parameter reconstruction, object detection, network dynamics, bio-inspired model

摘要： 基于生物学原理的深度神经网络（DNN）的发展给人工智能领域带来了革命性的突破，然而当前神经网络的发展却越来越脱离生物学原理，DNN越来越臃肿的模型对存储空间和计算力的需求越来越高，并且对于DNN在嵌入式/移动端设备上的部署带来了阻碍。针对这一问题，对生物学进化选择原理进行研究，并提出一种基于“进化”+“随机”+“选择”的全新神经网络算法。该方法在保持现有神经网络模型的基本框架的前提下，能极大简化现有模型的大小。首先对权值参数进行聚类，然后在参数的聚类质心值的基础上添加随机微扰进行参数重构，最后通过对重构模型进行图像分类和目标检测来实现准确度测试以及模型稳定性分析。在ImageNet数据集和COCO数据集上的实验结果表明，提出的模型重构方法在对图像分类和目标检测的测试准确度提升1%~3%的情况下，仍可将Darknet19、ResNet18、ResNet50以及YOLOv3等四种重构模型的体量压缩到原来的1/4~1/3，并还有进一步简化的可能。

关键词: 模型压缩, 深度神经网络, 参数重构, 目标检测, 网络动力学, 仿生模型

CLC Number:

TP183

ZHU Qianqian, LIU Yuan, LI Fu. Bio-inspired matrix reduction and quantization method for deep neural network[J]. Journal of Computer Applications, 2020, 40(10): 2817-2821.

朱倩倩, 刘渊, 李甫. 深度神经网络的仿生矩阵约简与量化方法[J]. 计算机应用, 2020, 40(10): 2817-2821.

References

[1] WONG A,FAMUORI M,SHAFIEE M J,et al. YOLO Nano:a highly compact You Only Look Once convolutional neural network for object detection[EB/OL].[2019-10-24]. https://arxiv.org/pdf/1910.01271.pdf.
[2] 于秀萍, 吕淑平, 陈志韬. 基于YOLO算法的多类目标识别[J]. 实验室研究与探索,2019,38(3):34-36,76.(YU X P,LYU S P,CHEN Z T. Multi-object recognition based on YOLO algorithm[J]. Research and Exploration in Laboratory,2019,38(3):34-36,76.)
[3] LIU Z,LI J,SHEN Z,et al. Learning efficient convolutional networks through network slimming[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2755-2763.
[4] STOCK P,JOULIN A,GRIBONVAL R,et al. And the bit goes down:revisiting the quantization of neural networks[EB/OL].[2019-12-24]. https://arxiv.org/pdf/1907.05686.pdf.
[5] HAN S,MAO H,DALLY W J. Deep compression:compressing deep neural networks with pruning, trained quantization and Huffman coding[EB/OL].[2019-08-22]. https://arxiv.org/pdf/1510.00149.pdf.
[6] DENIL M,SHAKIBI B,DINH L,et al. Predicting parameters in deep learning[C]//Proceeding of the 26th International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2013:2148-2156.
[7] 程龙, 刘洋. 脉冲神经网络:模型、学习算法与应用[J]. 控制与决策,2018,33(5):923-937.(CHENG L,LIU Y. Spiking neural networks:model,learning algorithms and applications[J]. Control and Decision,2018,33(5):923-937.)
[8] SUN M,CHEN X,ZHU Y,et al. Neural network model combined with pupil recovery for Fourier ptychographic microscopy[J]. Optics Express,2019,27(17):24161-24174.
[9] VOGELS T P,RAJAN K,ABBOTT L F. Neural network dynamics[J]. Annual Review of Neuroscience,2005,28:357-376.
[10] 陈莉君, 李卓. 基于深度神经压缩的YOLO加速研究[J]. 现代计算机,2019(11):3-7.(CHEN L J,LI Z. Research on YOLO acceleration based on deep compression[J]. Modern Computer, 2019(11):3-7.)
[11] 雷杰, 高鑫, 宋杰, 等. 深度网络模型压缩综述[J]. 软件学报, 2018,29(2):251-266.(LEI J,GAO X,SONG J,et al. Survey of deep neural network model compression[J]. Journal of Software,2018,29(2):251-266.)
[12] 谢斌红, 钟日新, 潘理虎, 等. 结合剪枝与流合并的卷积神经网络加速压缩方法[J]. 计算机应用,2020,40(3):621-625. (XIE B H, ZHONG R X, PAN L H, et al. Accelerated compression method for convolutional neural networks combining with pruning and stream merging[J]. Journal of Computer Applications,2020,40(3):621-625.)
[13] WU C W. ProdSumNet:reducing model parameters in deep neural networks via product-of-sums matrix decompositions[EB/OL].[2019-10-08]. https://arxiv.org/pdf/1809.02209.pdf.
[14] GUO Y,YAO A,CHEN Y. Dynamic network surgery for efficient DNNs[C]//Proceeding of the 30th International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2016:1379-1387.
[15] CHENG Y,WANG D,ZHOU P,et al. A survey of model compression and acceleration for deep neural networks[EB/OL].[2019-09-11]. https://arxiv.org/pdf/1710.09282.pdf.
[16] LIAO S,XIE Y,LIN X,et al. Reduced-complexity deep neural networks design using multi-level compression[J]. IEEE Transactions on Sustainable Computing,2019,4(2):245-251.
[17] GONG Y, LIU L, YANG M, et al. Compressing deep convolutional networks using vector quantization[EB/OL].[2019-08-20]. https://arxiv.org/pdf/1412.6115.pdf.
[18] HINTON G,VINYALS O,DEAN J. Distilling the knowledge in a neural network[EB/OL].[2019-08-18]. https://arxiv.org/pdf/1503.02531.pdf.
[19] REDMON J,DIVVALA S,GIRSHICK R,et al. You only look once:Unified,real-time object detection[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:779-788.
[20] REDMON J,FARHADI A. YOLO9000:better,faster,stronger[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:6517-6525.
[21] REDMON J,FARHADI A. YOLOv3:an incremental improvement[EB/OL].[2019-02-14]. https://arxiv.org/pdf/1804.02767.pdf.

Bio-inspired matrix reduction and quantization method for deep neural network

深度神经网络的仿生矩阵约简与量化方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[2]	MA Jialiang, CHEN Bin, SUN Xiaofei. General object detection framework based on improved Faster R-CNN [J]. Journal of Computer Applications, 2021, 41(9): 2712-2719.
[3]	HUANG Jishuang, ZHANG Hua, LI Yonglong, ZHAO Hao, WANG Haoran, FENG Chuncheng. Hydraulic tunnel defect recognition method based on dynamic feature distillation [J]. Journal of Computer Applications, 2021, 41(8): 2358-2365.
[4]	CHEN Jing, MAO Yingchi, CHEN Hao, WANG Longbao, WANG Zicheng. Dam defect object detection method based on improved single shot multibox detector [J]. Journal of Computer Applications, 2021, 41(8): 2366-2372.
[5]	FAN Wei, LI Chenxuan, XING Yan, HUANG Rui, PENG Hongjian. Binary classification to multiple classification progressive detection network for aero-engine damage images [J]. Journal of Computer Applications, 2021, 41(8): 2352-2357.
[6]	GAO Qinquan, HUANG Bingcheng, LIU Wenzhe, TONG Tong. Bamboo strip surface defect detection method based on improved CenterNet [J]. Journal of Computer Applications, 2021, 41(7): 1933-1938.
[7]	WANG Shuyan, HOU Zeyu, SUN Jiaze. Difference detection method of adversarial samples oriented to deep learning [J]. Journal of Computer Applications, 2021, 41(7): 1849-1856.
[8]	XIAO Zhenyuan, WANG Yihan, LUO Jianqiao, XIONG Ying, LI Bailin. RefineDet based on subsection weighted loss function [J]. Journal of Computer Applications, 2021, 41(7): 1928-1932.
[9]	LI Chao, LAN Hai, WEI Xian. Attention-based object detection with millimeter wave radar-lidar fusion [J]. Journal of Computer Applications, 2021, 41(7): 2137-2144.
[10]	ZHANG Mingming, LU Qingning, LI Wenzhong, SONG Hu. Deep neural network compression algorithm based on combined dynamic pruning [J]. Journal of Computer Applications, 2021, 41(6): 1589-1596.
[11]	ZHANG Wenye, SHANG Fangxin, GUO Hao. Mixed precision neural network quantization method based on Octave convolution [J]. Journal of Computer Applications, 2021, 41(5): 1299-1304.
[12]	GUO Wenxu, SU Yuanqi, LIU Yuehu. YOLOv3 compression and acceleration based on ZYNQ platform [J]. Journal of Computer Applications, 2021, 41(3): 669-676.
[13]	HOU Yunlong, ZHU Lei, CHEN Qin, LYU Suidong. Salient object detection based on difference of Gaussian feature network [J]. Journal of Computer Applications, 2021, 41(3): 706-713.
[14]	YANG Li, WANG Shihui, ZHU Bo. Point-of-interest recommendation algorithm combing dynamic and static preferences [J]. Journal of Computer Applications, 2021, 41(2): 398-406.
[15]	CAO Jianrong, LYU Junjie, WU Xinying, ZHANG Xu, YANG Hongjuan. Fall detection algorithm integrating motion features and deep learning [J]. Journal of Computer Applications, 2021, 41(2): 583-589.