Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (5): 1253-1259.DOI: 10.11772/j.issn.1001-9081.2019081374

• Artificial intelligence • Previous Articles     Next Articles

Gradient-based deep network pruning algorithm

WANG Zhongfeng1,2,3, XU Zhiyuan1,2,3,4, SONG Chunhe1,2,3, ZHANG Hongyu5, CAI Yingkai5   

  1. 1.State Key Laboratory of Robotics (Shenyang Institute of Automation, Chinese Academy of Sciences), ShenyangLiaoning 110016, China
    2.Key Laboratory of Networked Control System(Shenyang Institute of Automation, Chinese Academy of Sciences),ShenyangLiaoning 110016,China
    3.Institute of Robotics and Intelligent Manufacturing, Chinese Academy of Sciences, ShenyangLiaoning 110169, China
    4.University of Chinese Academy of Sciences, Beijing 100049, China
    5.State Grid Liaoning Electric Power Company Limited, ShenyangLiaoning 110016, China
  • Received:2019-08-07 Revised:2019-10-15 Online:2020-05-10 Published:2020-05-15
  • Contact: XU Zhiyuan, born in 1993, M. S. candidate. His research interests include deep learning compression.
  • About author:WANG Zhongfeng, born in 1976, Ph. D., research fellow. His research interests include ubiquitous electric power internet of things.XU Zhiyuan, born in 1993, M. S. candidate. His research interests include deep learning compression.SONG Chunhe, born in 1981, Ph. D., research fellow. His research interests include edge computing.ZHANG Hongyu, born in 1976, M. S., senior engineer. His research interests include power system and its automation.CAI Yingkai, born in 1979, M. S., senior engineer. His research interests include power marketing informationization.

基于梯度的深度网络剪枝算法

王忠锋1,2,3, 徐志远1,2,3,4, 宋纯贺1,2,3, 张宏宇5, 蔡颖凯5   

  1. 1.机器人学国家重点实验室(中国科学院 沈阳自动化研究所),沈阳 110016
    2.网络化控制系统重点实验室(中国科学院 沈阳自动化研究所),沈阳 110016
    3.中国科学院 机器人与智能制造创新研究院,沈阳 110169
    4.中国科学院大学,北京 100049
    5.国网辽宁省电力有限公司,沈阳 110016
  • 通讯作者: 徐志远(1993—)
  • 作者简介:王忠锋(1976—),男,辽宁沈阳人,研究员,博士,主要研究方向:泛在电力物联网; 徐志远(1993—),男,辽宁抚顺人,硕士研究生,主要研究方向:深度学习压缩; 宋纯贺(1981—),男,辽宁鞍山人,研究员,博士,主要研究方向:边缘计算; 张宏宇(1976—),男,辽宁义县人,高级工程师,硕士,主要研究方向:电力系统及其自动化; 蔡颖凯(1979—),男,辽宁鞍山人,高级工程师,硕士,主要研究方向:电力营销信息化。

Abstract:

Deep neural network models usually have a large number of redundant weight parameters. Calculating the deep network model requires a large amount of computing resources and storage pace, which makes the deep network model difficult to be deployed on some edge devices and embedded devices. To resolve this issue, a Gradient-based Deep network Pruning (GDP) algorithm was proposed. The core idea of GDP algorithm was to use the gradient as the basis for judging the importance of each weight. To eliminate the weights corresponding to the gradients smaller than the threshold, an adaptive method was used to find the threshold to screen the weights. The deep network model was retrained after pruning to restore the network performance. The experimental results show that the GDP algorithm reduces the computational cost by 35.3 percentage points with a precision loss of only 0.14 percentage points on the CIFAR-10 dataset. Compared with the state-of-the-art PFEC (Pruning Filters for Efficient ConvNets) algorithm, the GDP algorithm increases the network model accuracy by 0.13 percentage points, and reduces the computational cost by 1.1 percentage points, indicating that the proposed algorithm has superior performance of deep network in terms of both compression and acceleration.

Key words: deep network, compression and acceleration, pruning, adaptive threshold, neural network

摘要:

深度神经网络模型通常存在大量冗余的权重参数,计算深度网络模型需要占用大量的计算资源和存储空间,导致深度网络模型难以部署在一些边缘设备和嵌入式设备上。针对这一问题,提出了一种基于梯度的深度网络剪枝(GDP)算法。GDP算法核心思想是以梯度作为评判权值重要性的依据。首先,通过自适应的方法找出阈值进行权值参数的筛选;然后,剔除那些小于阈值的梯度所对应的权值;最后,重新训练剪枝后的深度网络模型来恢复网络精度。实验结果表明:在CIFAR-10数据集上,GDP算法在精度仅下降0.14个百分点的情况下,计算量减少了35.3个百分点;与当前流行的PFEC算法相比,GDP算法使网络模型精度提高了0.13个百分点,计算量下降了1.1个百分点,具有更优越的深度网络压缩与加速性能。

关键词: 深度网络, 压缩与加速, 剪枝, 自适应阈值, 神经网络

CLC Number: