Abstract:

With the rapid development of Artificial Intelligence （AI） in recent years， the demand for Deep Neural Network （DNN） from devices with limited resources such as embedded devices and mobile devices has increased sharply. The problem of how to compress neural networks without affecting the effect of DNNs has great theoretical and practical significance， and is a hot research topic in deep learning now. Firstly， aiming at the problem that DNN is difficult to be ported to resource-limited devices such as mobile devices due to their large models and large computational cost， the experimental performance of existing DNN compression algorithms in terms of memory usage， running speed， and compression effect was deeply analyzed， so that the influence factors of the DNN compression algorithm were explored. Then， the knowledge transfer structure composed of student network and teacher network was designed， the knowledge distillation， structural design， network pruning， and parameter quantization mechanisms were fused together， and a DNN optimization and compression model based on hybrid mechanism was proposed. Experimental comparison and analysis were conducted on mini-ImageNet dataset using AlexNet as the Benchmark. Experimental results show that the capacity of compressed AlexNet is reduced by 98.5% with 6.3% loss of accuracy， which verify the effectiveness of the proposed algorithm.

Key words: Deep Neural Network (DNN), network compression, network pruning, knowledge distillation, parameter quantization

摘要：

近年来人工智能（AI）应用飞速发展，嵌入式设备与移动设备等有限资源设备对深度神经网络（DNN）的需求急剧增加。如何在不影响DNN效果的基础上对神经网络进行压缩具有极大理论与现实意义，也是当下深度学习的热门研究话题。首先，针对DNN因模型大、计算量大而难以移植至移动设备等有限资源设备的问题，深入分析已有DNN压缩算法在内存占用、运行速度及压缩效果等方面的实验性能，从而挖掘DNN压缩算法的影响要素；然后，设计学生网络和教师网络组成的知识迁移结构，融合知识蒸馏、结构设计、网络剪枝和参数量化机制，提出基于混合机制的DNN优化压缩算法。在mini-ImageNet数据集上以AlexNet为Benchmark，进行实验比较与分析。实验结果表明，所提算法在压缩结果的准确率降低6.3%的情况下，使压缩后的AlexNet的容量减小98.5%，验证了所提算法的有效性。

关键词: 深度神经网络, 网络压缩, 网络剪枝, 知识蒸馏, 参数量化