关键词: 深度神经网络, 网络压缩, 网络剪枝, 知识蒸馏, 参数量化

Abstract:

With the rapid development of Artificial Intelligence （AI） in recent years， the demand for Deep Neural Network （DNN） from devices with limited resources such as embedded devices and mobile devices has increased sharply. The problem of how to compress neural networks without affecting the effect of DNNs has great theoretical and practical significance， and is a hot research topic in deep learning now. Firstly， aiming at the problem that DNN is difficult to be ported to resource-limited devices such as mobile devices due to their large models and large computational cost， the experimental performance of existing DNN compression algorithms in terms of memory usage， running speed， and compression effect was deeply analyzed， so that the influence factors of the DNN compression algorithm were explored. Then， the knowledge transfer structure composed of student network and teacher network was designed， the knowledge distillation， structural design， network pruning， and parameter quantization mechanisms were fused together， and a DNN optimization and compression model based on hybrid mechanism was proposed. Experimental comparison and analysis were conducted on mini-ImageNet dataset using AlexNet as the Benchmark. Experimental results show that the capacity of compressed AlexNet is reduced by 98.5% with 6.3% loss of accuracy， which verify the effectiveness of the proposed algorithm.

Key words: Deep Neural Network (DNN), network compression, network pruning, knowledge distillation, parameter quantization