关键词: 小样本, 模型蒸馏, 集成学习, 元学习, 特征合并

Abstract:

Aiming at the lack of training data of deep neural networks in image recognition， as well as the loss of detailed features and the large amount of distillation calculations in the multi-model distillation， a model distillation model based on training weak teacher networks about few-shots was proposed. Firstly， the weak teacher network set was trained through the Bootstrap aggregating （Bagging） algorithm in the ensemble learning algorithm. While retaining the detailed features of the image dataset， parallel computing was able to be realized to improve the efficiency of network generation. Then， the knowledge merging algorithm was combined， and single high-quality high-complexity teacher networks were formed based on the weak teacher network feature maps， thereby obtaining the image feature maps with more significant details. Finally， based on the current advanced model distillation， an ensemble distillation algorithm with meta-network improved with combined feature maps was proposed， which reduced the calculation of meta-network training and realized the training of the target network about few-shots at the same time. Experimental results show that the algorithm had a 6.39% relative improvement in accuracy compared to the distillation scheme that uses a high-quality network as the teacher network. Comparing the accuracy of the model which obtained by training and distilling the teacher networks with Adaptive Boosting （AdaBoost） algorithm and the accuracy of the model obtained by the ensemble distillation model， the difference is within the given error range. However， the network generation rate of the ensemble distillation algorithm was increased by 4.76 times compared with that of AdaBoost algorithm. Therefore， the proposed algorithm can effectively improve the accuracy and training efficiency of the target model about few-shots.

Key words: few-shot, model distillation, Ensemble Learning （EL）, meta learning, feature merging