关键词: 在线蒸馏, 图像分类, 异构协同蒸馏, 关系感知全局教师, 分歧-一致性约束

Abstract: Multi-branch online distillation for image classification often produces convergent branch representations, limiting the use of complementary knowledge. To improve single-branch deployment, a heterogeneous collaborative distillation algorithm guided by a relation-aware teacher is proposed. Channel, spatial, and channel-spatial attention branches are constructed on a shared shallow backbone; cross-branch relation modeling generates a global teacher; and a divergence-consistency constraint preserves diversity early and promotes fusion later. On CIFAR-10, CIFAR-100, and Tiny-ImageNet, the proposed algorithm achieves the best result in all 13 backbone settings, outperforming the strongest competitor by 1.34 percentage points on average and the single-branch baseline by 2.96 points. It consistently surpasses Deep Mutual Learning, Collaborative Learning, Knowledge Distillation by On-the-Fly Native Ensemble, Online Knowledge Distillation with Diverse Peers, Peer Collaborative Learning, and Feature Fusion for Online Mutual Knowledge Distillation, with larger gains on complex datasets.

Key words: online distillation, image classification, heterogeneous collaborative distillation, relation-aware global teacher, divergence-consistency constraint