关键词: 灾难性遗忘, 联邦类增量学习, 多头自注意力, 标签语义嵌入, 无数据知识蒸馏

Abstract: Catastrophic forgetting poses a significant challenge to Federated Class Incremental Learning (FCIL), leading to performance degradation during continuous tasks. To address this issue, a federated class incremental learning method with multi-head self-attention for label semantic embedding ——ATTLSE (ATTention Label Semantic Embedding) was proposed. First, Label Semantic Embedding (LSE) with multi-head self-attention was integrated with a generator. Second, during the stage of data-free knowledge distillation, the generator enhanced with multi-head self-attention was used to produce more meaningful data samples, which guided the training of client models. Finally, the impact of catastrophic forgetting in FCIL was mitigated. Experimental results demonstrate that, the average accuracy of the ATTLSE method on the CIFAR-100 and Tiny_ImageNet datasets improved by approximately 1 to 6 percentage points compared to the LANDER method, further alleviating the catastrophic forgetting issue in continuous tasks for Federated Class Incremental Learning.

Key words: catastrophic forgetting, Federated Class Incremental Learning (FCIL), multi-head self-attention, label semantic embedding, data-free knowledge distillation