Journal of Computer Applications

    Next Articles

Federated class-incremental learning method with multi-head self-attention for label semantic embedding

  

  • Received:2024-10-14 Revised:2024-12-20 Accepted:2024-12-20 Online:2024-12-25 Published:2024-12-25

融合多头自注意力的标签语义嵌入联邦类增量学习方法

王虎1,王晓峰1,2*,李可1,马云洁3   

  1. 1.北方民族大学 计算机科学与工程学院,银川 750021

    2.图形图像智能处理国家民委重点实验室(北方民族大学),银川 750021 3.北方民族大学 数学与信息科学学院,银川 750021

  • 通讯作者: 王晓峰
  • 基金资助:
    国家自然科学基金;宁夏青年拔尖人才项目

Abstract: Catastrophic forgetting poses a significant challenge to Federated Class Incremental Learning (FCIL), leading to performance degradation during continuous tasks. To address this issue, a federated class incremental learning method with multi-head self-attention for label semantic embedding ——ATTLSE (ATTention Label Semantic Embedding) was proposed. First, Label Semantic Embedding (LSE) with multi-head self-attention was integrated with a generator. Second, during the stage of data-free knowledge distillation, the generator enhanced with multi-head self-attention was used to produce more meaningful data samples, which guided the training of client models. Finally, the impact of catastrophic forgetting in FCIL was mitigated. Experimental results demonstrate that, the average accuracy of the ATTLSE method on the CIFAR-100 and Tiny_ImageNet datasets improved by approximately 1 to 6 percentage points compared to the LANDER method, further alleviating the catastrophic forgetting issue in continuous tasks for Federated Class Incremental Learning.

Key words: catastrophic forgetting, Federated Class Incremental Learning (FCIL), multi-head self-attention, label semantic embedding, data-free knowledge distillation 

摘要: 灾难性遗忘对联邦类增量学习(FCIL)构成了显著挑战,导致联邦类增量学习进行持续任务时性能下降的问题。针对此问题,提出一种融合多头自注意力的标签语义嵌入的联邦类增量学习方法ATTLSE (ATTention Label Semantic Embedding)。首先,该方法通过融合多头自注意力的标签语义嵌入(LSE)和生成器;其次,在无数据知识蒸馏阶段,ATTLSE依靠融合多头自注意力的生成器,可以生成更多有意义的数据样本来指导用户端模型的训练,最后缓解了灾难性遗忘问题在FCIL中的影响。实验结果表明,在CIFAR-100和Tiny_ImageNet数据集上与LANDER方法相比,ATTLSE平均准确率提升了1~6个百分点左右,进一步缓解了持续任务在联邦类增量学习上的灾难性遗忘问题。

关键词: 灾难性遗忘, 联邦类增量学习, 多头自注意力, 标签语义嵌入, 无数据知识蒸馏

CLC Number: