Federated class-incremental learning method of label semantic embedding with multi-head self-attention

doi:10.11772/j.issn.1001-9081.2024101458

Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (10): 3083-3090.DOI: 10.11772/j.issn.1001-9081.2024101458

• Artificial intelligence • Previous Articles

Federated class-incremental learning method of label semantic embedding with multi-head self-attention

Hu WANG¹, Xiaofeng WANG¹^,²(), Ke LI¹, Yunjie MA³

^1.School of Computer Science and Engineering，North Minzu University，Yinchuan Ningxia 750021，China
^2.The Key Laboratory of Images and Graphics Intelligent Processing of State Ethnic Affairs Commission （North Minzu University），Yinchuan Ningxia 750021，China
^3.School of Mathematics and Information Science，North Minzu University，Yinchuan Ningxia 750021，China

Received:2024-10-16 Revised:2024-12-20 Accepted:2024-12-20 Online:2024-12-25 Published:2025-10-10
Contact: Xiaofeng WANG
About author:WANG Hu， born in 1998， M. S. candidate. His research interests include federated learning， knowledge distillation.
WANG Xiaofeng， born in 1980， Ph. D.， associate professor. His research interests include algorithm design and analysis， artificial intelligence.
LI Ke， born in 2000， M. S. candidate. His research interests include federated learning， multi-task learning.
MA Yunjie， born in 1998， M. S. candidate. Her research interests include three-way decision， fuzzy set.
Supported by:
Ningxia Natural Science Foundation(2024AAC03165);Ningxia Youth Top Talent Project(2021)

融合多头自注意力的标签语义嵌入联邦类增量学习方法

王虎¹, 王晓峰¹^,²(), 李可¹, 马云洁³

^1.北方民族大学计算机科学与工程学院，银川 750021
^2.图形图像智能处理国家民委重点实验室（北方民族大学），银川 750021
^3.北方民族大学数学与信息科学学院，银川 750021

通讯作者: 王晓峰
作者简介:王虎（1998—），男，江苏南京人，硕士研究生，CCF会员，主要研究方向：联邦学习、知识蒸馏
王晓峰（1980—），男（回族），甘肃会宁人，副教授，博士，CCF会员，主要研究方向：算法设计与分析、人工智能 Email:xfwang@nmu.edu.cn
李可（2000—），男，河南开封人，硕士研究生，CCF会员，主要研究方向：联邦学习、多任务学习
马云洁（1998—），女，山西晋城人，硕士研究生，CCF会员，主要研究方向：三支决策、模糊集。
基金资助:
宁夏自然科学基金资助项目(2024AAC03165);宁夏自然科学基金资助项目(2024AAC03169);宁夏青年拔尖人才项目(2021)

Abstract

Abstract:

Catastrophic forgetting poses a significant challenge to Federated Class-Incremental Learning （FCIL）， leading to performance degradation of continuous tasks in FCIL. To address this issue， an FCIL method of Label Semantic Embedding （LSE） with Multi-Head Self-Attention （MHSA） — ATTLSE （ATTention Label Semantic Embedding） was proposed. Firstly， an LSE with MHSA was integrated with a generator. Secondly， during the stage of Data-Free Knowledge Distillation （DFKD）， the generator with MHSA was used to produce more meaningful data samples， which guided the training of client models and reduced the influence of catastrophic forgetting problem in FCIL. Experiments were carried out on the CIFAR-100 and Tiny_ImageNet datasets. The results demonstrate that the average accuracy of ATTLSE is improved by 0.06 to 6.45 percentage points compared to LANDER （Label Text Centered Data-Free Knowledge Transfer） method， so as to solve the catastrophic forgetting problem to certain extent of continuous tasks in FCIL.

Key words: catastrophic forgetting, Federated Class-Incremental Learning (FCIL), Multi-Head Self-Attention (MHSA), Label Semantic Embedding (LSE), Data-Free Knowledge Distillation (DFKD)

摘要：

灾难性遗忘对联邦类增量学习（FCIL）构成了显著挑战，导致进行FCIL持续任务时性能下降的问题。针对此问题，提出一种融合多头自注意力（MHSA）的标签语义嵌入（LSE）的FCIL方法——ATTLSE（ATTention Label Semantic Embedding）。首先，融合MHSA的LSE和生成器；其次，在无数据知识蒸馏（DFKD）阶段，依靠融合MHSA的生成器生成更多有意义的数据样本，以指导用户端模型的训练，并降低灾难性遗忘问题在FCIL中的影响。实验结果表明，在CIFAR-100和Tiny_ImageNet数据集上，与LANDER（Label Text Centered Data-Free Knowledge Transfer）方法相比，ATTLSE的平均准确率提升了0.06~6.45个百分点，缓解了持续任务在联邦类增量学习上的灾难性遗忘问题。

关键词: 灾难性遗忘, 联邦类增量学习, 多头自注意力, 标签语义嵌入, 无数据知识蒸馏

CLC Number:

TP181

Hu WANG, Xiaofeng WANG, Ke LI, Yunjie MA. Federated class-incremental learning method of label semantic embedding with multi-head self-attention[J]. Journal of Computer Applications, 2025, 45(10): 3083-3090.

王虎, 王晓峰, 李可, 马云洁. 融合多头自注意力的标签语义嵌入联邦类增量学习方法[J]. 《计算机应用》唯一官方网站, 2025, 45(10): 3083-3090.

Figures/Tables 5

References 31

[1]	NGUYEN D C， DING M， PATHIRANA P N， et al. Federated learning for COVID-19 detection with generative adversarial networks in edge cloud computing［J］. IEEE Internet of Things Journal， 2022， 9（12）： 10257-10271.
[2]	LEE S， CHOI D H. Federated reinforcement learning for energy management of multiple smart homes with distributed energy resources［J］. IEEE Transactions on Industrial Informatics， 2022， 18（1）： 488-497.
[3]	DONG J， WANG L， FANG Z， et al. Federated class-incremental learning［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 10154-10163.
[4]	ZHU X， LI G， HU W. Heterogeneous federated knowledge graph embedding learning and unlearning［C］// Proceedings of the ACM Web Conference 2023. New York： ACM， 2023： 2444-2454.
[5]	LOPES R G， FENU S， STARNER T. Data-free knowledge distillation for deep neural networks［EB/OL］. ［2024-12-18］..
[6]	NAYAK G K， MOPURI K R， SHAJ V， et al. Zero-shot knowledge distillation in deep networks［C］// Proceedings of the 36th International Conference on Machine Learning. New York： JMLR.org， 2019：4743-4751.
[7]	TRAN M T， LE T， LE X M， et al. NAYER： noisy layer data generation for efficient and effective data-free knowledge distillation［C］// Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2024： 23860-23869.
[8]	李雪，姚光乐，王洪辉，等. 基于样本增量学习的遥感影像分类［J］. 计算机应用， 2024， 44（3）： 732-736.
	LI X， YAO G L， WANG H H， et al. Remote sensing image classification based on sample incremental learning［J］. Journal of Computer Applications， 2024， 44（3）： 732-736.
[9]	王虎，王晓峰，李可. 基于潜在空间生成器的联邦知识蒸馏［J］. 计算机应用研究， 2024， 41（11）：3281-3287.
	WANG H， WANG X F， LI K. Knowledge distillation in federated learning based on latent space generator［J］. Application Research of Computers， 2024， 41（11）：3281-3287.
[10]	GAO Q， ZHAO C， GHANEM B， et al. R-DFCIL： relation-guided representation learning for data-free class incremental learning［C］// Proceedings of the 2022 European Conference on Computer Vision， LNCS 13683. Cham： Springer， 2022： 423-439.
[11]	LUO K， WANG S， FU Y， et al. DFRD： data-free robustness distillation for heterogeneous federated learning［C］// Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2023： 17854-17866.
[12]	VASWANI A， SHAZEER N， PARMAR N， et al. Attention is all you need［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2017： 6000-6010.
[13]	KARRAS T， LAINE S， AITTALA M， et al. Analyzing and improving the image quality of StyleGAN［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 8107-8116.
[14]	LEE R， FERNANDEZ-MARQUES J， HU S X， et al. Recurrent early exits for federated learning with heterogeneous clients［C］// Proceedings of the 41st International Conference on Machine Learning. New York： JMLR.org， 2024： 26568-26588.
[15]	YOON J， JEONG W， LEE G， et al. Federated continual learning with weighted inter-client transfer［C］// Proceedings of the 38th International Conference on Machine Learning. New York： JMLR.org， 2021： 12073-12086.
[16]	ZHANG Z， ZHANG Y， GUO D， et al. Communication-efficient federated continual learning for distributed learning system with Non-IID data［J］. SCIENCE CHINA Information Sciences， 2023， 66（2）： No.122102.
[17]	GALLI F， JUNG K， BISWAS S， et al. Advancing personalized federated learning： group privacy， fairness， and beyond［J］. SN Computer Science， 2023， 4（6）： No.831.
[18]	BATRA H， CLARK R. EVCL： elastic variational continual learning with weight consolidation［EB/OL］. ［2024-12-18］..
[19]	MA Y， XIE Z， WANG J， et al. Continual federated learning based on knowledge distillation［C］// Proceedings of the 31st International Joint Conference on Artificial Intelligence. California： ijcai.org， 2022： 2182-2188.
[20]	ZHANG H， XIE Z， ZAREI R， et al. Adaptive client selection in resource constrained federated learning systems： a deep reinforcement learning approach［J］. IEEE Access， 2021， 9： 98423-98432.
[21]	HU K， LU M， LI Y， et al. A federated incremental learning algorithm based on dual attention mechanism［J］. Applied Sciences， 2022， 12（19）： No.10025.
[22]	JIN Z， ZHOU J， LI B， et al. FL-IIDS： a novel federated learning-based incremental intrusion detection system［J］. Future Generations Computer Systems， 2024， 151： 57-70.
[23]	BABAKNIYA S， FABIAN Z， HE C， et al. A data-free approach to mitigate catastrophic forgetting in federated class incremental learning for vision tasks［C］// Proceedings of the 37th International Conference on Neural Information Processing Systems. Red Hook： Curran Associates Inc.， 2023： 66408-66425.
[24]	LE Q， MIKOLOV T. Distributed representations of sentences and documents［C］// Proceedings of the 31st International Conference on Machine Learning. New York： JMLR.org， 2014： 1188-1196.
[25]	KRIZHEVSKY A. Learning multiple layers of features from tiny images［R/OL］. ［2024-12-18］..
[26]	LE Y， YANG X. Tiny ImageNet visual recognition challenge［R/OL］. ［2024-12-18］..
[27]	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778.
[28]	LI Z， HOIEM D. Learning without forgetting［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（12）： 2935-2947.
[29]	KIRKPATRICK J， PASCANU R， RABINOWITZ N， et al. Overcoming catastrophic forgetting in neural networks［J］. Proceedings of the National Academy of Sciences of the United States of America， 2017， 114（13）： 3521-3526.
[30]	ZHANG J， CHEN C， ZHUANG W， et al. TARGET： federated class-continual learning via exemplar-free distillation［C］// Proceedings of the 2023 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE， 2023： 4759-4770.
[31]	TRAN M T， LE T， LE X M， et al. Text-enhanced data-free approach for federated class-incremental learning［C］// Proceedings of the 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2024： 23870-23880.

狄利克雷分布	任务数	平均准确率/%
		CIFAR-100						Tiny_ImageNet
		Finetune	FedEWC	FedLwF	TARGET	LANDER	ATTLSE	Finetune	FedEWC	FedLwF	TARGET	LANDER	ATTLSE
1.0	5	17.88	24.02	45.67	30.50	57.13	59.07	12.22	15.14	26.46	11.28	23.65	25.81
1.0	8	11.02	15.40	35.70	17.75	43.58	45.42	7.57	9.93	19.65	6.83	8.71	9.93
0.5	5	17.23	21.40	45.98	23.40	50.68	57.13	11.94	14.99	26.85	11.26	18.03	19.89
0.5	8	11.12	13.77	34.02	14.94	38.22	43.39	7.37	9.06	21.15	6.41	7.46	9.71
0.1	5	15.37	18.32	40.75	24.03	47.65	50.02	11.28	13.50	24.67	9.70	13.74	19.89
0.1	8	9.81	11.24	29.26	15.49	34.70	36.61	7.01	8.54	18.05	5.97	3.77	3.83

狄利克雷分布	任务数	平均准确率/%
		CIFAR-100						Tiny_ImageNet
		Finetune	FedEWC	FedLwF	TARGET	LANDER	ATTLSE	Finetune	FedEWC	FedLwF	TARGET	LANDER	ATTLSE
1.0	5	17.88	24.02	45.67	30.50	57.13	59.07	12.22	15.14	26.46	11.28	23.65	25.81
1.0	8	11.02	15.40	35.70	17.75	43.58	45.42	7.57	9.93	19.65	6.83	8.71	9.93
0.5	5	17.23	21.40	45.98	23.40	50.68	57.13	11.94	14.99	26.85	11.26	18.03	19.89
0.5	8	11.12	13.77	34.02	14.94	38.22	43.39	7.37	9.06	21.15	6.41	7.46	9.71
0.1	5	15.37	18.32	40.75	24.03	47.65	50.02	11.28	13.50	24.67	9.70	13.74	19.89
0.1	8	9.81	11.24	29.26	15.49	34.70	36.61	7.01	8.54	18.05	5.97	3.77	3.83

任务数	平均准确率/%
	CIFAR-100		Tiny_ImageNet
	ATTLSE*	ATTLSE	ATTLSE*	ATTLSE
1	84.25	84.83	63.76	64.40
2	75.25	74.50	44.24	45.68
3	66.83	66.39	35.41	36.93
4	59.77	60.48	25.90	28.10
5	55.70	57.13	18.83	19.89
6	48.71	50.82	17.09	18.00
7	46.90	48.85	11.94	13.65
8	41.30	43.39	8.92	9.71

任务数	平均准确率/%
	CIFAR-100		Tiny_ImageNet
	ATTLSE*	ATTLSE	ATTLSE*	ATTLSE
1	84.25	84.83	63.76	64.40
2	75.25	74.50	44.24	45.68
3	66.83	66.39	35.41	36.93
4	59.77	60.48	25.90	28.10
5	55.70	57.13	18.83	19.89
6	48.71	50.82	17.09	18.00
7	46.90	48.85	11.94	13.65
8	41.30	43.39	8.92	9.71

Federated class-incremental learning method of label semantic embedding with multi-head self-attention

融合多头自注意力的标签语义嵌入联邦类增量学习方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 5

References 31

Related Articles 2

Recommended Articles

Metrics

[1]	Pengcheng SONG, Lijun GUO, Rong ZHANG. Weakly supervised video anomaly detection with local-global temporal dependency [J]. Journal of Computer Applications, 2025, 45(1): 240-246.
[2]	Yichi CHEN, Bin CHEN. Review of lifelong learning in computer vision [J]. Journal of Computer Applications, 2023, 43(6): 1785-1795.