Multi-model collaborative continual learning framework based on filtering mechanism

doi:10.11772/j.issn.1001-9081.2025091087

Abstract

Abstract:

To address the difficulty of balancing stability and plasticity in the existing continual learning methods， a multi-model collaborative continual learning framework based on filtering mechanism was proposed. The framework is composed of a main model， an auxiliary model， and a student model. The main model was designed to focus on predicting and outputting high-confidence results， the auxiliary model was used to memorize hard samples that the main model failed to learn， and the student model was employed to inherit and integrate the recognition ability of a cascaded teacher model constructed from the main and auxiliary models. The three models were optimized collaboratively during each task process. Experimental results show that in task-incremental scenarios， the proposed framework achieves higher AAC （Average ACcuracy）， BWT （BackWard Transfer）， and FWT （ForWard Transfer） than typical methods such as EWC （Elastic Weight Consolidation）， GEM （Gradient Episodic Memory）， and iCaRL （incremental Classifier and Representation Learning） on both the Fashion-MNIST and CIFAR-10 datasets， demonstrating stronger stable memory and adaptive capabilities； in class-incremental scenarios， the framework has good applicability on both the 10-class Fashion-MNIST and the 100-class CIFAR-100 datasets， with its AAC higher than that of the comparative methods significantly.

Key words: continuous learning, catastrophic forgetting, Knowledge Distillation (KD), multi-model, stability, plasticity

摘要：

针对现有的持续学习方法在稳定性与可塑性之间通常难以兼顾的问题，提出一种基于筛选机制的多模型协同持续学习框架。该框架由主模型、辅助模型与学生模型组成：主模型着眼于预测输出高可信度结果，辅助模型记忆主模型难以学习的困难样本，学生模型则继承与整合主模型和辅助模型级联构成的教师模型的识别能力，三者在每一轮任务过程中协同优化。实验结果表明，在任务增量场景下，该框架在Fashion-MNIST和CIFAR-10数据集上的平均准确率（AAC）、BWT（BackWard Transfer）、FWT（ForWard Transfer）等指标均优于EWC（Elastic Weight Consolidation）、GEM（Gradient Episodic Memory）、iCaRL（incremental Classifier and Representation Learning）等典型方法，体现出更强的稳定记忆与灵活适应能力；在类增量场景下，无论是只有10类的Fashion-MNIST还是100类的CIFAR-100数据集，该框架均有良好的适用性，它的AAC亦显著高于对比方法。

关键词: 持续学习, 灾难性遗忘, 知识蒸馏, 多模型, 稳定性, 可塑性

CLC Number:

TP391.4

Yuzhen WU, Chen PAN, Qiannan LI. Multi-model collaborative continual learning framework based on filtering mechanism[J]. Journal of Computer Applications, 2026, 46(10): 3093-3100.

吴雨臻, 潘晨, 栗倩楠. 基于筛选机制的多模型协同持续学习框架[J]. 《计算机应用》唯一官方网站, 2026, 46(10): 3093-3100.

Figures/Tables 12

References 37

[1]	Wang L， Zhang X， Su H， et al. A comprehensive survey of continual learning： theory， method and application［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2024， 46（8）： 5362-5383.
[2]	McCloskey M， Cohen N J. Catastrophic interference in connectionist networks： the sequential learning problem［J］. Psychology of Learning and Motivation， 1989， 24： 109-165.
[3]	Ganaie M A， Hu M， Malik A K， et al. Ensemble deep learning： a review［J］. Engineering Applications of Artificial Intelligence， 2022， 115： No.105151.
[4]	Parisi G I， Kemker R， Part J L， et al. Continual lifelong learning with neural networks： a review［J］. Neural Networks， 2019， 113： 54-71.
[5]	de Lange M， Aljundi R， Masana M， et al. A continual learning survey： defying forgetting in classification tasks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2022， 44（7）： 3366-3385.
[6]	Kirkpatrick J， Pascanu R， Rabinowitz N， et al. Overcoming catastrophic forgetting in neural networks［J］. Proceedings of the National Academy of Sciences of the United States of America， 2017， 114（13）： 3521-3526.
[7]	Zenke F， Poole B， Ganguli S. Continual learning through synaptic intelligence［J］. Proceedings of Machine Learning Research， 2017， 70： 3987-3995.
[8]	Aljundi R， Babiloni F， Elhoseiny M， et al. Memory aware synapses： learning what （not） to forget［C］// ECCV 2018， LNCS 11207. Cham： Springer， 2018： 144-161.
[9]	Kumar S， Marklund H， van Roy B. Maintaining plasticity in continual learning via regenerative regularization［PP/OL］. V3. arXiv （2024-10-03）［2025-09-16］..
[10]	Rebuffi S A， Kolesnikov A， Sperl G， et al. iCaRL： Incremental classifier and representation learning［C］// CVPR 2017. Piscataway： IEEE， 2017： 5533-5542.
[11]	Lopez-Paz D， Ranzato M. Gradient episodic memory for continual learning［C］// NeurIPS 2017. Red Hook： Curran Associates Inc.， 2017： 6470-6479.
[12]	Bellitto G， Salanitri F P， Pennisi M， et al. Saliency-driven experience replay for continual learning［C］// NeurIPS 2024. Red Hook： Curran Associates Inc.， 2024： 103356-103383.
[13]	Smith J S， Valkov L， Halbe S， et al. Adaptive memory replay for continual learning［C］// CVPRW 2024. Piscataway： IEEE， 2024： 3605-3615.
[14]	Gou J， Yu B， Maybank S J， et al. Knowledge distillation： a survey［J］. International Journal of Computer Vision， 2021， 129（6）： 1789-1819.
[15]	Li Z， Hoiem D. Learning without forgetting［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2018， 40（12）： 2935-2947.
[16]	Heo B， Kim J， Yun S， et al. A comprehensive overhaul of feature distillation［C］// ICCV 2019. Piscataway： IEEE， 2019： 1921-1930.
[17]	Guo Z， Yan H， Li H， et al. Class attention transfer based knowledge distillation［C］// CVPR 2023. Piscataway： IEEE， 2023： 11868-11877.
[18]	Huang G. Dynamic neural networks： advantages and challenges［J］. National Science Review， 2024， 11（8）： No.nwae088.
[19]	Mu S， Lin S. A comprehensive survey of mixture-of-experts： algorithms， theory， and applications［PP/OL］. V3. arXiv （2025-04-18）［2025-09-28］..
[20]	Hinton G， Vinyals O， Dean J. Distilling the knowledge in a neural network［PP/OL］. arXiv （2015-03-09）［2025-09-16］..
[21]	LeCun Y， Bottou L， Bengio Y， et al. Gradient-based learning applied to document recognition［J］. Proceedings of the IEEE， 1998， 86（11）： 2278-2324.
[22]	Xiao H， Rasul K， Vollgraf R. Fashion-MNIST： a novel image dataset for benchmarking machine learning algorithms［PP/OL］. V2. arXiv （2017-09-15）［2025-09-16］..
[23]	Krizhevsky A. Learning multiple layers of features from tiny images［R/OL］. （2009-04-08）［2025-09-16］..
[24]	Krizhevsky A， Sutskever I， Hinton G E. ImageNet classification with deep convolutional neural networks［C］// NeurIPS 2012 — Volume 1. Red Hook： Curran Associates Inc.， 2012： 1097-1105.
[25]	He K， Zhang X， Ren S， et al. Deep residual learning for image recognition［C］// CVPR 2016. Piscataway： IEEE， 2016： 770-778.
[26]	Buzzega P， Boschini M， Porrello A， et al. Dark experience for general continual learning： a strong， simple baseline［C］// NeurIPS 2020. Red Hook： Curran Associates Inc.， 2020： 15920-15930.
[27]	Dohare S， Hernandez-Garcia J F， Lan Q， et al. Loss of plasticity in deep continual learning［J］. Nature， 2024， 632（8026）： 768-774.
[28]	Younis S， Seeger B. Data-free generative replay for class-incremental learning on imbalanced data［PP/OL］. arXiv （2024-06-07）［2025-10-05］..
[29]	Xin X， Zhong Y， Hou Y， et al. Memory-free generative replay for class-incremental learning［PP/OL］. arXiv （2021-09-01）［2025-10-05］..
[30]	Smith J， Hsu Y C， Balloch J， et al. Always be dreaming： a new approach for data-free class-incremental learning［C］// ICCV 2021. Piscataway： IEEE， 2021： 9354-9364.
[31]	Zhu F， Zhang X Y， Wang C， et al. Prototype augmentation and self-supervision for incremental learning［C］// CVPR 2021. Piscataway： IEEE， 2021： 5867-5876.
[32]	Zhu F， Cheng Z， Zhang X Y， et al. Class-incremental learning via dual augmentation［C］// NeurIPS 2021. Red Hook： Curran Associates Inc.， 2021： 14306-14318.
[33]	Gao Q， Zhao C， Ghanem B， et al. R-DFCIL： relation guided representation learning for data-free class incremental learning［C］// ECCV 2022， LNCS 13683. Cham： Springer， 2022： 423-439.
[34]	Zhu K， Zhai W， Cao Y， et al. Self-sustaining representation expansion for non-exemplar class-incremental learning［C］// CVPR 2022. Piscataway： IEEE， 2022： 9286-9295.
[35]	Petit G， Popescu A， Schindler H， et al. FeTriL： feature translation for exemplar-free class-incremental learning［C］// WACV 2023. Piscataway： IEEE， 2023： 3900-3909.
[36]	Rypeść G， Cygert S， Khan V， et al. Divide and not forget： ensemble of selectively trained experts in continual learning［PP/OL］. V3. arXiv （2024-03-19）［2025-09-28］..
[37]	Meng Z， Zhang J， Yang C， et al. DiffClass： diffusion-based class incremental learning［C］// ECCV 2024， LNCS 15145. Cham： Springer， 2025： 142-159.

模型	10 000张测试样本上的准确率
模型	AlexNet	ResNet-18	ResNet-50
Model_1	99.34	99.38	99.22
Model_1+Model_2	99.49	99.55	99.39
Model_3	99.55	99.53	99.50
新Model_1+新Model_2	99.67	99.56	99.58

模型	10 000张测试样本上的准确率
模型	AlexNet	ResNet-18	ResNet-50
Model_1	99.34	99.38	99.22
Model_1+Model_2	99.49	99.55	99.39
Model_3	99.55	99.53	99.50
新Model_1+新Model_2	99.67	99.56	99.58

迭代轮数	训练数据规模/10³	困难样本数据规模
迭代轮数	训练数据规模/10³	AlexNet	ResNet-18	ResNet-50
第一轮	20	243	152	168
第二轮	30	62	8	48

迭代轮数	训练数据规模/10³	困难样本数据规模
迭代轮数	训练数据规模/10³	AlexNet	ResNet-18	ResNet-50
第一轮	20	243	152	168
第二轮	30	62	8	48

方法	Fashion-MNIST			CIFAR-10
方法	AAC	BWT	FWT	AAC	BWT	FWT
EWC^［6］	0.937 5	-0.063 7	0.798 7	0.903 5	-0.075 5	0.717 9
GEM^［11］	0.953 7	-0.051 4	0.794 8	0.919 6	-0.065 9	0.729 6
iCaRL^［10］	0.912 3	-0.033 5	0.768 1	0.879 0	-0.070 2	0.694 5
DER^［26］	0.950 9	-0.054 1	0.818 4	0.926 8	-0.070 5	0.749 8
CBP^［27］	0.942 4	-0.064 0	0.794 7	0.909 6	-0.071 6	0.720 5
本文框架	0.955 0	-0.050 0	0.834 4	0.927 6	-0.064 5	0.748 9