Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (7): 2073-2081.DOI: 10.11772/j.issn.1001-9081.2022071122
Special Issue: 第39届CCF中国数据库学术会议(NDBC 2022)
• The 39th CCF National Database Conference (NDBC 2022) • Previous Articles Next Articles
Mengjie LAN, Jianping CAI(), Lan SUN
Received:
2022-07-12
Revised:
2022-08-15
Accepted:
2022-08-17
Online:
2023-07-20
Published:
2023-07-10
Contact:
Jianping CAI
About author:
LAN Mengjie, born in 1998, M. S. candidate. Her research interests include federated learning, differential privacy.通讯作者:
蔡剑平
作者简介:
蓝梦婕(1998—),女,福建三明人,硕士研究生,CCF学生会员,主要研究方向:联邦学习、差分隐私;CLC Number:
Mengjie LAN, Jianping CAI, Lan SUN. Self-regularization optimization methods for Non-IID data in federated learning[J]. Journal of Computer Applications, 2023, 43(7): 2073-2081.
蓝梦婕, 蔡剑平, 孙岚. 非独立同分布数据下的自正则化联邦学习优化方法[J]. 《计算机应用》唯一官方网站, 2023, 43(7): 2073-2081.
Add to citation manager EndNote|Ris|BibTeX
URL: https://www.joca.cn/EN/10.11772/j.issn.1001-9081.2022071122
变量名 | 描述 | 默认值 |
---|---|---|
学习率 | 0.01 | |
batch size | 64 | |
本地更新Epoch次数 | 20 | |
客户端总数 | 1 000 | |
第 | 10 | |
训练总轮次 | 200 | |
狄利克雷分布参数 | 0.5 |
Tab. 1 Experimental parameters
变量名 | 描述 | 默认值 |
---|---|---|
学习率 | 0.01 | |
batch size | 64 | |
本地更新Epoch次数 | 20 | |
客户端总数 | 1 000 | |
第 | 10 | |
训练总轮次 | 200 | |
狄利克雷分布参数 | 0.5 |
数据集 | 正确率/% | |||||
---|---|---|---|---|---|---|
FedAvg | FedProx | SCAFFOLD | FedSR | Dyn-FedSR | ||
MNIST | 88.58 | 88.45 | 88.58 | 89.28 | 89.35 | |
89.33 | 88.87 | 89.32 | 89.91 | 89.83 | ||
89.70 | 89.35 | 89.64 | 90.17 | 90.13 | ||
Fashion-MNIST | 70.32 | 76.86 | 71.52 | 81.19 | 81.10 | |
78.58 | 78.40 | 78.54 | 81.12 | 81.10 | ||
81.63 | 81.70 | 82.08 | 81.78 | 81.80 | ||
EMNIST | 47.15 | 47.02 | 47.55 | 54.65 | 54.18 | |
56.51 | 56.43 | 56.19 | 61.68 | 61.49 | ||
57.59 | 58.55 | 58.34 | 63.38 | 63.52 | ||
CIFAR-10 | 38.59 | 39.23 | 39.12 | 40.17 | 40.07 | |
40.25 | 40.27 | 41.38 | 42.55 | 42.61 | ||
41.35 | 41.72 | 41.43 | 43.03 | 43.38 |
Tab. 2 Accuracy comparison under different Non-IID levels
数据集 | 正确率/% | |||||
---|---|---|---|---|---|---|
FedAvg | FedProx | SCAFFOLD | FedSR | Dyn-FedSR | ||
MNIST | 88.58 | 88.45 | 88.58 | 89.28 | 89.35 | |
89.33 | 88.87 | 89.32 | 89.91 | 89.83 | ||
89.70 | 89.35 | 89.64 | 90.17 | 90.13 | ||
Fashion-MNIST | 70.32 | 76.86 | 71.52 | 81.19 | 81.10 | |
78.58 | 78.40 | 78.54 | 81.12 | 81.10 | ||
81.63 | 81.70 | 82.08 | 81.78 | 81.80 | ||
EMNIST | 47.15 | 47.02 | 47.55 | 54.65 | 54.18 | |
56.51 | 56.43 | 56.19 | 61.68 | 61.49 | ||
57.59 | 58.55 | 58.34 | 63.38 | 63.52 | ||
CIFAR-10 | 38.59 | 39.23 | 39.12 | 40.17 | 40.07 | |
40.25 | 40.27 | 41.38 | 42.55 | 42.61 | ||
41.35 | 41.72 | 41.43 | 43.03 | 43.38 |
算法 | MNIST | Fashion-MNIST | EMNIST | CIFAR-10 | ||||
---|---|---|---|---|---|---|---|---|
轮次 | 加速比 | 轮次 | 加速比 | 轮次 | 加速比 | 轮次 | 加速比 | |
FedAvg | 100 | 1.00 | 100 | 1.00 | 100 | 1.00 | 100 | 1.00 |
FedProx | 76 | 1.32 | 66 | 1.52 | 82 | 1.22 | 73 | 1.37 |
SCAFFOLD | 70 | 1.43 | 61 | 1.64 | 92 | 1.09 | 73 | 1.37 |
FedSR | 66 | 1.52 | 40 | 2.50 | 80 | 1.25 | 55 | 1.82 |
Dyn-FedSR | 61 | 1.64 | 34 | 2.94 | 73 | 1.37 | 61 | 1.64 |
Tab. 3 Communication efficiency comparison under default parameter setting
算法 | MNIST | Fashion-MNIST | EMNIST | CIFAR-10 | ||||
---|---|---|---|---|---|---|---|---|
轮次 | 加速比 | 轮次 | 加速比 | 轮次 | 加速比 | 轮次 | 加速比 | |
FedAvg | 100 | 1.00 | 100 | 1.00 | 100 | 1.00 | 100 | 1.00 |
FedProx | 76 | 1.32 | 66 | 1.52 | 82 | 1.22 | 73 | 1.37 |
SCAFFOLD | 70 | 1.43 | 61 | 1.64 | 92 | 1.09 | 73 | 1.37 |
FedSR | 66 | 1.52 | 40 | 2.50 | 80 | 1.25 | 55 | 1.82 |
Dyn-FedSR | 61 | 1.64 | 34 | 2.94 | 73 | 1.37 | 61 | 1.64 |
训练间隔轮次 | 正确率/% | ||
---|---|---|---|
FedSR | Dyn-FedSR | FedAvg | |
55 | 89.91 | 89.83 | 89.33 |
25 | 88.50 | 88.45 | 88.36 |
14 | 87.51 | 87.46 | 87.43 |
9 | 87.10 | 87.05 | 86.94 |
Tab. 4 Influence of different training interval on accuracy
训练间隔轮次 | 正确率/% | ||
---|---|---|---|
FedSR | Dyn-FedSR | FedAvg | |
55 | 89.91 | 89.83 | 89.33 |
25 | 88.50 | 88.45 | 88.36 |
14 | 87.51 | 87.46 | 87.43 |
9 | 87.10 | 87.05 | 86.94 |
1 | McMAHAN H B, MOORE E, RAMAGE D, et al. Communication-efficient learning of deep networks from decentralized data [C]// Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. New York: JMLR.org, 2017: 1273-1282. |
2 | TAN B, LIU B, ZHENG V, et al. A federated recommender system for online services [C]// Proceedings of the 14th ACM Conference on Recommender Systems. New York: ACM, 2020: 579-581. 10.1145/3383313.3411528 |
3 | MUHAMMAD K, WANG Q Q, O’REILLY-MORGAN D, et al. FedFast: going beyond average for faster training of federated recommender systems [C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2020: 1234-1242. 10.1145/3394486.3403176 |
4 | CHEN Y Q, QIN X, WANG J D, et al. FedHealth: a federated transfer learning framework for wearable healthcare[J]. IEEE Intelligent Systems, 2020, 35 (4): 83-93. 10.1109/mis.2020.2988604 |
5 | BRISIMI T S, CHEN R D, MELA T, et al. Federated learning of predictive models from federated electronic health records[J]. International Journal of Medical Informatics, 2018, 112: 59-67. 10.1016/j.ijmedinf.2018.01.007 |
6 | JIANG J C, KANTARCI B, OKTUG S, et al. Federated learning in smart city sensing: challenges and opportunities[J]. Sensors, 2020, 20 (21): No.6230. 10.3390/s20216230 |
7 | KAIROUZ P, McMAHAN H B, AVENT B, et al. Advances and open problems in federated learning[J]. Foundations and Trends in Machine Learning, 2021, 14 (1/2): 1-210. |
8 | LI X, HUANG K X, YANG W H, et al. On the convergence of FedAvg on Non-IID data[EB/OL]. (2020-06-25) [2022-05-06]. . |
9 | GAO D S, LIU Y, HUANG A B, et al. Privacy-preserving heterogeneous federated transfer learning [C]// Proceedings of the 2019 IEEE International Conference on Big Data. Piscataway: IEEE, 2019: 2552-2559. 10.1109/bigdata47090.2019.9005992 |
10 | LIU Y, KANG Y, XING C P, et al. A secure federated transfer learning framework[J]. IEEE Intelligent Systems, 2020, 35 (4): 70-82. 10.1109/mis.2020.2988525 |
11 | SMITH V, CHIANG C K, SANJABI M, et al. Federated multi-task learning [C]// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2017: 4427-4437. |
12 | JIANG Y H, KONEČNÝ J, RUSH K, et al. Improving federated learning personalization via model agnostic meta learning[EB/OL]. (2019-09-27) [2022-05-06]. . |
13 | KHODAK M, BALCAN M F F, TALWALKAR A. Adaptive gradient-based meta-learning methods [C]// Proceedings of the 33rd International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2019: 5917-5928. |
14 | FALLAH A, MOKHTARI A, OZDAGLAR A. Personalized federated learning with theoretical guarantees: a model-agnostic meta-learning approach [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2020: 3557-3568. |
15 | ZHAO Y, LI M, LAI L Z, et al. Federated learning with non-IID data[EB/OL]. (2022-07-21) [2022-08-06]. . |
16 | LI T, SAHU A K, ZAHEER M, et al. Federated optimization in heterogeneous networks[C/OL]// Proceedings of the 3rd Machine Learning and Systems Conference [2022-05-06]. . 10.1109/ieeeconf44664.2019.9049023 |
17 | KARIMIREDDY S P, KALE S, MOHRI M, et al. SCAFFOLD: stochastic controlled averaging for federated learning [C]// Proceedings of the 37th International Conference on Machine Learning. New York: JMLR.org, 2020: 5132-5143. |
18 | LI Q B, HE B S, SONG D. Model-contrastive federated learning [C]// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2021: 10708-10717. 10.1109/cvpr46437.2021.01057 |
19 | ACAR D A E, ZHAO Y, NAVARRO R M, et al. Federated learning based on dynamic regularization[EB/OL]. (2021-11-09) [2022-08-06]. . |
20 | WANG J Y, LIU Q H, LIANG H, et al. Tackling the objective inconsistency problem in heterogeneous federated optimization [C]// Proceedings of the 34th International Conference on Neural Information Processing Systems. Red Hook, NY: Curran Associates Inc., 2020: 7611-7623. |
[1] | Zheyuan SHEN, Keke YANG, Jing LI. Personalized federated learning method based on dual stream neural network [J]. Journal of Computer Applications, 2024, 44(8): 2319-2325. |
[2] | Xuebin CHEN, Changsheng QU. Overview of backdoor attacks and defense in federated learning [J]. Journal of Computer Applications, 2024, 44(11): 3459-3469. |
[3] | Shuaihua ZHANG, Shufen ZHANG, Mingchuan ZHOU, Chao XU, Xuebin CHEN. Malicious traffic detection model based on semi-supervised federated learning [J]. Journal of Computer Applications, 2024, 44(11): 3487-3494. |
[4] | Jie WU, Xuezhong QIAN, Wei SONG. Personalized federated learning based on similarity clustering and regularization [J]. Journal of Computer Applications, 2024, 44(11): 3345-3353. |
[5] | Chunyong YIN, Yongcheng ZHOU. Automatically adjusted clustered federated learning for double-ended clustering [J]. Journal of Computer Applications, 2024, 44(10): 3011-3020. |
[6] | Hui ZHOU, Yuling CHEN, Xuewei WANG, Yangwen ZHANG, Jianjiang HE. Deep shadow defense scheme of federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(1): 223-232. |
[7] | Jianyang CUI, Ying CAI, Yu ZHANG, Yanfang FAN. Authenticatable privacy-preserving scheme based on signcryption from lattice for vehicular ad hoc network [J]. Journal of Computer Applications, 2024, 44(1): 233-241. |
[8] | Wanzhen CHEN, En ZHANG, Leiyong QIN, Shuangxi HONG. Privacy-preserving federated learning algorithm based on blockchain in edge computing [J]. Journal of Computer Applications, 2023, 43(7): 2209-2216. |
[9] | Chunyong YIN, Rui QU. Federated learning algorithm based on personalized differential privacy [J]. Journal of Computer Applications, 2023, 43(4): 1160-1168. |
[10] | Teng WANG, Zheng HUO, Yaxin HUANG, Yilin FAN. Review on privacy-preserving technologies in federated learning [J]. Journal of Computer Applications, 2023, 43(2): 437-449. |
[11] | Qian CHEN, Zheng CHAI, Zilong WANG, Jiawei CHEN. Poisoning attack detection scheme based on generative adversarial network for federated learning [J]. Journal of Computer Applications, 2023, 43(12): 3790-3798. |
[12] | Wenbo LI, Bo LIU, Lingling TAO, Fen LUO, Hang ZHANG. Deep spectral clustering algorithm with L1 regularization [J]. Journal of Computer Applications, 2023, 43(12): 3662-3667. |
[13] | Guishuang XU, Xinchun YIN. Certificateless conditional privacy-preserving authentication scheme for VANET [J]. Journal of Computer Applications, 2023, 43(11): 3358-3367. |
[14] | ZHENG Sai, LI Tianrui, HUANG Wei. Federated learning algorithm for communication cost optimization [J]. Journal of Computer Applications, 2023, 43(1): 1-7. |
[15] | Hongliang LI, Nong ZHANG, Ting SUN, Xiang LI. Performance interference analysis and prediction for distributed machine learning jobs [J]. Journal of Computer Applications, 2022, 42(6): 1649-1655. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||