Federated learning algorithm for communication cost optimization

doi:10.11772/j.issn.1001-9081.2021122054

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (1): 1-7.DOI: 10.11772/j.issn.1001-9081.2021122054

Special Issue: 人工智能

• Artificial intelligence • Next Articles

Federated learning algorithm for communication cost optimization

ZHENG Sai^1,2, LI Tianrui^1,2, HUANG Wei^1,2

1.School of Computing and Artificial Intelligence， Southwest Jiaotong University， Chengdu Sichuan 611756， China
2.Sichuan Key Laboratory of Cloud Computing and Intelligent Technique （Southwest Jiaotong University）， Chengdu Sichuan 611756， China

Received:2021-12-08 Revised:2022-01-24 Online:2022-03-02
Contact: LI Tianrui， born in 1969， Ph. D.， professor. His research interests include big data， cloud computing， data mining， machine learning， granular computing， rough sets.
About author:ZHENG Sai， born in 1996， M. S. candidate. His research interests include federated learning， machine learning；LI Tianrui， born in 1969， Ph. D.， professor. His research interests include big data， cloud computing， data mining， machine learning， granular computing， rough sets；HUANG Wei， born in 1994， Ph. D. candidate. Her research interests include federated learning， data mining；
Supported by:
This work is partially supported by National Key Research and Development Program of China （2019YFB2101802）， National Natural Science Foundation of China （62176221）.

面向通信成本优化的联邦学习算法

郑赛^1,2, 李天瑞^1,2, 黄维^1,2

1.西南交通大学计算机与人工智能学院，成都 611756
2.云计算与智能技术四川省高校重点实验室（西南交通大学），成都 611756

通讯作者: 李天瑞（1969—），男，福建莆田人，教授，博士，CCF杰出会员，主要研究方向：大数据、云计算、数据挖掘、机器学习、粒度计算、粗糙集trli@swjtu.edu.cn
作者简介:郑赛（1996—），男，浙江金华人，硕士研究生，主要研究方向：联邦学习、机器学习；李天瑞（1969—），男，福建莆田人，教授，博士，CCF杰出会员，主要研究方向：大数据、云计算、数据挖掘、机器学习、粒度计算、粗糙集；黄维（1994—），女，福建莆田人，博士研究生，主要研究方向：联邦学习、机器学习、数据挖掘；
基金资助:
国家重点研发计划项目（2019YFB2101802）；国家自然科学基金资助项目（62176221）。

Abstract

Abstract: Federated Learning （FL） is a machine learning setting that can protect data privacy， however， the problems of high communication cost and client heterogeneity hinder the large?scale implementation of federated learning. To solve these two problems， a federated learning algorithm for communication cost optimization was proposed. First， the generative models from the clients were received and simulated data were generated by the server. Then， the simulated data were used by the server to train the global model and send it to the clients， and the final models were obtained by the clients through fine?tuning the global model. In the proposed algorithm only one round of communication between clients and the server was needed， and the fine?tuning of the client models was used to solve the problem of client heterogeneity. When the number of clients is 20， experiments were carried out on MNIST and CIFAR?10 dataset. The results show that the proposed algorithm can reduce the amount of communication data to 1/10 of that of Federated Averaging （FedAvg） algorithm on the MNIST dataset， and can reduce the amount of communication data to 1/100 of that of Federated Averaging （FedAvg） algorithm on the CIFAR-10 dataset with the premise of ensuring accuracy.

Key words: Federated Learning (FL), optimization algorithm, communication cost, single?round communication, generative model

摘要： 联邦学习是一种能够保护数据隐私的机器学习设置，然而高昂的通信成本和客户端的异质性问题阻碍了联邦学习的规模化落地。针对这两个问题，提出一种面向通信成本优化的联邦学习算法。首先，服务器接收来自客户端的生成模型并生成模拟数据；然后，服务器利用模拟数据训练全局模型并将其发送给客户端，客户端利用全局模型进行微调后得到最终模型。所提算法仅需要客户端与服务器之间的一轮通信，并且利用微调客户端模型来解决客户端异质性问题。在客户端数量为20个时，在MNIST和CIFAR?10这两个数据集上进行了实验。结果表明，所提算法能够在保证准确率的前提下，在MNIST数据集上将通信的数据量减少至联邦平均（FedAvg）算法的1/10，在CIFAR-10数据集上将通信数据量减少至FedAvg算法的1/100。

关键词: 联邦学习, 优化算法, 通信成本, 单轮通信, 生成模型

CLC Number:

TP181

ZHENG Sai, LI Tianrui, HUANG Wei. Federated learning algorithm for communication cost optimization[J]. Journal of Computer Applications, 2023, 43(1): 1-7.

郑赛, 李天瑞, 黄维. 面向通信成本优化的联邦学习算法[J]. 《计算机应用》唯一官方网站, 2023, 43(1): 1-7.

References

1 LeCUN Y， BENGIO Y， HINTON G. Deep learning［J］. Nature， 2015， 521（7553）： 436-444. 10.1038/nature14539
2 McMAHAN H B， MOORE E， RAMAGE D， et al. Communication? efficient learning of deep networks from decentralized data［C］// Proceedings of the 20th International Conference on Artificial Intelligence and Statistics. New York： JMLR.org， 2017： 1273-1282.
3 KAIROUZ P， MCMAHAN H B， AVENT B， et al. Advances and open problems in federated learning［J］. Foundations and Trends? in Machine Learning， 2021， 14（1/2）： 1-210. 10.1561/2200000083
4 HARD A， RAO K， MATHEWS R， et al. Federated learning for mobile keyboard prediction［EB/OL］. （2019-02-28）［2021-03-28］.https：//arxiv.org/pdf/1811.03604.pdf.
5 史鼎元，王晏晟，郑鹏飞，等. 面向企业数据孤岛的联邦排序学习［J］. 软件学报， 2021， 32（3）： 669-688. 10.13328/j.cnki.jos.006174 SHI D Y， WANG Y S， ZHENG P F， et al. Cross?silo federated learning?to?rank［J］. Journal of Software， 2021， 32（3）： 669-688. 10.13328/j.cnki.jos.006174
6 MUHAMMAD K， WANG Q Q， O'REILLY?MORGAN D， et al. FedFast： going beyond average for faster training of federated recommender systems［C］// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York： ACM， 2020： 1234-1242. 10.1145/3394486.3403176
7 LI T， SAHU A K， TALWALKAR A， et al. Federated learning： challenges， methods， and future directions［J］. IEEE Signal Processing Magazine， 2020， 37（3）： 50-60. 10.1109/msp.2020.2975749
8 LI T， SAHU A K， ZAHEER M， et al. Federated optimization in heterogeneous networks［C/OL］// Proceedings of the 3rd Machine Learning and Systems Conference. ［2022-01-21］.https：//proceedings.mlsys.org/paper/2020/file/38af86134b65d0f10fe33d30 dd76442e-Paper.pdf. 10.1109/ieeeconf44664.2019.9049023
9 KULKARNI V， KULKARNI M， PANT A. Survey of personalization techniques for federated learning［C］// Proceedings of the 4th World Conference on Smart Trends in Systems， Security and Sustainability. Piscataway： IEEE， 2020： 794-797. 10.1109/worlds450073.2020.9210355
10 PAN S J， YANG Q. A survey on transfer learning［J］. IEEE Transactions on Knowledge and Data Engineering， 2010， 22（10）： 1345-1359. 10.1109/tkde.2009.191
11 HINTON G， VINYALS O， DEAN J. Distilling the knowledge in a neural network［EB/OL］. （2015-03-09）［2021-04-23］.https：//arxiv.org/pdf/1503.02531.pdf.
12 KHODAK M， BALCAN M F， TALWALKAR A. Adaptive gradient?based meta?learning methods［C/OL］// Proceedings of the 33rd Conference on Neural Information Processing Systems. ［2021-04-23］.https：//proceedings.neurips.cc/paper/2019/file/f4aa0dd960521e045ae2f20621fb4ee9-Paper.pdf.
13 SMITH V， CHIANG C K， SANJABI M， et al. Federated multi? task learning［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 4427-4437.
14 WANG K K， MATHEWS R， KIDDON C， et al. Federated evaluation of on?device personalization［EB/OL］. （2019-10-22）［2022-01-22］.https：//arxiv.org/pdf/1910.10252.pdf.
15 YU T， BAGDASARYAN E， SHMATIKOV V. Salvaging federated learning by local adaptation［EB/OL］. （2022-03-03）［2022-03-22］.https：//arxiv.org/pdf/2002.04758.pdf.
16 LI D L， WANG J P. FedMD： heterogenous federated learning via model distillation［EB/OL］. （2019-10-08）［2022-01-19］.https：//arxiv.org/pdf/1910.03581.pdf.
17 YAO X， HUANG C F， SUN L F. Two?stream federated learning： reduce the communication costs［C］// Proceedings of the 2018 IEEE International Conference on Visual Communications and Image Processing. Piscataway： IEEE， 2018： 1-4. 10.1109/vcip.2018.8698609
18 CALDAS S， KONE?NY J， McMAHAN H B， et al. Expanding the reach of federated learning by reducing client resource requirements［EB/OL］. （2019-01-08）［2021-03-09］.https：//arxiv.org/pdf/1812.07210.pdf.
19 SRIVASTAVA N， HINTON G， KRIZHEVSKY A， et al. Dropout： a simple way to prevent neural networks from overfitting［J］. Journal of Machine Learning Research， 2014， 15： 1929-1958.
20 芦效峰，廖钰盈， LIO P，等. 一种面向边缘计算的高效异步联邦学习机制［J］. 计算机研究与发展， 2020， 57（12）： 2571-2582. 10.7544/issn1000-1239.2020.20190754 LU X F， LIAO Y Y， LIO P， et al. An asynchronous federated learning mechanism for edge network computing［J］. Journal of Computer Research and Development， 2020， 57（12）： 2571-2582. 10.7544/issn1000-1239.2020.20190754
21 CHEN Y， SUN X， JIN Y. Communication?efficient federated deep learning with layerwise asynchronous model update and temporally weighted aggregation［J］. IEEE Transactions on Neural Networks and Learning Systems， 2019， 31（10）： 4229-4238. 10.1109/tnnls.2019.2953131
22 LIAN X R， ZHANG C， ZHANG H， et al. Can decentralized algorithms outperform centralized algorithms？ a case study for decentralized parallel stochastic gradient descent［C］// Proceedings of the 31st Annual Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017： 5336-5346. 10.1109/yac.2016.7804900
23 GOODFELLOW I J， POUGET?ABADIE J， MIRZA M， et al. Generative adversarial nets［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2014， 2： 2672-2680.
24 KINGMA D P， WELLING M. Auto?encoding variational Bayes［EB/OL］. （2014-05-011）［2021-03-06］.https：//arxiv.org/pdf/1312.6114.pdf.
25 ARJOVSKY M， CHINTALA S， BOTTOU L. Wasserstein generative adversarial networks［C］// Proceedings of the 34th International Conference on Machine Learning. New York： JMLR.org， 2017： 214-223.
26 MIRZA M， OSINDERO S. Conditional generative adversarial nets［EB/OL］. （2014-11-06）［2021-03-09］.https：//arxiv.org/pdf/1411.1784.pdf.
27 YOSINSKI J， CLUNE J， BENGIO Y， et al. How transferable are features in deep neural networks？［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2014： 3320-3328.
28 HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
29 ACAR D A E， ZHAO Y， NAVARRO R M， et al. Federated learning based on dynamic regularization［EB/OL］. （2021-11-09）［2021-12-13］.https：//arxiv.org/pdf/2111.04263.pdf.

[1]	Zheyuan SHEN, Keke YANG, Jing LI. Personalized federated learning method based on dual stream neural network [J]. Journal of Computer Applications, 2024, 44(8): 2319-2325.
[2]	Kaiwen ZHAO, Peng WANG, Xiangrong TONG. Two-stage search-based constrained evolutionary multitasking optimization algorithm [J]. Journal of Computer Applications, 2024, 44(5): 1415-1422.
[3]	Xiuxi WEI, Maosong PENG, Huajuan HUANG. Node coverage optimization of wireless sensor network based on multi-strategy improved butterfly optimization algorithm [J]. Journal of Computer Applications, 2024, 44(4): 1009-1017.
[4]	Yaqin CHEN, Peng WANG. Potential barrier estimation criterion based on quantum dynamics framework of optimization algorithm [J]. Journal of Computer Applications, 2024, 44(4): 1180-1186.
[5]	Lingling GUO, Zhiqiang LI, Menghuan DUAN. Application of quantum approximate optimization algorithm in exact cover problems [J]. Journal of Computer Applications, 2024, 44(3): 849-854.
[6]	Xuebin CHEN, Changsheng QU. Overview of backdoor attacks and defense in federated learning [J]. Journal of Computer Applications, 2024, 44(11): 3459-3469.
[7]	Jie WU, Xuezhong QIAN, Wei SONG. Personalized federated learning based on similarity clustering and regularization [J]. Journal of Computer Applications, 2024, 44(11): 3345-3353.
[8]	Chunyong YIN, Yongcheng ZHOU. Automatically adjusted clustered federated learning for double-ended clustering [J]. Journal of Computer Applications, 2024, 44(10): 3011-3020.
[9]	Hui ZHOU, Yuling CHEN, Xuewei WANG, Yangwen ZHANG, Jianjiang HE. Deep shadow defense scheme of federated learning based on generative adversarial network [J]. Journal of Computer Applications, 2024, 44(1): 223-232.
[10]	Bona XUAN, Jin LI, Yafei SONG, Zexuan MA. Malicious code classification method based on improved MobileNetV2 [J]. Journal of Computer Applications, 2023, 43(7): 2217-2225.
[11]	Mengjie LAN, Jianping CAI, Lan SUN. Self-regularization optimization methods for Non-IID data in federated learning [J]. Journal of Computer Applications, 2023, 43(7): 2073-2081.
[12]	Wanzhen CHEN, En ZHANG, Leiyong QIN, Shuangxi HONG. Privacy-preserving federated learning algorithm based on blockchain in edge computing [J]. Journal of Computer Applications, 2023, 43(7): 2209-2216.
[13]	Heming JIA, Shanglong LI, Lizhen CHEN, Qingxin LIU, Di WU, Rong ZHENG. Remora optimization algorithm based on chaotic host switching mechanism [J]. Journal of Computer Applications, 2023, 43(6): 1759-1767.
[14]	Lin SUN, Jinxu HUANG, Jiucheng XU. Feature selection for imbalanced data based on neighborhood tolerance mutual information and whale optimization algorithm [J]. Journal of Computer Applications, 2023, 43(6): 1842-1854.
[15]	Zhihui GAO, Meng HAN, Shujuan LIU, Ang LI, Dongliang MU. Survey of high utility itemset mining methods based on intelligent optimization algorithm [J]. Journal of Computer Applications, 2023, 43(6): 1676-1686.

Federated learning algorithm for communication cost optimization

面向通信成本优化的联邦学习算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics