Gradient descent with momentum algorithm based on differential privacy in convolutional neural network

doi:10.11772/j.issn.1001-9081.2022121881

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (12): 3647-3653.DOI: 10.11772/j.issn.1001-9081.2022121881

Special Issue: 人工智能

• Artificial intelligence • Next Articles

Gradient descent with momentum algorithm based on differential privacy in convolutional neural network

Yu ZHANG, Ying CAI(), Jianyang CUI, Meng ZHANG, Yanfang FAN

Computer School，Beijing Information Science and Technology University，Beijing 100101，China

Received:2022-12-26 Revised:2023-03-19 Accepted:2023-03-24 Online:2023-04-12 Published:2023-12-10
Contact: Ying CAI
About author:ZHANG Yu， born in 1997， M. S. candidate. Her research interests include deep learning， differential privacy.
CUI Jianyang， born in 1996， M. S. candidate. His research interests include vehicular ad hoc network， privacy protection.
ZHANG Meng，born in 1996， M. S. candidate. His research interests include image retrieval， privacy protection.
FAN Yanfang， born in 1979， Ph. D， associate professor. Her research interests include information security， internet of vehicles， edge computing.
Supported by:
Natural Science Foundation of Beijing-Haidian Original Innovation Joint Fund(L192023)

卷积神经网络中基于差分隐私的动量梯度下降算法

张宇, 蔡英(), 崔剑阳, 张猛, 范艳芳

北京信息科技大学计算机学院，北京 100101

通讯作者: 蔡英
作者简介:张宇（1997—），女，河北石家庄人，硕士研究生，主要研究方向：深度学习、差分隐私
崔剑阳（1996—），男（满族），河北承德人，硕士研究生，主要研究方向：车载自组织网络、隐私保护
张猛（1996—），男，河北定州人，硕士研究生，主要研究方向：图像检索、隐私保护
范艳芳（1979—），女，山西运城人，副教授，博士，主要研究方向：信息安全、车联网、边缘计算。
基金资助:
北京市自然科学基金-海淀原始创新联合基金资助项目(L192023)

Abstract

Abstract:

To address the privacy leakage problem caused by the model parameters memorizing some features of the data during the training process of the Convolutional Neural Network （CNN） models， a Gradient Descent with Momentum algorithm based on Differential Privacy in CNN （DPGDM） was proposed. Firstly， the Gaussian noise meeting differential privacy was added to the gradient in the backpropagation process of model optimization， and the noise-added gradient value was used to participate in the model parameter update process， so as to achieve differential privacy protection for the overall model. Secondly， to reduce the impact of the introduction of differential privacy noise on convergence speed of the model， a learning rate decay strategy was designed and then the gradient descent with momentum algorithm was improved. Finally， to reduce the influence of noise on the accuracy of the model， the value of the noise scale was adjusted dynamically during model optimization， thereby changing the amount of noise that needs to be added to the gradient in each round of iteration. Experimental results show that compared with DP-SGD （Differentially Private Stochastic Gradient Descent） algorithm， the proposed algorithm can improve the accuracy of the model by about 5 and 4 percentage points at privacy budget of 0.3 and 0.5， respectively， proving that by using the proposed algorithm， the model usability is improved and privacy protection of the model is achieved.

Key words: Convolutional Neural Network (CNN), differential privacy, gradient descent with momentum algorithm, deep learning, privacy protection

摘要：

针对卷积神经网络（CNN）模型的训练过程中，模型参数记忆数据部分特征导致的隐私泄露问题，提出一种CNN中基于差分隐私的动量梯度下降算法（DPGDM）。首先，在模型优化的反向传播过程中对梯度添加满足差分隐私的高斯噪声，并用加噪后的梯度值参与模型参数的更新过程，从而实现对模型整体的差分隐私保护；其次，为了减少引入差分隐私噪声对模型收敛速度的影响，设计学习率衰减策略，改进动量梯度下降算法；最后，为了降低噪声对模型准确率的影响，在模型优化过程中动态地调整噪声尺度的值，从而改变在每一轮迭代中需要对梯度加入的噪声量。实验结果表明，与DP-SGD （Differentially Private Stochastic Gradient Descent）相比，所提算法可以在隐私预算为0.3和0.5时，模型准确率分别提高约5和4个百分点。可见，所提算法提高了模型的可用性，并实现了对模型的隐私保护。

关键词: 卷积神经网络, 差分隐私, 动量梯度下降算法, 深度学习, 隐私保护

CLC Number:

TP309

Yu ZHANG, Ying CAI, Jianyang CUI, Meng ZHANG, Yanfang FAN. Gradient descent with momentum algorithm based on differential privacy in convolutional neural network[J]. Journal of Computer Applications, 2023, 43(12): 3647-3653.

张宇, 蔡英, 崔剑阳, 张猛, 范艳芳. 卷积神经网络中基于差分隐私的动量梯度下降算法[J]. 《计算机应用》唯一官方网站, 2023, 43(12): 3647-3653.

Figures/Tables 10

Tab. 1 Experimental parameters

参数名	不同实验数据集下的参数值
参数名	MNIST	Fashion-MNIST	CIFAR-10
批处理数据的样本数N	250	256	1 500
模型训练轮次	100	100	300
学习率初始值 $η 0$	0.04	0.04	0.20
噪声尺度初始值 $σ 0$	2	2	15
噪声尺度最小值 $σ m i n$	0.18	0.16	0.10
动量超参数m	0.9	0.9	0.9

Tab. 1 Experimental parameters

参数名	不同实验数据集下的参数值
参数名	MNIST	Fashion-MNIST	CIFAR-10
批处理数据的样本数N	250	256	1 500
模型训练轮次	100	100	300
学习率初始值 $η 0$	0.04	0.04	0.20
噪声尺度初始值 $σ 0$	2	2	15
噪声尺度最小值 $σ m i n$	0.18	0.16	0.10
动量超参数m	0.9	0.9	0.9

Fig.1 Accuracy comparison of different algorithms on MNIST dataset

Fig.2 Accuracy comparison among different algorithms under different privacy budgets on MNIST dataset

Fig.3 Model accuracy comparison among different algorithms under different amount of noise

Fig.4 Comparison of model accuracy under different learning rates

Tab.2 Accuracy comparison of different algorithms on Fashion-MNIST dataset

算法	准确率	损失的准确率
NO-PRIVACY	89.82	0.00
DPGDM	85.55	4.27
DP-SGD	78.80	11.02
DP-PSO	81.40	8.42

Fig.5 Model accuracy comparison among different algorithms under different privacy budgets on Fashion-MNIST dataset

Tab. 3 Accuracy comparison of different algorithms on CIFAR-10 dataset

算法	准确率	损失的准确率
NO-PRIVACY	70.12	0.00
DPGDM	68.72	1.40
DP-SGD	64.59	5.53
DP-SGD with Tempered Sigmoid^［20］	66.20	3.92

Fig.6 Comparison of model accuracy on CIFAR-10 dataset

Fig.7 Loss function changes on CIFAR-10 dataset

References 29

1	ALZUBAIDI L， ZHANG J， HUMAIDI A J， et al. Review of deep learning： concepts， CNN architectures， challenges， applications， future directions［J］. Journal of Big Data， 2021， 8： Article No. 53. 10.1186/s40537-021-00444-8
2	SUN Y， XUE B， ZHANG M， et al. Automatically designing CNN architectures using the genetic algorithm for image classification［J］. IEEE Transactions on Cybernetics， 2020， 50（9）： 3840-3854. 10.1109/tcyb.2020.2983860
3	季长清，高志勇，秦静，等.基于卷积神经网络的图像分类算法综述［J］.计算机应用，2022，42（4）：1044-1049. 10.11772/j.issn.1001-9081.2021071273
	JI C Q， GAO Z Y， QIN J， et al. Review of image classification algorithms based on convolutional neural network［J］. Journal of Computer Applications， 2022，42（4）：1044-1049. 10.11772/j.issn.1001-9081.2021071273
4	HUSAIN S S， BOBER M. REMAP： multi-layer entropy-guided pooling of dense CNN features for image retrieval［J］. IEEE Transactions on Image Processing， 2019， 28（10）： 5201-5213. 10.1109/tip.2019.2917234
5	FREDRIKSON M， JHA S， RISTENPART T. Model inversion attacks that exploit confidence information and basic countermeasures［C］// Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. New York： ACM， 2015： 1322-1333. 10.1145/2810103.2813677
6	HERNANDEZ MARCANO N J， MOLLER M， HANSEN S， et al. On fully homomorphic encryption for privacy-preserving deep learning ［C］// Proceedings of the 2019 IEEE Globecom Workshops. Piscataway： IEEE， 2019： 1-6. 10.1109/gcwkshps45667.2019.9024625
7	A-T TRAN， T-D LUONG， KARNJANA J， et al. An efficient approach for privacy preserving decentralized deep learning models based on secure multi-party computation［J］. Neurocomputing， 2021， 422： 245-262. 10.1016/j.neucom.2020.10.014
8	MEDEN B， EMERŠIČ Ž， ŠTRUC V， et al. k-Same-Net： k-anonymity with generative deep neural networks for face deidentification ［J］. Entropy， 2018， 20（1）： 60. 10.3390/e20010060
9	DWORK C. Differential privacy［C］// Proceedings of the 33rd International Colloquium on Automata， Languages and Programming. Berlin： Springer， 2006： 1-12. 10.1007/11787006_1
10	CAI Y， ZHANG Y， QU J， et al. Differential privacy preserving dynamic data release scheme based on Jensen-Shannon divergence［J］. China Communications， 2022，19（6）：11-21. 10.23919/jcc.2022.06.002
11	屈晶晶，蔡英，范艳芳，等. 基于k-prototype聚类的差分隐私混合数据发布算法［J］. 计算机科学与探索， 2021， 15（1）：109-118. 10.3778/j.issn.1673-9418.2003048
	QU J J， CAI Y， FAN Y F， et al. Differentially private mixed data release algorithm based on k-prototype clustering［J］. Journal of Frontiers of Computer Science and Technology， 2021，15（1）：109-118. 10.3778/j.issn.1673-9418.2003048
12	ZHANG Y， CAI Y， ZHANG M， et al. A survey on privacy-preserving deep learning with differential privacy ［C］// Proceedings of the 2021 International Conference on Big Data and Security. Singapore： Springer， 2022： 18-30. 10.1007/978-981-19-0852-1_2
13	SHOKRI R， SHMATIKOV V. Privacy-preserving deep learning ［C］// Proceedings of the 22nd ACM SIGSAC Conference on Computer and Communications Security. New York： ACM， 2015： 1310-1321. 10.1145/2810103.2813687
14	ABADI M， CHU A， GOODFELLOW I， et al. Deep learning with differential privacy［C］// Proceedings of the 2016 ACM SIGSAC Conference on Computer and Communications Security. New York： ACM， 2016： 308-318. 10.1145/2976749.2978318
15	YUAN D， ZHU X， WEI M， et al. Collaborative deep learning for medical image analysis with differential privacy ［C］// Proceedings of the 2019 IEEE Global Communications Conference. Piscataway： IEEE， 2019： 1-6. 10.1109/globecom38437.2019.9014259
16	ARACHCHIGE P C M， BERTOK P， KHALIL I， et al. Local differential privacy for deep learning ［J］. IEEE Internet of Things Journal， 2019， 7（7）： 5827-5842. 10.1109/jiot.2019.2952146
17	GONG M， PAN K， XIE Y， et al. Preserving differential privacy in deep neural networks with relevance-based adaptive noise imposition［J］. Neural Networks， 2020， 125： 131-141. 10.1016/j.neunet.2020.02.001
18	YU L， LIU L， PU C， et al. Differentially private model publishing for deep learning ［C］// Proceedings of the 2019 IEEE Symposium on Security and Privacy. Piscataway： IEEE， 2019： 332-349. 10.1109/sp.2019.00019
19	ZILLER A， USYNIN D， BRAREN R， et al. Medical imaging deep learning with differential privacy［J］. Scientific Reports， 2021， 11： Article No. 13524. 10.1038/s41598-021-93030-0
20	PAPERNOT N， THAKURTA A， SONG S， et al. Tempered sigmoid activations for deep learning with differential privacy［J］. Proceedings of the AAAI Conference on Artificial Intelligence， 2021， 35（10）： 9312-9321. 10.1609/aaai.v35i10.17123
21	李敏，李红娇，陈杰.差分隐私保护下的Adam优化算法研究［J］.计算机应用与软件，2020，37（6）：253-258，296. 10.3969/j.issn.1000-386x.2020.06.044
	LI M， LI H J， CHEN J. Adam optimization algorithm based on differential privacy protection［J］. Computer Applications and Software， 2020，37（6）：253-258，296. 10.3969/j.issn.1000-386x.2020.06.044
22	余方超，方贤进，张又文，等.增强深度学习中的差分隐私防御机制［J］.南京大学学报（自然科学），2021，57（1）：10-20.
	YU F C， FANG X J， ZHANG Y W， et al. Enhanced differential privacy defense mechanism in deep learning［J］. Journal of Nanjing University （Natural Science）， 2021，57（1）：10-20.
23	YAMASHITA R， NISHIO M， DO R K G， et al. Convolutional neural networks： an overview and application in radiology［J］. Insights into Imaging， 2018， 9（4）： 611-629. 10.1007/s13244-018-0639-9
24	KATTENBORN T， LEITLOFF J， SCHIEFER F， et al. Review on Convolutional Neural Networks （CNN） in vegetation remote sensing［J］. ISPRS Journal of Photogrammetry and Remote Sensing， 2021， 173： 24-49. 10.1016/j.isprsjprs.2020.12.010
25	KIRANYAZ S， AVCI O， ABDELJABER O， et al. 1D convolutional neural networks and applications： a survey［J］. Mechanical Systems and Signal Processing， 2021， 151： 107398. 10.1016/j.ymssp.2020.107398
26	MIRONOV I. Rényi differential privacy［C］// Proceedings of the 2017 IEEE 30th Computer Security Foundations Symposium. Piscataway： IEEE， 2017： 263-275. 10.1109/csf.2017.11
27	谭作文，张连福.机器学习隐私保护研究综述［J］.软件学报，2020，31（7）：2127-2156. 10.13328/j.cnki.jos.006052
	TAN Z W， ZHANG L F. Survey on privacy preserving techniques for machine learning ［J］. Journal of Software， 2020，31（7）：2127-2156. 10.13328/j.cnki.jos.006052
28	YOUSEFPOUR A， SHILOV I， SABLAYROLLES A， et al. Opacus： user-friendly differential privacy library in PyTorch ［EB/OL］. ［2022-08-22］..
29	张攀峰，吴丹华，董明刚. 基于粒子群优化的差分隐私深度学习模型［J］. 计算机工程， 2023，49（9）： 144-157. 10.19678/j.issn.1000-3428.0065590
	ZHANG P F， WU D H， DONG M G. Differential privacy deep learning model based on particle swarm optimization ［J］. Computer Engineering， 2023，49（9）： 144-157. 10.19678/j.issn.1000-3428.0065590

[1]	Zhizheng ZHANG, Xiaojian ZHANG, Junqing WANG, Guanghui FENG. Federated spatial data publication method with differential privacy and secure aggregation [J]. Journal of Computer Applications, 2024, 44(9): 2777-2784.
[2]	Tingwei CHEN, Jiacheng ZHANG, Junlu WANG. Random validation blockchain construction for federated learning [J]. Journal of Computer Applications, 2024, 44(9): 2770-2776.
[3]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[4]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[5]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[6]	Yun LI, Fuyou WANG, Peiguang JING, Su WANG, Ao XIAO. Uncertainty-based frame associated short video event detection method [J]. Journal of Computer Applications, 2024, 44(9): 2903-2910.
[7]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[8]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[9]	Hong CHEN, Bing QI, Haibo JIN, Cong WU, Li’ang ZHANG. Class-imbalanced traffic abnormal detection based on 1D-CNN and BiGRU [J]. Journal of Computer Applications, 2024, 44(8): 2493-2499.
[10]	Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557.
[11]	Yanjie GU, Yingjun ZHANG, Xiaoqian LIU, Wei ZHOU, Wei SUN. Traffic flow forecasting via spatial-temporal multi-graph fusion [J]. Journal of Computer Applications, 2024, 44(8): 2618-2625.
[12]	Qianhong SHI, Yan YANG, Yongquan JIANG, Xiaocao OUYANG, Wubo FAN, Qiang CHEN, Tao JIANG, Yuan LI. Multi-granularity abrupt change fitting network for air quality prediction [J]. Journal of Computer Applications, 2024, 44(8): 2643-2650.
[13]	Zheng WU, Zhiyou CHENG, Zhentian WANG, Chuanjian WANG, Sheng WANG, Hui XU. Deep learning-based classification of head movement amplitude during patient anaesthesia resuscitation [J]. Journal of Computer Applications, 2024, 44(7): 2258-2263.
[14]	Dongwei WANG, Baichen LIU, Zhi HAN, Yanmei WANG, Yandong TANG. Deep network compression method based on low-rank decomposition and vector quantization [J]. Journal of Computer Applications, 2024, 44(7): 1987-1994.
[15]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.

Gradient descent with momentum algorithm based on differential privacy in convolutional neural network

卷积神经网络中基于差分隐私的动量梯度下降算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 29

Related Articles 15

Recommended Articles

Metrics