Efficient homomorphic neural network supporting privacy-preserving training

doi:10.11772/j.issn.1001-9081.2021101775

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (12): 3792-3800.DOI: 10.11772/j.issn.1001-9081.2021101775

Special Issue: 网络空间安全

• Cyber security • Previous Articles Next Articles

Efficient homomorphic neural network supporting privacy-preserving training

Yang ZHONG¹^,², Renwan BI¹^,², Xishan YAN¹, Zuobin YING³, Jinbo XIONG¹^,²()

^1.College of Computer and Cyber Security，Fujian Normal University，Fuzhou Fujian 350117，China
^2.Fujian Provincial Key Lab of Network Security and Cryptology （Fujian Normal University），Fuzhou Fujian 350007，China
^3.Faculty of Data Science，City University of Macau，Macau 999078，China

Received:2021-10-15 Revised:2022-01-18 Accepted:2022-01-24 Online:2022-03-04 Published:2022-12-10
Contact: Jinbo XIONG
About author:ZHONG Yang， born in 1995， M. S. candidate. His research interests include secure deep learning.
BI Renwan， born in 1996， Ph. D. candidate. His research interests include secure deep learning， secure multi-party computing.
YAN Xishan， born in 1979， Ph. D. candidate. His research interests include information security.
YING Zuobin， born in 1982， Ph. D.， assistant professor. His research interests include security for cloud computing.
Supported by:
National Natural Science Foundation of China(61872088)

支持隐私保护训练的高效同态神经网络

钟洋¹^,², 毕仁万¹^,², 颜西山¹, 应作斌³, 熊金波¹^,²()

^1.福建师范大学计算机与网络空间安全学院, 福州 350117
^2.福建省网络安全与密码技术重点实验室(福建师范大学), 福州 350007
^3.澳门城市大学数据科学学院, 澳门 999078

通讯作者: 熊金波
作者简介:钟洋（1995—），男，湖南湘西人，硕士研究生，CCF会员，主要研究方向：安全深度学习
毕仁万（1996—），男，湖南常德人，博士研究生，CCF会员，主要研究方向：安全深度学习、安全多方计算
颜西山（1979—），男，江西吉安人，博士研究生，主要研究方向：信息安全
应作斌（1982—），男，安徽芜湖人，助理教授，博士，主要研究方向：云计算安全
基金资助:
国家自然科学基金资助项目(61872088)

Abstract

Abstract:

Aiming at the problems of low computational efficiency and insufficient accuracy in the privacy-preserving neural network based on homomorphic encryption， an efficient Homomorphic Neural Network （HNN） under three-party collaborative supporting privacy-preserving training was proposed. Firstly， in order to reduce the computational cost of ciphertext-ciphertext multiplication in homomorphic encryption， the idea of secret sharing was combined to design a secure fast multiplication protocol to convert the ciphertext-ciphertext multiplication into plaintext-ciphertext multiplication with low complexity. Then， in order to avoid multiple iterations of ciphertext polynomials generated during the construction of HNN and improve the nonlinear calculation accuracy， a secure nonlinear calculation method was studied， which executed the corresponding nonlinear operator for the confused plaintext message with random mask. Finally， the security， correctness and efficiency of the proposed protocols were analyzed theoretically， and the effectiveness and superiority of HNN were verified by experiments. Experimental results show that compared with the dual server scheme PPML （Privacy Protection Machine Learning）， HNN has the training efficiency improved by 18.9 times and the model accuracy improved by 1.4 percentage points.

Key words: homomorphic encryption, privacy-preserving, neural network, model training, nonlinear calculation

摘要：

针对基于同态加密的隐私保护神经网络中存在的计算效率低和精度不足问题，提出一种三方协作下支持隐私保护训练的高效同态神经网络（HNN）。首先，为降低同态加密中密文乘密文运算产生的计算开销，结合秘密共享思想设计了一种安全快速的乘法协议，将密文乘密文运算转换为复杂度较低的明文乘密文运算；其次，为避免构建HNN时产生的密文多项式多轮迭代，并提高非线性计算精度，研究了一种安全的非线性计算方法，从而对添加随机掩码的混淆明文消息执行相应的非线性算子；最后，对所设计协议的安全性、正确性及效率进行了理论分析，并对HNN的有效性及优越性进行了实验验证。实验结果表明，相较于双服务器方案PPML，HNN的训练速度提高了18.9倍，模型精度提高了1.4个百分点。

关键词: 同态加密, 隐私保护, 神经网络, 模型训练, 非线性计算

CLC Number:

TP309.2

Yang ZHONG, Renwan BI, Xishan YAN, Zuobin YING, Jinbo XIONG. Efficient homomorphic neural network supporting privacy-preserving training[J]. Journal of Computer Applications, 2022, 42(12): 3792-3800.

钟洋, 毕仁万, 颜西山, 应作斌, 熊金波. 支持隐私保护训练的高效同态神经网络[J]. 《计算机应用》唯一官方网站, 2022, 42(12): 3792-3800.

Figures/Tables 10

Fig.1 System model

Fig. 2 Overview of HNN scheme

Tab.1 Computational complexity of secure forward propagation process

网络层	PMult	CMult	Add	Enc	Dec
$F C 1$	$k m$	0	$(k + 1) m$	0	0
ReLU	0	$3 m$	$m$	$m$	$m$
$F C 2$	$m t$	0	$(m + 1) t$	0	0
Softmax	0	$3 t$	$2 t$	$5 t$	$2 t$

Tab.1 Computational complexity of secure forward propagation process

网络层	PMult	CMult	Add	Enc	Dec
$F C 1$	$k m$	0	$(k + 1) m$	0	0
ReLU	0	$3 m$	$m$	$m$	$m$
$F C 2$	$m t$	0	$(m + 1) t$	0	0
Softmax	0	$3 t$	$2 t$	$5 t$	$2 t$

Tab.2 Computational complexity of secure back propagation process

网络层	PMult	CMult	Add
$F C 1$	$t (m + n)$	$m$	$t m$
$F C 2$	$m n$	0	0
Softmax	0	0	$t$

Tab.2 Computational complexity of secure back propagation process

网络层	PMult	CMult	Add
$F C 1$	$t (m + n)$	$m$	$t m$
$F C 2$	$m n$	0	0
Softmax	0	0	$t$

Tab.3 Communication complexity of secure computing protocols

协议	通信轮数			消息大小
协议	$S 1$ 与 $S 2$	$S 1$ 与 $S 3$	$S 2$ 与 $S 3$	消息大小
SFMP	1	2	1	$3 P + C$
SRP	1	2	2	$P + 5 C$
SREP	1	2	2	$P + 4 C$
SEP	1	2	2	$P + 4 C$

Tab.3 Communication complexity of secure computing protocols

协议	通信轮数			消息大小
协议	$S 1$ 与 $S 2$	$S 1$ 与 $S 3$	$S 2$ 与 $S 3$	消息大小
SFMP	1	2	1	$3 P + C$
SRP	1	2	2	$P + 5 C$
SREP	1	2	2	$P + 4 C$
SEP	1	2	2	$P + 4 C$

Tab. 4 Comparison of training time of different bit-widths

比特长度	运行时间/s
80	148
128	220
176	431

Tab.5 Comparison of computational cost between CCM and PCM operations

乘法深度	PCM/ms	CCM/ms	倍数
3	3.60	41.3	11.4
5	11.01	98.9	9.0
7	12.10	123.9	10.2
9	13.80	150.2	10.8

Tab. 6 Comparison of computational cost among different schemes

网络	方案	批量大小	批量样本时间/10³s	单张样本时间/s
N1	PPML^［15］	192	10.470	54.56
N1	HNN	192	0.550	2.88
N2	FHESGD^［23］	60	118.000	1 966.00
N2	HNN	192	0.118	0.61
N3	Glyph^［17］	60	2.900	48.33
N3	HNN	192	0.220	1.14

Tab.7 Comparison of model accuracy among different schemes

方案	迭代次数	模型精度/%
FHESGD^［23］	5	93.0
PPML^［15］	5	95.0
Glyph^［17］	5	96.4
HNN	3	96.4

Fig. 3 Model accuracy comparison between HNN scheme and neural network in plaintext environment

References 23

1	耿艺宁，刘帅师，刘泰廷，等. 基于计算机视觉的行人检测技术综述［J］. 计算机应用， 2021， 41（S1）： 43-50.
	GENG Y N， LIU S S， LIU T Y， et al. Survey of pedestrian detection technology based on computer vision［J］. Journal of Computer Applications， 2021， 41（S1）： 43-50.
2	LUO X Y， LI L， WAN H， et al. Phone Keypad Voice Recognition （PKVR）： an integrated experiment for digital signal processing education［C］// Proceedings of the 2020 IEEE Frontiers in Education Conference. Piscataway： IEEE， 2020： 1-4. 10.1109/fie44824.2020.9273815
3	GUO Y Y， BI L， AHN E， et al. A spatiotemporal volumetric interpolation network for 4D dynamic medical image［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 4725-4734. 10.1109/cvpr42600.2020.00478
4	徐占洋，郑克长. 云计算下基于改进遗传算法的聚类融合算法［J］. 计算机应用， 2018， 38（2）： 458-463. 10.11772/j.issn.1001-9081.2017071749
	XU Z Y， ZHEN K C. Clustering ensemble algorithm based on improved genetic algorithm in cloud computing［J］. Journal of Computer Applications， 2018， 38（2）： 458-463. 10.11772/j.issn.1001-9081.2017071749
5	熊金波，毕仁万，陈前昕，等. 边缘协作的轻量级安全区域建议网络［J］. 通信学报， 2020， 41（10）： 188-201.
	XIONG J B， BI R W， CHEN Q X， et al. Towards edge-collaborative， lightweight and secure region proposal network［J］. Journal on Communications， 2020， 41（10）： 188-201.
6	WONG J C. The Cambridge Analytica scandal changed the world — but it didn’t change Facebook［N/OL］. GuardianThe. （2019-03-18）［2021-10-12］..
7	HARWELL D. Thousands of zoom video calls left exposed on open web［N/OL］. The Washington Post. （2020-04-03）［2021-10-12］..
8	毕仁万，陈前昕，熊金波，等. 面向深度神经网络的安全计算协议设计方法［J］. 网络与信息安全学报， 2020， 6（4）： 130-139. 10.11959/j.issn.2096-109x.2020050
	BI R W， CHEN Q X， XIONG J B， et al. Design method of secure computing protocol for deep neural network［J］. Chinese Journal of Network and Information Security， 2020， 6（4）： 130-139. 10.11959/j.issn.2096-109x.2020050
9	XIE P T， BILENKO M， FINLEY T， et al. Crypto-nets： neural networks over encrypted data［EB/OL］. （2020-04-03）［2021-10-12］..
10	GILAD-BACHRACH R， DOWLIN N， LAINE K， et al. CryptoNets： applying neural networks to encrypted data with high throughput and accuracy［C］// Proceedings of the 33rd International Conference on Machine Learning. New York： JMLR.org， 2016： 201-210.
11	CHABANNE H， DE WARGNY A， MILGRAM J， et al. Privacy-preserving classification on deep neural network［EB/OL］. （2017-03-24）［2021-10-12］..
12	BADAWI A AL， JIN C， LIN J， et al. Towards the AlexNet moment for homomorphic encryption： HCNN， the first homomorphic CNN on encrypted data with GPUs［J］. IEEE Transactions on Emerging Topics in Computing， 2021， 9（3）： 1330-1343. 10.1109/tetc.2020.3014636
13	XIONG J B， BI R W， TIAN Y L， et al. Toward lightweight， privacy-preserving cooperative object classification for connected autonomous vehicles［J］. IEEE Internet of Things Journal， 2022， 9（4）： 2787-2801. 10.1109/jiot.2021.3093573
14	ZHANG Q C， YANG L T， CHEN Z K. Privacy preserving deep computation model on cloud for big data feature learning［J］. IEEE Transactions on Computers， 2016， 65（5）： 1351-1362. 10.1109/tc.2015.2470255
15	HESAMIFARD E， TAKABI H， GHASEMI M， et al. Privacy-preserving machine learning in cloud［C］// Proceedings of the 2017 Cloud Computing Security Workshop. New York： ACM， 2017： 39-43. 10.1145/3140649.3140655
16	BOURSE F， MINELLI M， MINIHOLD M， et al. Fast homomorphic evaluation of deep discretized neural networks［C］// Proceedings of the 2018 Annual International Cryptology Conference， LNCS 10993. Cham： Springer， 2018： 483-512.
17	LOU Q， FENG B， FOX G C， et al. Glyph： fast and accurately training deep neural networks on encrypted data［C/OL］// Proceedings of the 34th Conference on Neural Information Processing Systems. ［2021-09-20］.. 10.1109/iconip.2002.1198214
18	BRAKERSKI Z， VAIKUNTANATHAN V. Efficient fully homomorphic encryption from （standard） LWE［J］. SIAM Journal on Computing， 2014， 43（2）： 831-871. 10.1137/120868669
19	LYUBASHEVSKY V， PEIKERT C， REGEV O. On ideal lattices and learning with errors over rings［C］// Proceedings of the 2010 Annual International Conference on the Theory and Applications of Cryptographic Techniques， LNCS 6110. Berlin： Springer， 2010： 1-23.
20	LIU X M， QIN B D， DENG R H， et al. An efficient privacy-preserving outsourced computation over public data［J］. IEEE Transactions on Services Computing， 2017， 10（5）： 756-770. 10.1109/tsc.2015.2511008
21	XIONG J B， BI R W， ZHAO M F， et al. Edge-assisted privacy-preserving raw data sharing framework for connected autonomous vehicles［J］. IEEE Wireless Communications， 2020， 27（3）： 24-30. 10.1109/mwc.001.1900463
22	NANDAKUMAR K， RATHA N， PANKANTI S， et al. Towards deep neural network training on encrypted data［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2019： 40-48. 10.1109/cvprw.2019.00011
23	HALEVI S， SHOUP V. Design and implementation of HElib： a homomorphic encryption library［EB/OL］. （2020-12-25）［2021-10-12］.. 10.1007/s00145-020-09368-7

[1]	Guanglei YAO, Juxia XIONG, Guowu YANG. Flower pollination algorithm based on neural network optimization [J]. Journal of Computer Applications, 2024, 44(9): 2829-2837.
[2]	Ying HUANG, Jiayu YANG, Jiahao JIN, Bangrui WAN. Siamese mixed information fusion algorithm for RGBT tracking [J]. Journal of Computer Applications, 2024, 44(9): 2878-2885.
[3]	Xingyao YANG, Yu CHEN, Jiong YU, Zulian ZHANG, Jiaying CHEN, Dongxiao WANG. Recommendation model combining self-features and contrastive learning [J]. Journal of Computer Applications, 2024, 44(9): 2704-2710.
[4]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[5]	Hang YANG, Wanggen LI, Gensheng ZHANG, Zhige WANG, Xin KAI. Multi-layer information interactive fusion algorithm based on graph neural network for session-based recommendation [J]. Journal of Computer Applications, 2024, 44(9): 2719-2725.
[6]	Yu DU, Yan ZHU. Constructing pre-trained dynamic graph neural network to predict disappearance of academic cooperation behavior [J]. Journal of Computer Applications, 2024, 44(9): 2726-2731.
[7]	Na WANG, Lin JIANG, Yuancheng LI, Yun ZHU. Optimization of tensor virtual machine operator fusion based on graph rewriting and fusion exploration [J]. Journal of Computer Applications, 2024, 44(9): 2802-2809.
[8]	Yun LI, Fuyou WANG, Peiguang JING, Su WANG, Ao XIAO. Uncertainty-based frame associated short video event detection method [J]. Journal of Computer Applications, 2024, 44(9): 2903-2910.
[9]	Tingjie TANG, Jiajin HUANG, Jin QIN. Session-based recommendation with graph auxiliary learning [J]. Journal of Computer Applications, 2024, 44(9): 2711-2718.
[10]	Rui ZHANG, Pengyun ZHANG, Meirong GAO. Self-optimized dual-modal multi-channel non-deep vestibular schwannoma recognition model [J]. Journal of Computer Applications, 2024, 44(9): 2975-2982.
[11]	Jinjin LI, Guoming SANG, Yijia ZHANG. Multi-domain fake news detection model enhanced by APK-CNN and Transformer [J]. Journal of Computer Applications, 2024, 44(9): 2674-2682.
[12]	Yubo ZHAO, Liping ZHANG, Sheng YAN, Min HOU, Mao GAO. Relation extraction between discipline knowledge entities based on improved piecewise convolutional neural network and knowledge distillation [J]. Journal of Computer Applications, 2024, 44(8): 2421-2429.
[13]	Chunxue ZHANG, Liqing QIU, Cheng’ai SUN, Caixia JING. Purchase behavior prediction model based on two-stage dynamic interest recognition [J]. Journal of Computer Applications, 2024, 44(8): 2365-2371.
[14]	Tong CHEN, Fengyu YANG, Yu XIONG, Hong YAN, Fuxing QIU. Construction method of voiceprint library based on multi-scale frequency-channel attention fusion [J]. Journal of Computer Applications, 2024, 44(8): 2407-2413.
[15]	Rui SHI, Yong LI, Yanhan ZHU. Adversarial sample attack algorithm of modulation signal based on equalization of feature gradient [J]. Journal of Computer Applications, 2024, 44(8): 2521-2527.

Efficient homomorphic neural network supporting privacy-preserving training

支持隐私保护训练的高效同态神经网络

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 10

References 23

Related Articles 15

Recommended Articles

Metrics