Inference delay optimization of branchy neural network model based on edge computing

doi:10.11772/j.issn.1001-9081.2019081406

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (2): 342-346.DOI: 10.11772/j.issn.1001-9081.2019081406

• DPCS 2019 • Previous Articles Next Articles

Inference delay optimization of branchy neural network model based on edge computing

Qi FAN¹^,², Zhuo LI¹^,²(), Xin CHEN²

^1.Beijing Key Laboratory of Internet Culture and Digital Dissemination Research （Beijing Information Science & Technology University），Beijing 100101，China
^2.School of Computer Science，Beijing Information Science & Technology University，Beijing 100101，China

Received:2019-07-31 Revised:2019-09-05 Accepted:2019-09-23 Online:2020-02-26 Published:2020-02-10
Contact: Zhuo LI
About author:FAN Qi， born in 1995， M. S. candidate. His research interests include edge computing.
CHEN Xin， born in 1965， Ph. D.， professor. His research interests include network performance evaluation， network security.
Supported by:
the National Natural Science Foundation of China(61502040);the Beijing Municipal Program for Top Talent Cultivation(CIT&TCD201804055);the Qinxin Talent Program of Beijing Information Science and Technology University, the Open Program of Beijing Key Laboratory of Internet Culture and Digital Dissemination Research(ICDDXN001)

基于边缘计算的分支神经网络模型推断延迟优化

樊琦¹^,², 李卓¹^,²(), 陈昕²

^1.网络文化与数字传播北京市重点实验室（北京信息科技大学），北京 100101
^2.北京信息科技大学计算机学院，北京 100101

通讯作者: 李卓
作者简介:樊琦（1995—），男，山西运城人，硕士研究生，主要研究方向：边缘计算
陈昕（1965—），男，江西南昌人，教授，博士，CCF会员，主要研究方向：网络性能评价、网络安全。
基金资助:
国家自然科学基金资助项目(61502040);北京市属高校高水平教师队伍建设支持计划青年拔尖人才培育计划资助项目(CIT&TCD201804055);北京信息科技大学“勤信人才”培养计划资助项目;网络文化与数字传播北京市重点实验室开放课题资助项目(ICDDXN001)

Abstract

Abstract:

Aiming at the long delay of inference tasks in Deep Neural Network （DNN） on cloud servers， a branchy neural network deployment model based on edge computing was proposed. The distributed deployment problem of DNNs in edge computing scenarios was analyzed， and was proved to be NP-hard. A Deployment algorithm based on Branch and Bound （DBB） was designed to select appropriate edge computing nodes to reduce inference delay. And a Selection Node Exit （SNE） algorithm was designed and implemented to select the appropriate edge computing nodes for different tasks to exit the inference task. The simulation results show that， compared with the approach of deploying neural network model on the cloud， the branchy neural network model based on edge computing reduces the inference delay by 36% on average.

Key words: edge computing, branchy neural network, Deep Neural Network (DNN), inference delay, deployment problem

摘要：

针对云服务器上深度神经网络（DNN）模型推断任务延迟过高的问题，提出基于边缘计算的分支神经网络部署模型。分析了边缘计算场景中深度神经网络的分布式部署问题，证明该问题是NP-难的。设计了一种基于分支定界思想的部署算法（DBB），选择合适的边缘计算节点部署模型以减少推断任务的延迟。设计并实现了选择节点退出（SNE）算法，为不同任务选择合适的边缘计算节点来退出推断任务。仿真实验结果表明，与在云端部署神经网络模型的方法相比，基于边缘计算的分支神经网络模型的推断延迟平均降低了36%。

关键词: 边缘计算, 分支神经网络, 深度神经网络, 推断延迟, 部署问题

CLC Number:

TP393.01

Qi FAN, Zhuo LI, Xin CHEN. Inference delay optimization of branchy neural network model based on edge computing[J]. Journal of Computer Applications, 2020, 40(2): 342-346.

樊琦, 李卓, 陈昕. 基于边缘计算的分支神经网络模型推断延迟优化[J]. 《计算机应用》唯一官方网站, 2020, 40(2): 342-346.

Figures/Tables 8

Fig. 1 Deep learning model comparison of cloud computing and edge computing

Fig. 2 Running scenario of inference task

Fig. 3 Main structure of branchy neural network model

Tab. 1 Branchy model running delay on edge devices

分支	JetsonTK1	JetsonTX1	JetsonTX2	JetsonNaNo
1	11.04	8.58	6.34	10.52
2	15.58	8.54	6.08	12.68
3	11.80	7.92	6.14	11.34
4	11.72	7.55	6.08	8.18
5	6.50	6.28	5.17	7.81
6	4.76	6.13	5.20	4.53

Fig. 4 Inference delay comparison of the proposed method and cloud server method

Fig. 5 Simulation state of the network

Fig. 6 Task inference delay comparison of different data sources in edge computing scenario

Tab. 2 Comparison of inference accuracy in edge and cloud scenarios with limited network

场景	延迟要求
场景	50 ms	150 ms	250 ms	350 ms
云服务器场景	$- 1$	-1	-1	0.82
边缘场景	$- 1$	0.66	0.74	0.79

Tab. 2 Comparison of inference accuracy in edge and cloud scenarios with limited network

场景	延迟要求
场景	50 ms	150 ms	250 ms	350 ms
云服务器场景	$- 1$	-1	-1	0.82
边缘场景	$- 1$	0.66	0.74	0.79

References 20

1	CHEN X， KUNDU K， ZHANG Z， et al. Monocular 3D object detection for autonomous driving［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2147-2156. 10.1109/cvpr.2016.236
2	ZHENG Y， LIU Y， HANSEN J H L. Navigation-orientated natural spoken language understanding for intelligent vehicle dialogue［C］// Proceedings of the 2017 IEEE Intelligent Vehicles Symposium. Piscataway： IEEE， 2017： 559-564. 10.1109/ivs.2017.7995777
3	KANG L， YE P， LI Y， et al. Convolutional neural networks for no-reference image quality assessment［C］// Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2014：1733-1740. 10.1109/cvpr.2014.224
4	TEERAPITTAYANON S， MCDANEL B， KUNG H T. Distributed deep neural networks over the cloud， the edge and end devices［C］// Proceedings of the IEEE 37th International Conference on Distributed Computing Systems. Piscataway： IEEE， 2017： 328-339. 10.1109/icdcs.2017.226
5	ARMBRUST M， FOX A， GRIFFITH R， et al. Above the clouds： a Berkeley view of cloud computing， UCB/EECS-2009-28［R］. Berkeley： Electrical Engineering and Computer Sciences University of California， 2009.
6	HOWARD A G， ZHU M， CHEN B， et al. MobileNets： efficient convolutional neural networks for mobile vision applications［EB/OL］. ［2019-04-22］. .
7	SYAMKUMAR M， BARFORD P， DURAIRAJAN R. Deployment characteristics of "the edge" in mobile edge computing［C］// Proceedings of the 2018 Workshop on Mobile Edge Communications. New York： ACM， 2018：43-49. 10.1145/3229556.3229557
8	LI M， ANDERSEN D G， PARK J W， et al. Scaling distributed machine learning with the parameter server［C］// Proceedings of the 11th USENIX Conference on Operating Systems Design and Implementation. Berkeley： USENIX Association， 2014：583-598. 10.1145/2640087.2644155
9	IANDOLA F N， MOSKEWICZ M W， ASHRAF K， et al. FireCaffe： near-linear acceleration of deep neural network training on compute clusters［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：2592-2600. 10.1109/cvpr.2016.284
10	TEERAPITTAYANON S， MCDANEL B， KUNG H T. BranchyNet： Fast inference via early exiting from deep neural networks［C］// Proceedings of the 23rd International Conference on Pattern Recognition. Piscataway： IEEE， 2016：2464-2469. 10.1109/icpr.2016.7900006
11	MAO J， CHEN X， NIXON K W， et al. MoDNN： local distributed mobile computing system for deep neural network［C］// Proceedings of the 2017 Design， Automation and Test in Europe Conference and Exhibition. Piscataway： IEEE， 2017： 1396-1401. 10.23919/date.2017.7927211
12	LI E， ZHOU Z， CHEN X. Edge intelligence： on-demand deep learning model co-inference with device-edge synergy［C］// Proceedings of the 2018 Workshop on Mobile Edge Communications. New York： ACM， 2018： 31-36. 10.1145/3229556.3229562
13	KANG Y， HAUSWALD J， GAO C， et al. Neurosurgeon： collaborative intelligence between the cloud and mobile edge［J］. ACM SIGPLAN Notices， 2017， 52（4）：615-629. 10.1145/3093336.3037698
14	LI H， OTA K， DONG M. Learning IoT in edge： deep learning for the Internet of things with edge computing［J］. IEEE Network， 2018， 32（1）：96-101. 10.1109/mnet.2018.1700202
15	HAN S， MAO H， DALLY W J. Deep compression： compressing deep neural networks with pruning， trained quantization and Huffman coding［C］// Proceedings of the 2016 International Conference on Learning Representations. ［S.l.］： ICLR， 2016： 351-360.
16	蔡瑞初，钟椿荣，余洋，等.面向“边缘”应用的卷积神经网络量化与压缩方法［J］.计算机应用，2018，38（9）：2449-2454.
	CAI R C， ZHONG C R， YU Y， et al. CNN quantization and compression strategy for edge computing applications［J］. Journal of Computer Applications， 2018， 38（9）：2449-2454.
17	赵梓铭，刘芳，蔡志平，等.边缘计算：平台、应用与挑战［J］.计算机研究与发展，2018，55（2）：327-337. 10.7544/issn1000-1239.2018.20170228
	ZHAO Z M， LIU F， CAI Z P， et al. Edge computing： platforms， applications and challenges ［J］. Journal of Computer Research and Development， 2018， 55（2）：327-337. 10.7544/issn1000-1239.2018.20170228
18	SAHNI S， GONZALES T. P-complete approximation problem［J］. Journal of the ACM， 1976， 23（3）：555-565. 10.1145/321958.321975
19	符永铨，李东升.边缘计算环境下应用驱动的网络延迟测量与优化技术［J］.计算机研究与发展，2018，55（3）：512-523. 10.1109/access.2018.2834623
	FU Y Q， LI D S. Application driven network latency measurement analysis and optimization techniques edge computing environment： a survey［J］. Journal of Computer Research and Development， 2018， 55（3）：512-523. 10.1109/access.2018.2834623
20	QI H， SPARKS E R， TALWALKAR A. Paleo： a performance model for deep neural networks［C/OL］// Proceedings of the 5th International Conference on Learning Representations. ［S.l.］： ICRL， 2017 ［2019-04-06］. ？id=SyVVJ85lg.

[1]	GUO Mian, ZHANG Jinyou. Computation offloading policy for machine learning in mobile edge computing environments [J]. Journal of Computer Applications, 2021, 41(9): 2639-2645.
[2]	LI Zhuo, SONG Zihui, SHEN Xin, CHEN Xin. Local differential privacy protection mechanism for mobile crowd sensing with edge computing [J]. Journal of Computer Applications, 2021, 41(9): 2678-2686.
[3]	WANG Shuyan, HOU Zeyu, SUN Jiaze. Difference detection method of adversarial samples oriented to deep learning [J]. Journal of Computer Applications, 2021, 41(7): 1849-1856.
[4]	DONG Wentao, LI Zhuo, CHEN Xin. Online short video content distribution strategy based on federated learning [J]. Journal of Computer Applications, 2021, 41(6): 1551-1556.
[5]	ZHANG Mingming, LU Qingning, LI Wenzhong, SONG Hu. Deep neural network compression algorithm based on combined dynamic pruning [J]. Journal of Computer Applications, 2021, 41(6): 1589-1596.
[6]	WANG Yijie, FAN Jiafei, WANG Chenyu. Two-stage task offloading strategy based on game theory in cloud-edge environment [J]. Journal of Computer Applications, 2021, 41(5): 1392-1398.
[7]	MAO Yingchi, XU Xuesong, LIU Pengfei. Multi-user task offloading strategy based on stable allocation [J]. Journal of Computer Applications, 2021, 41(3): 786-793.
[8]	YANG Li, WANG Shihui, ZHU Bo. Point-of-interest recommendation algorithm combing dynamic and static preferences [J]. Journal of Computer Applications, 2021, 41(2): 398-406.
[9]	Ying LEI, Wanbo ZHENG, Wei WEI, Yunni XIA, Xiaobo LI, Chengwu LIU, Hong XIE. Task offloading method based on probabilistic performance awareness and evolutionary game strategy in “cloud + edge” hybrid environment [J]. Journal of Computer Applications, 2021, 41(11): 3302-3308.
[10]	WANG Daiwei, XU Gaochao, LI Long. Task allocation strategy in unmanned aerial vehicle-assisted mobile edge computing [J]. Journal of Computer Applications, 2021, 41(10): 2928-2936.
[11]	WU Guangfu, WANG Yingjun. Secure storage and sharing scheme of internet of vehicles data based on hybrid architecture of blockchain and cloud-edge computing [J]. Journal of Computer Applications, 2021, 41(10): 2885-2892.
[12]	CHEN Yanru, ZHANG Tujingwa, DU Qian, RAN Maoliang, WANG Hongjun. Prediction of indoor thermal comfort level of high-speed railway station based on deep forest [J]. Journal of Computer Applications, 2021, 41(1): 258-264.
[13]	WU Yuxin, CAI Ting, ZHANG Dabin. Computing power trading and pricing in mobile edge computing based on Stackelberg game [J]. Journal of Computer Applications, 2020, 40(9): 2683-2690.
[14]	LUO Bin, YU Bo. Computation offloading strategy based on particle swarm optimization in mobile edge computing [J]. Journal of Computer Applications, 2020, 40(8): 2293-2298.
[15]	LIU Xing, YANG Zhen, WANG Xinjun, ZHU Heng. Content distribution acceleration strategy in mobile edge computing [J]. Journal of Computer Applications, 2020, 40(5): 1389-1391.

Inference delay optimization of branchy neural network model based on edge computing

基于边缘计算的分支神经网络模型推断延迟优化

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 20

Related Articles 15

Recommended Articles

Metrics