Review of network protocol recognition techniques

doi:10.11772/j.issn.1001-9081.2019050949

Abstract

Abstract: Since the protocol classification of network traffic is a prerequisite for protocol analysis and network management, the network protocol recognition techniques were researched and reviewed. Firstly, the target of network protocol recognition was described, and the general process of protocol recognition was analyzed. The practical requirements for protocol recognition were discussed, and the criteria for evaluating protocol recognition methods were given. Then, the research status of network protocol techniques was summarized from two categories:packet-based protocol recognition methods and flow-based protocol recognition methods, and the variety of techniques used for protocol recognition were analyzed and compared. Finally, with the defects of current protocol recognition methods and the practical application requirements considered, the research trend of protocol recognition techniques was forecasted.

Key words: application-level protocol, network traffic, protocol recognition, feature engineering, network management

摘要： 网络流量的协议类型识别是进行协议分析和网络管理的前提，为此研究综述了网络协议识别技术。首先，描述了网络协议识别的目标，分析了协议识别的一般流程，探讨了协议识别的现实需求，给出了评估协议识别方法的标准；然后，从基于数据包的协议识别和基于数据流的协议识别两个类别分析了网络协议技术的研究现状，并对协议识别的各类技术进行了比较分析；最后，针对目前协议识别方法的缺陷和应用需求，对协议识别技术的研究趋势进行了展望。

关键词: 应用层协议, 网络流量, 协议识别, 特征工程, 网络管理

CLC Number:

TP393.02

FENG Wenbo, HONG Zheng, WU Lifa, FU Menglin. Review of network protocol recognition techniques[J]. Journal of Computer Applications, 2019, 39(12): 3604-3614.

冯文博, 洪征, 吴礼发, 付梦琳. 网络协议识别技术综述[J]. 计算机应用, 2019, 39(12): 3604-3614.

References

[1] SANDERS C. Practical Packet Analysis:Using Wireshark to Solve Real-World Network Problems[M]. San Francisco:No Starch Press, 2011:192-194.
[2] 陈亮,龚俭,徐选.应用层协议识别算法综述[J].计算机科学,2007,34(7):73-75.(CHEN L, GONG J, XU X. A survey of application-level protocol identification algorithm[J]. Computer Science, 2007, 34(7):73-75.)
[3] SourceForge. L7-filter:application layer packet classifier for Linux[EB/OL].[2019-04-14]. http://l7-filter.sourceforge.net/.
[4] DERI L, MARTINELLI M, BUJLOW T, et al. nDPI:open-source high-speed deep packet inspection[C]//Proceedings of the 2014 Wireless Communications and Mobile Computing Conference. Piscataway:IEEE, 2014:617-622.
[5] ALCOCK S, NELSON R. Libprotoident:traffic classification using lightweight packet inspection:technical report[R/OL].[2019-04-14]. http://www.wand.net.nz/publications/lpireport.
[6] MOORE A W, ZUEV D. Internet traffic classification using Bayesian analysis techniques[J]. ACM SIGMETRICS Performance Evaluation Review, 2005, 33(1):50-60.
[7] CAIDA. CAIDA data-overview of datasets, monitors and reports[EB/OL].[2019-04-14]. http://www.caida.org/data/overview/.
[8] Università degli Studi di Brescia. UNIBS:data sharing[EB/OL].[2019-04-14]. http://netweb.ing.unibs.it/~ntw/tools/traces.
[9] The MAWI Working Group. MAWI working group traffic archive[EB/OL].[2019-04-14]. http://mawi.wide.ad.jp/mawi/.
[10] WAND network research group. WITS:Waikato Internet traffic storage[EB/OL].[2019-04-14]. https://wand.net.nz/wits/.
[11] LIPPMANN R, HAINES J W, FRIED D J, et al. The 1999 DARPA off-line intrusion detection evaluation[J]. Computer Networks, 2000, 34(4):579-595.
[12] TOUCH J, MANKIN A, KOHLER E, et al. Service name and transport protocol port number registry[EB/OL].[2019-04-14]. https://www.iana.org/assignments/service-names-port-numbers/service-names-port-numbers.xhtml.
[13] MADHUKAR A, WILLIAMSON C. A longitudinal study of P2P traffic classification[C]//Proceedings of the 14th International Symposium on Modeling, Analysis, and Simulation. Piscataway:IEEE, 2006:179-188.
[14] YOON S H, PARK J W, PARK J S, et al. Internet application traffic classification using fixed IP-port[C]//Proceedings of the 12th Asia-pacific Network Operations and Management Conference on Management Symposium, LNCS 5787. Berlin:Springer, 2009:21-30.
[15] KARAGIANNIS T, BROIDO A, FALOUTSOS M, et al. Transport layer identification of P2P traffic[C]//Proceedings of the 4th ACM SIGCOMM Internet Measurement Conference. New York:ACM, 2004:121-134.
[16] 汪立东,钱丽萍,王大伟,等.网络流量分类方法与实践[M].北京:人民邮电出版社,2013:122-126.(WANG L D, QIAN L P, WANG D W, et al. Network Traffic Classification[M]. Beijing:Posts and Telecom Press, 2013:122-126.)
[17] SEN S, SPATSCHECK O, WANG D. Accurate, scalable in-network identification of P2P traffic using application signatures[C]//Proceedings of the 2004 13th International Conference on World Wide Web. New York:ACM, 2004:512-521.
[18] YUN X, WANG Y, ZHANG Y, et al. A semantics-aware approach to the automated network protocol identification[J]. IEEE/ACM Transactions on Networking, 2016, 24(1):583-595.
[19] MOORE A W, PAPAGIANNAKI K. Toward the accurate identification of network applications[C]//Proceedings of the 6th International Workshop on Passive and Active Network Measurement, LNCS 3431. Berlin:Springer, 2005:41-54.
[20] BUJLOW T, CARELA-ESPANOL V, BARLET-ROS P. Independent comparison of popular DPI tools for traffic classification[J]. Computer Networks, 2015, 76:75-89.
[21] CHEN H, HU Z, YE Z, et al. A new model for P2P traffic identification based on DPI and DFI[C]//Proceedings of the 2009 International Conference on Information Engineering and Computer Science. Piscataway:IEEE, 2009:1-3.
[22] WANG C, ZHOU X, YOU F, et al. Design of P2P traffic identification based on DPI and DFI[C]//Proceedings of the 2009 International Symposium on Computer Network and Multimedia Technology. Piscataway:IEEE, 2009:1-4.
[23] 叶文晨,汪敏,陈云寰,等.一种联合DPI和DFI的网络流量检测方法[J].计算机工程,2011,37(10):102-104,107.(YE W C, WANG M, CHEN Y H, et al. Network flow inspection method of joint DPI and DFI[J]. Computer Engineering, 2011, 37(10):102-104, 107.)
[24] 林冠洲.网络流量识别关键技术研究[D].北京:北京邮电大学,2011:10.(LIN G Z. Research on the key technologies of network traffic classification[D]. Beijing:Beijing University of Posts and Telecommunications, 2011:10.)
[25] 付文亮,嵩天,周舟.RocketTC:一种基于FPGA的高性能网络流量分类架构[J].计算机学报,2014,37(2):414-422.(FU W L, SONG T, ZHOU Z. RocketTC:a high throughput traffic classification architecture on FPGA[J]. Chinese Journal of Computers, 2014, 37(2):414-422.)
[26] KUMAR S, DHARMAPURIKAR S, YU F, et al. Algorithms to accelerate multiple regular expressions matching for deep packet inspection[J]. ACM SIGCOMM Computer Communication Review, 2006, 36(4):339-350.
[27] TONG D, QU Y R, PRASANNA V K. Accelerating decision tree based traffic classification on FPGA and multicore platforms[J]. IEEE Transactions on Parallel and Distributed Systems, 2017, 28(11):3046-3059.
[28] FINAMORE A, MELLIA M, MEO M, et al. KISS:stochastic packet inspection classifier for UDP traffic[J]. IEEE/ACM Trans on Networking, 2010, 18(5):1505-1515.
[29] CROTTI M, DUSI M, GRINGOLI F, et al. Traffic classification through simple statistical fingerprinting[J]. ACM SIGCOMM Computer Communication Review, 2007, 37(1):5-16.
[30] KARAGIANNIS T, PAPAGIANNAKI K, FALOUTSOS M. BLINC:multilevel traffic classification in the dark[C]//Proceedings of the 2005 Conference on Applications, Technologies, Architectures, and Protocols for Computer Communications. New York:ACM, 2005:229-240.
[31] ILIOFOTOU M, PAPPU P, FALOUTSOS M, et al. Network monitoring using Traffic Dispersion Graphs (TDGs)[C]//Proceedings of the 7th ACM SIGCOMM Conference on Internet Measurement. New York:ACM, 2007:315-320.
[32] 李君,张顺颐,王浩云,等.基于贝叶斯网络的Peer-to-Peer识别方法[J].应用科学学报,2009,27(2):124-130.(LI J, ZHANG S Y, WANG H Y, et al. Peer-to-peer traffic identification using Bayesian networks[J]. Journal of Applied Sciences, 2009, 27(2):124-130)
[33] 张泽鑫,李俊,常向青.基于特征加权的朴素贝叶斯流量分类方法研究[J].高技术通讯,2016,26(2):119-128.(ZHANG Z X, LI J, CHANG X Q. Internet traffic classification using the attribute weighted naive Bayes algorithm[J]. High Technology Letters, 2016, 26(2):119-128.)
[34] 孙德山.支持向量机分类与回归方法研究[D].长沙:中南大学,2004:10.(SUN D S. The researches on support vector machine classification and regression methods[D]. Changsha:Central South University, 2004:10.)
[35] LI Z, YUAN R, GUAN X. Accurate classification of the Internet traffic based on the SVM method[C]//Proceedings of the 2007 IEEE International Conference on Communications. Piscataway:IEEE, 2007:1373-1378.
[36] YANG A, JIANG S, DENG H. A P2P network traffic classification method using SVM[C]//Proceedings of the 9th International Conference for Young Computer Scientists. Piscataway:IEEE, 2008:398-403.
[37] GROLEAT T, ARZEL M, VATON S. Hardware acceleration of SVM-based traffic classification on FPGA[C]//Proceedings of the 8th International Wireless Communications and Mobile Computing Conference. Piscataway:IEEE, 2012:443-449.
[38] 王一鹏,云晓春,张永铮,等.基于主动学习和SVM方法的网络协议识别技术[J].通信学报,2013,34(10):135-142.(WANG Y P, YUN X C, ZHANG Y Z, et al. Network protocol identification based on active learning and SVM algorithm[J]. Journal on Communications, 2013, 34(10):135-142.)
[39] 曹杰.基于SVM的网络流量特征降维与分类方法研究[D].长春:吉林大学,2017:10.(CAO J. Research of feature reduction and traffic classification method based on SVM[D]. Changchun:Jilin University, 2017:10.)
[40] LOBATO A G P, LOPEZ M A, SANZ I J, et al. An adaptive real-time architecture for zero-day threat detection[C]//Proceedings of the 2018 International Conference on Communications. Piscataway:IEEE, 2018:1-6.
[41] 谭骏,陈兴蜀,杜敏,等.基于自适应BP神经网络的网络流量识别算法[J].电子科技大学学报,2012,41(4):580-585.(TAN J, CHEN X S, DU M, et al. Internet traffic identification algorithm based on adaptive BP neural network[J]. Journal of University of Electronic Science and Technology of China, 2012, 41(4):580-585)
[42] WANG W, ZHU M, ZENG X, et al. Malware traffic classification using convolutional neural network for representation learning[C]//Proceedings of the 2017 International Conference on Information Networking. Piscataway, NJ:IEEE, 2017:712-717.
[43] 王勇,周慧怡,俸皓,等.基于深度卷积神经网络的网络流量分类方法[J].通信学报,2018,39(1):14-23.(WANG Y, ZHOU H Y, FENG H, et al. Network traffic classification method basing on CNN[J]. Journal on Communications, 2018, 39(1):14-23.)
[44] JAIN A V. Network traffic identification with convolutional neural networks[C]//Proceedings of the IEEE 16th International Conference on Dependable, Autonomic and Secure Computing, 16th International Conference on Pervasive Intelligence and Computing, 4th International Conference on Big Data Intelligence and Computing and 2018 International Conference on Cyber Science and Technology. Piscataway:IEEE, 2018:1001-1007.
[45] 陈雪娇,王攀,俞家辉. 基于卷积神经网络的加密流量识别方法[J].南京邮电大学学报(自然科学版),2018,38(6):40-45.(CHEN X J, WANG P, YU J H. CNN based encrypted traffic identification method[J]. Journal of Nanjing University of Posts and Telecommunications (Natural Science Edition), 2018, 38(6):40-45)
[46] 叶松.基于现代网络的深度学习应用协议识别技术研究与实现[J].软件导刊,2018,17(10):194-199.(YE S. Research and implementation of deep learning application protocol recognition technology based on modern network[J]. Software Guide, 2018, 17(10):194-199.)
[47] REN J, WANG Z. A novel deep learning method for application identification in wireless network[J]. China Communications, 2018, 15(10):73-83.
[48] WILLIAMS N, ZANDER S, ARMITAGE G. A preliminary performance comparison of five machine learning algorithms for practical IP traffic flow classification[J]. ACM SIGCOMM Computer Communication Review, 2006, 36(5):5-16.
[49] 徐鹏,林森.基于C4.5决策树的流量分类方法[J].软件学报,2009,20(10):2692-2704.(XU P, LIN S. Internet traffic classification using C4.5 decision tree[J]. Journal of Software, 2009, 20(10):2692-2704.)
[50] 周剑峰,阳爱民,刘吉财.基于改进的C4.5算法的网络流量分类方法[J].计算机工程与应用,2012,48(5):71-74.(ZHOU J F, YANG A M, LIU J C. Traffic classification approach based on improved C4.5 algorithm[J]. Computer Engineering and Applications, 2012, 48(5):71-74)
[51] 王朝正.基于Hadoop的C4.5决策树及其在网络流量中的应用[D].重庆:重庆邮电大学,2016:10.(WANG C Z. C4.5 decision tree based on Hadoop and its application in network traffic[D]. Chongqing:Chongqing University of Posts and Telecommunications, 2016:10.)
[52] 程珊,钮焱,李军.基于网络资源的KNN网络流量分类模型的研究[J].湖北工业大学学报,2016,31(4):75-79,82.(CHENG S, NIU Y, LI J. A study on network traffic classification model of KNN based on network resources[J]. Journal of Hubei University of Technology, 2016, 31(4):75-79, 82.)
[53] SU M Y. Using clustering to improve the KNN-based classifiers for online anomaly network traffic identification[J]. Journal of Network and Computer Applications, 2011, 34(2):722-730.
[54] ZHANG J, XIANG Y, WANG Y, et al. Network traffic classification using correlation information[J]. IEEE Transactions on Parallel and Distributed Systems, 2013, 24(1):104-117.
[55] WU D, CHEN X, CHEN C, et al. On addressing the imbalance problem:a correlated KNN approach for network traffic classification[C]//Proceedings of the 2015 International Conference on Network and System Security, LNCS 8792. Cham:Springer, 2015:138-151.
[56] MCGREGOR A, HALL M, LORIER P, et al. Flow clustering using machine learning techniques[C]//Proceedings of the 2004 International Workshop on Passive and Active Network Measurement, LNCS 3015. Berlin:Springer, 2004:205-214.
[57] LIU S, HU J, HAO S, et al. Improved EM method for Internet traffic classification[C]//Proceedings of the 8th International Conference on Knowledge and Smart Technology. Piscataway:IEEE, 2016:13-17.
[58] ZANDER S, NGUYEN T, ARMITAGE G. Automated traffic classification and application identification using machine learning[C]//Proceedings of the 2005 IEEE Conference on Local Computer Networks 30th Anniversary. Piscataway:IEEE, 2005:250-257.
[59] LIU Y, LI W, LI Y. Network traffic classification using k-means clustering[C]//Proceedings of the 2nd International Multi-symposiums on Computer and Computational Sciences. Piscataway:IEEE, 2007:360-365.
[60] 彭大芹,项磊,李司坤,等.多协议下智能家居协议的分类方法[J].重庆邮电大学学报(自然科学版),2018,30(3):321-328.(PENG D Q, XIANG L, LI S K, et al. Classification of intelligent home protocol under multi-protocols[J]. Journal of Chongqing University of Posts and Telecommunications (Natural Science Edition), 2018, 30(3):321-328.)
[61] 何震凯,阳爱民,刘永定,等.一种使用DBSCAN聚类的网络流量分类方法[J].计算机应用研究,2009,26(9):3461-3464.(HE Z K, YANG A M, LIU Y D, et al. Method of network traffic classification using DBSCAN clustering[J]. Application Research of Computers, 2009, 26(9):3461-3464.)
[62] 张凤荔,周洪川,张俊娇,等.基于改进凝聚层次聚类的协议分类算法[J].计算机工程与科学,2017,39(4):796-803.(ZHANG F L, ZHOU H C, ZHANG J J, et al. A protocol classification algorithm based on improved AGENS[J]. Computer Engineering and Science, 2017, 39(4):796-803.)
[63] 周文刚,陈雷霆,董仕.基于谱聚类的网络流量分类识别算法[J].电子测量与仪器学报,2013,27(12):1114-1119.(ZHOU W G, CHEN L T, DONG S. Network traffic classification algorithm based on spectral clustering[J]. Journal of Electronic Measurement and Instrument, 2013, 27(12):1114-1119.)
[64] BERNAILLE L, TEIXEIRA R, SALAMATIAN K. Early application identification[C]//Proceedings of the 2006 ACM Conference on Emerging Network Experiment and Technology. New York:ACM, 2006:Article No. 6.
[65] 赵英,韩春昊.马尔科夫模型在网络流量分类中的应用与研究[J].计算机工程,2018,44(5):291-295.(ZHAO Y, HAN C H. Application and research of Markov model in network traffic classification[J]. Computer Engineering, 2018, 44(5):291-295.)
[66] ERMAN J, MAHANTI A, ARLITT M, et al. Semi-supervised network traffic classification[J]. ACM SIGMETRICS Performance Evaluation Review, 2007, 35(1):369-370.
[67] ZHANG J, CHEN C, XIANG Y, et al. Semi-supervised and compound classification of network traffic[C]//Proceedings of the 32nd International Conference on Distributed Computing Systems Workshops. Piscataway:IEEE, 2012:617-621.
[68] 丁伟,徐杰,卓文辉.基于层次聚类的网络流识别算法研究[J].通信学报,2014,35(Z1):41-45.(DING W, XU J, ZHUO W H. Net traffic identifier based on hierarchical clustering[J]. Journal on Communications, 2014, 35(Z1):41-45.)
[69] WANG Y, XUE H, LIU Y, et al. Statistical network protocol identification with unknown pattern extraction[J]. Annals of Telecommunications, 2019, 74(7/8):473-482.
[70] CARELA-ESPANOL V, BUJLOW T, BARLET-ROS P. Is our ground-truth for traffic classification reliable?[C]//Proceedings of the 2014 International Conference on Passive and Active Network Measurement, LNCS 8362. Cham:Springer, 2014:98-108.