Multi-source data parallel preprocessing method based on similar connection

doi:10.11772/j.issn.1001-9081.2018071869

Journal of Computer Applications ›› 2019, Vol. 39 ›› Issue (1): 57-60.DOI: 10.11772/j.issn.1001-9081.2018071869

Previous Articles Next Articles

Multi-source data parallel preprocessing method based on similar connection

GUO Fangfang, CHAO Luomeng, ZHU Jianwen

School of Computer Science and Technology, Harbin Engineering University, Harbin Heilongjiang 150001, China

Received:2018-07-19 Revised:2018-09-25 Online:2019-01-10 Published:2019-01-21
Supported by:
This work is partially supported by the National Science and Technology Major Project (2016ZX03001023-005), the National Industry-University-Research Cooperation Project (2016ZTE01-03-06), the Fundamental Research Funds for the Central Universities (HEUCF100601).

基于相似连接的多源数据并行预处理方法

郭方方, 潮洛蒙, 朱建文

哈尔滨工程大学计算机科学与技术学院, 哈尔滨 150001

通讯作者: 潮洛蒙
作者简介:郭方方(1974-),男,黑龙江哈尔滨人,副教授,博士,主要研究方向:云计算、网络安全、P2P网络;潮洛蒙(1994-),男(蒙古族),内蒙古通辽人,硕士研究生,主要研究方向:网络安全、机器学习、数据分析;朱建文(1989-),男,四川成都人,硕士研究生,主要研究方向:网络安全、数据分析、安全态势感知。
基金资助:
国家科技重大专项（2016ZX03001023-005）；国家级产学研合作项目（2016ZTE01-03-06）；中央高校基本科研业务费专项（HEUCF100601）。

Abstract

Abstract: With the development of large-scale network environments and big data-related technologies, traditional data fusion analysis technology faces new challenges. Focusing on poor flexibility and low processing efficiency in current multi-source data fusion analysis process, a multi-source data parallel preprocessing method based on similar connection was proposed, in which the idea of dividing and conquering and paralleling was adopted. Firstly, the preprocessing method was improved to increase the flexibility by unifying similar semantics in multi-source data and retaining personality semantics. Secondly, an improved parallel MapReduce framework was proposed to improve the efficiency of similar connections. The experimental results show that the proposed method reduces total data volume by 32% while ensuring data integrity. Compared with traditional MapReduce framework, the improved framework decreases 43.91% of time consumed; therefore, the proposed method can effectively improve the efficiency of multi-source data fusion analysis.

Key words: network security, multi-source data, data preprocessing, similar connection, MapReduce

摘要： 大规模网络环境和大数据相关技术的发展对传统数据融合分析技术提出了新的挑战。针对目前多源数据融合分析过程灵活性差、处理效率低的问题，提出了一种基于相似连接的多源数据并行预处理方法，该方法采用了分治和并行的思想。首先，通过对多源数据中的相似语义进行统一、对个性语义进行保留的预处理方法提高了灵活性；其次，提出了一种改进的并行MapReduce框架，提高了相似连接的效率。实验结果表明，所提方法在保证数据完整性的基础上，使总的数据量减小了32%。与传统的MapReduce框架相比，改进后的框架在耗费时间方面减小了43.91%，因此该方法可以有效提高多源数据融合分析的效率。

关键词: 网络安全, 多源数据, 数据预处理, 相似连接, MapReduce

CLC Number:

TP274

GUO Fangfang, CHAO Luomeng, ZHU Jianwen. Multi-source data parallel preprocessing method based on similar connection[J]. Journal of Computer Applications, 2019, 39(1): 57-60.

郭方方, 潮洛蒙, 朱建文. 基于相似连接的多源数据并行预处理方法[J]. 计算机应用, 2019, 39(1): 57-60.

References

[1] MAJIDI M, OSKUOEE M. Improving pattern recognition accuracy of partial discharges by new data preprocessing methods[J]. Electric Power Systems Research, 2015, 119:100-110.
[2] LI G, DENG D, WANG J, et al. Pass-join:a partition-based method for similarity joins[J]. Proceedings of the VLDB Endowment, 2011, 5(3):253-264.
[3] WANG J, LI G, FE J. Fast-join:an efficient method for fuzzy token matching based string similarity join[C]//Proceedings of the 2011 International Conference on Data Engineering. Piscataway, NJ:IEEE, 2011:458-469.
[4] 陈一帆,赵翔,何培俊,等.BMGSJoin:一种基于MapReduce的图相似度连接算法[J].模式识别与人工智能,2015,28(5):472-480.(CHEN Y F, ZHAO X, HE P J, et al. BMGSJoin:a MapReduce based graph similarity join algorithm[J]. Pattern Recognition & Artificial Intelligence, 2015, 28(5):472-480.)
[5] 陈子军,张娟娜,刘文远.MapReduce框架下基于范围的空间文本相似连接[J].小型微型计算机系统,2015,36(10):2245-2251.(CHEN Z J, ZHANG J N, LIU W Y. Range-based spatial text similarity connection under MapReduce framework[J]. Journal of Chinese Computer Systems, 2015, 36(10):2245-2251.)
[6] BU Y, HOWE B, BALAZINSKA M, et al. HaLoop:efficient iterative data processing on large clusters[J]. Proceedings of the VLDB Endowment, 2010, 3(1/2):285-296.
[7] ZHANG Y, GAO Q, GAO L, et al. iMapReduce:a distributed computing framework for iterative computation[J]. Journal of Grid Computing, 2012, 10(1):47-68.
[8] 荣垂田,徐天任,杜小勇.基于划分的集合相似连接[J].计算机研究与发展,2012,49(10):2066-2076.(RONG C T, XU T R, DU X Y. Partition-based set similarity join[J]. Journal of Computer Research and Development, 2012, 49(10):2066-2076.)
[9] STUART J A, OWENS J D. Multi-GPU MapReduce on GPU clusters[C]//Proceedings of the 2011 International Conference on Parallel & Distributed Processing Symposium. Piscataway, NJ:IEEE, 2011:1068-1079.
[10] LIN F, SONG C, XU X, et al. Sensing from the bottom:smart insole enabled patient handling activity recognition through manifold learning[C]//Proceedings of the 2016 International Conference on Connected Health:Applications, Systems and Engineering Technologies. Piscataway, NJ:IEEE, 2016:254-263.
[11] LU J, WANG G, DENG W, et al. Multi-manifold deep metric learning for image set classification[C]//Proceedings of the 2015 International Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:1137-1145.
[12] 刘雪莉,王宏志,李建中,等.基于实体的相似性连接算法[J].软件学报,2015,26(6):1421-1437.(LIU X L, WANG H Z, LI J Z, et al. Entity-based similarity join algorithm[J]. Journal of Software, 2015, 26(6):1421-1437.)

[1]	WANG Yue, JIANG Yiming, LAN Julong. Intrusion detection based on improved triplet network and K-nearest neighbor algorithm [J]. Journal of Computer Applications, 2021, 41(7): 1996-2002.
[2]	ZHANG Quanlong, WANG Huaibin. Intrusion detection model based on combination of dilated convolution and gated recurrent unit [J]. Journal of Computer Applications, 2021, 41(5): 1372-1377.
[3]	TANG Yanqiang, LI Chenghai, SONG Yafei. Network security situation prediction based on improved particle swarm optimization and extreme learning machine [J]. Journal of Computer Applications, 2021, 41(3): 768-773.
[4]	HANG Mengxin, CHEN Wei, ZHANG Renjie. Abnormal flow detection based on improved one-dimensional convolutional neural network [J]. Journal of Computer Applications, 2021, 41(2): 433-440.
[5]	GU Tong, XU Guoliang, LI Wanlin, LI Jiahao, WANG Zhiyuan, LUO Jiangtao. Intelligent house price evaluation model based on ensemble LightGBM and Bayesian optimization strategy [J]. Journal of Computer Applications, 2020, 40(9): 2762-2767.
[6]	CHENG Xiaohui, NIU Tong, WANG Yanjun. Wireless sensor network intrusion detection system based on sequence model [J]. Journal of Computer Applications, 2020, 40(6): 1680-1684.
[7]	PAN Chunxia, YANG Qiuhui, TAN Wukun, DENG Huixin, WU Jia. Data preprocessing method in software defect prediction [J]. Journal of Computer Applications, 2020, 40(11): 3273-3279.
[8]	CHI Yaping, MO Chongwei, YANG Yintan, CHEN Chunxia. Design and implementation of intrusion detection model for software defined network architecture [J]. Journal of Computer Applications, 2020, 40(1): 116-122.
[9]	WANG Jiaxin, FENG Yi, YOU Rui. Network security measurment based on dependency relationship graph and common vulnerability scoring system [J]. Journal of Computer Applications, 2019, 39(6): 1719-1727.
[10]	DU Junxiong, CHEN Wei, LI Xueyan. Contextual authentication method based on device fingerprint of Internet of Things [J]. Journal of Computer Applications, 2019, 39(2): 464-469.
[11]	WANG Cheng, CUI Ziwei, DU Zilin, GAO Yueer. Repairing of missing bus arrival data based on DBSCAN algorithm and multi-source data [J]. Journal of Computer Applications, 2019, 39(11): 3184-3190.
[12]	WU Xuchen, PIAO Chunhui, JIANG Xuehong. Siting model of electric taxi charging station based on GPU parallel computing [J]. Journal of Computer Applications, 2019, 39(10): 3071-3078.
[13]	MA Youzhong, ZHANG Zhihui, LIN Chunjie. Research progress in similarity join query of big data [J]. Journal of Computer Applications, 2018, 38(4): 978-986.
[14]	CAO Yunpeng, WANG Haifeng. Communication optimization for intermediate data of MapReduce computing model [J]. Journal of Computer Applications, 2018, 38(4): 1078-1083.
[15]	ZHANG Chengchang, ZHANG Huayu, LUO Jianchang, HE Feng. Massive data analysis of power utilization based on improved K-means algorithm and cloud computing [J]. Journal of Computer Applications, 2018, 38(1): 159-164.

Multi-source data parallel preprocessing method based on similar connection

基于相似连接的多源数据并行预处理方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics