Stream data anomaly detection method based on long short-term memory network and sliding window

doi:10.11772/j.issn.1001-9081.2019111970

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (5): 1335-1339.DOI: 10.11772/j.issn.1001-9081.2019111970

• Data science and technology • Previous Articles Next Articles

Stream data anomaly detection method based on long short-term memory network and sliding window

QIU Yuan¹, Chang Xiangmao¹, QIU Qian², PENG Cheng¹, SU Shanting¹

1.College of Computer Science and Technology, Nanjing University of Aeronautics and Astronautics, NanjingJiangsu 211106, China
2.College of Artificial Intelligence and Data Science, Hebei University of Technology, Tianjin 300401, China

Received:2019-11-04 Revised:2019-11-25 Online:2020-05-10 Published:2020-05-15
Contact: CHANG Xiangmao, born in 1982, Ph. D., associate professor. His research interests include internet of things, intelligent health monitoring based on wearable devices, sensory data processing and analysis of machine learning algorithms.
About author:QIU Yuan, born in 1995, M. S. candidate. Her research interests include anomaly detection, deep learning.CHANG Xiangmao, born in 1982, Ph. D., associate professor. His research interests include internet of things, intelligent health monitoring based on wearable devices, sensory data processing and analysis of machine learning algorithms.QIU Qian, born in 1995, M. S. candidate. Her research interests include social network, deep learning.PENG Cheng, born in 1995, M. S. candidate. His research interests include state detection, deep learning.SU Shanting, born in 1994, M. S. candidate. Her research interests include fault detection, machine learning.

基于长短期记忆网络和滑动窗口的流数据异常检测方法

仇媛¹, 常相茂¹, 仇倩², 彭程¹, 苏善婷¹

1.南京航空航天大学计算机科学与技术学院，南京 211106
2.河北工业大学人工智能与数据科学学院，天津 300401

通讯作者: 常相茂(1982—)
作者简介:仇媛(1995—)，女，河北石家庄人，硕士研究生，CCF会员，主要研究方向：异常检测、深度学习；常相茂(1982—),男,山东淄博人,副教授, 博士，主要研究方向：物联网、基于可穿戴设备的智能健康监测、机器学习算法的感知数据处理及分析；仇倩(1995—)，女，河北石家庄人，硕士研究生，主要研究方向：社会网络、深度学习；彭程(1995—)，男，安徽合肥人，硕士研究生，CCF会员，主要研究方向：状态检测、深度学习；苏善婷(1994—)，女，江苏常熟人，硕士研究生，CCF会员，主要研究方向：故障检测、机器学习。

Abstract

Abstract:

Aiming at the characteristics of large volume, rapid generation and concept drift of current stream data, a stream data anomaly detection method based on Long Short-Term Memory (LSTM) network and sliding window was proposed. Firstly, the LSTM network was used for data prediction, and the difference between the predicted value and the actual value was calculated. For each datum, the appropriate sliding window was selected, and the distribution modeling was performed to all the differences in the sliding window interval, then the probability of data anomaly was calculated according to the probability density of each difference in the current distribution. The LSTM network was not only able to predict data, but also able to predict and learn at the same time, as well as update and adjust the network in real time to ensure the validity of the model. The use of sliding windows was able to make the allocation of abnormal scores more reasonable. Finally, the simulation data made on the basis of real data were used for experiment. The experimental results verify that the average Area Under Curve (AUC) value of the proposed method in low-noise environment is 0.187 and 0.05 higher than that of direct difference detection and Abnormal data Distribution Modeling (ADM) method, respectively.

Key words: stream data, anomaly detection, sliding window, Long Short-Term Memory (LSTM) network, neural network

摘要：

针对目前流数据存在数量巨大、生成迅速和概念漂移的特点，提出了一种基于长短期记忆(LSTM)网络和滑动窗口的流数据异常检测方法。首先采用LSTM网络进行数据预测，之后计算预测值与实际值的差值。对于每个数据，选择合适的滑动窗口，将滑动窗口区间内的所有差值进行分布建模，再根据每个差值在当前分布的概率密度来计算数据异常可能性。LSTM网络不仅可以进行数据预测，还可以边预测边学习，实时更新调整网络，保证模型的有效性；而利用滑动窗口可以使得异常分数的分配更为合理。最后使用在真实数据基础上制造的模拟数据进行了实验。实验结果验证了所提方法在低噪声环境下比直接利用差值进行检测和异常数据分布建模法（ADM）方法的平均曲线下面积(AUC)值分别提高了0.187和0.05。

关键词: 流数据, 异常检测, 滑动窗口, 长短期记忆网络, 神经网络

CLC Number:

TP391.4

QIU Yuan, Chang Xiangmao, QIU Qian, PENG Cheng, SU Shanting. Stream data anomaly detection method based on long short-term memory network and sliding window[J]. Journal of Computer Applications, 2020, 40(5): 1335-1339.

仇媛, 常相茂, 仇倩, 彭程, 苏善婷. 基于长短期记忆网络和滑动窗口的流数据异常检测方法[J]. 计算机应用, 2020, 40(5): 1335-1339.

References

1 WU K , ZHANG K , FAN W , et al . RS-Forest: a rapid density estimator for streaming anomaly detection[C]// Proceedings of the 2014 IEEE International Conference on Data Mining. Piscataway: IEEE, 2014:600-609.
2 SUN D , HU Y , SHI Z , et al . An efficient anomaly detection framework for electromagnetic streaming data[C]// Proceedings of the 4th International Conference on Big Data and Computing. New York: ACM, 2019:151-155.
3 DING Z , FEI M . An anomaly detection approach based on isolation forest algorithm for streaming data using sliding window[J]. IFAC Proceedings Volumes, 2013, 46(20):12-17.
4 GUPTA M , GAO J , AGGARWAL C C , et al . Outlier detection for temporal data: a survey[J]. IEEE Transactions on Knowledge and Data Engineering, 2014, 26(9):2250-2267.
5 THAKKAR P , VALA J , PRAJAPATI V . Survey on outlier detection in data stream[J]. International Journal of Computer Applications, 2016, 136(2):13-16.
6 LI X , HAN J . Mining approximate top-k subspace anomalies in multi-dimensional time-series data[EB/OL].[2019-05-20]. https:// pdfs.semanticscholar.org/fe68/274342529b66baa0f38e026b607b-63faba9c.pdf.
7 SOLBERG H E , LAHTI A . Detection of outliers in reference distributions: performance of Horn’s algorithm[J]. Clinical Chemistry, 2005, 51(12):2326-2332.
8 李春生,于澍,刘小刚 . 基于改进距离和的异常点检测算法研究[J].计算机技术与发展, 2019, 29(3):97-100. (LI C S, YU S, LIU X G. Research on outlier detection algorithm based on improved distance [J]. Computer Technology and Development, 2019, 29(3):97-100.)
9 蒋华,张红福,罗一迪,等 . 基于KL距离的自适应阈值网络流量异常检测[J]. 计算机工程, 2019, 45(4):108-113, 118. JIANG H , ZHANG H F , LUO Y D , et al . Adaptive threshold network traffic anomaly detection based on KL distance[J]. Computer Engineering, 2019, 45(4): 108-113, 118.
10 JIN W , TUNG A K H , HAN J , et al . Ranking outliers using symmetric neighborhood relationship [C]// Proceedings of the 2006 Pacific-Asia Conference on Knowledge Discovery and Data Mining, LNCS 3918. Berlin: Springer, 2006: 577-593.
11 阮嘉琨,蔡延光,乐冰 . 基于DBSCAN密度聚类算法的高速公路交通流异常数据检测[J]. 工业控制计算机, 2019, 32(7):92-94. (RUAN J K, CAI Y G, LE B. Highway traffic anomaly data detection based on DBSCAN density clustering algorithm[J]. Industrial Control Computer, 2019, 32(7):92-94.)
12 PAN X , TAN J , KAVULYA S , et al . GaneSHA: black-Box diagnosis of MapReduce systems[J]. ACM SIGMETRICS Performance Evaluation Review, 2010, 37(3):8-13.
13 GUPTA M , SHARMA A B , CHEN H , et al . Context-aware time series anomaly detection for complex systems[EB/OL].[2019-05-20]. https://www.microsoft.com/en-us/research/wp-content/uploads/2013/01/gupta13_sdma.pdf.
14 CHANDOLA V , MITHAL V , KUMAR V . Comparative evaluation of anomaly detection techniques for sequence data[C]// Proceedings of the 8th IEEE International Conference on Data Mining. Piscataway: IEEE, 2008:743-748.
15 SUN P , CHAWLA S , ARUNASALAM . Mining for outliers in sequential databases[EB/OL].[2019-05-20].https://archive.siam.org/meetings/sdm06/proceedings/009sunp.pdf.
16 CHEN P Y , YANG S , MCCANN J A . Distributed real-time anomaly detection in networked industrial sensing systems[J]. IEEE Transactions on Industrial Electronics, 2015, 62(6):3832-3842.
17 FARIA E R , GAMA J , CARVALHO A C P L F . Novelty detection algorithm for data streams multi-class problems[C]// Proceedings of the 28th Annual ACM Symposium on Applied Computing. New York: ACM, 2013:795-800.
18 YU Y , GUO S , LAN S , et al . Anomaly intrusion detection for evolving data stream based on semi-supervised learning[C]// Proceedings of the 15th International Conference on Neural Information Processing, LNCS 5506. Berlin: Springer, 2008:571-578.
19 AHMAD S , LAVIN A , PURDY S , et al . Unsupervised real-time anomaly detection for streaming data[J]. Neurocomputing, 2017, 262: 134-147.
20 SZMIT M , SZMIT A . Use of holt-winters method in the analysis of network traffic: case study[C]// Proceedings of the 2011 International Conference on Computer Networks, CCIS 160. Berlin: Springer, 2011:224-231.
21 BASSEVILLE M , NIKIFOROV I V . Detection of Abrupt Change Theory and Application[M]. Upper Saddle River: Prentice Hall, 1993: 23-26.
22 BIANCO A M , BEN M G, MARTíNEZ E J , et al . Outlier detection in regression models with ARIMA errors using robust estimates[J]. Journal of Forecasting, 2001, 20(8):565-579.
23 GREFF K , SRIVASTAVA R K , KOUTNíK J , et al . LSTM: a search space odyssey[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(10):2222-2232.
24 Machine Learning Repository UCI . Condition monitoring of hydraulic systems[DB/OL]. [2019-07-20] http://archive.ics.uci.edu/ml/datasets/Condition+monitoring+of+hydraulic+systems.
25 ZHANG Y , SZABO C , SHENG Q Z . Cleaning environmental sensing data streams based on individual sensor reliability[C]// Proceedings of the 2014 International Conference on Web Information Systems Engineering, LNCS 8787. Cham: Springer, 2014: 405- 414.
26 ZIMEK A , CAMPELLO R J G B , SANDER J . Ensembles for unsupervised outlier detection: challenges and research questions a position paper[J]. ACM SIGKDD Explorations Newsletter, 2014,15(1):11-22.

[1]	SONG Zhongshan, LIANG Jiarui, ZHENG Lu, LIU Zhenyu, TIE Jun. Remote sensing scene classification based on bidirectional gated scale feature fusion [J]. Journal of Computer Applications, 2021, 41(9): 2726-2735.
[2]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[3]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[4]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[5]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[6]	MOU Changning, WANG Haipeng, ZHOU Piyu, HOU Xinhang. De novo peptide sequencing by tandem mass spectrometry based on graph convolutional neural network [J]. Journal of Computer Applications, 2021, 41(9): 2773-2779.
[7]	LIU Zichen, LI Xiaojuan, WEI Wei. Automatic patent price evaluation based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(9): 2532-2538.
[8]	JING Xinghong, SUN Guodong, HE Shibiao, LIAO Yong. Time-varying channel estimation method based on sliding window filtering and polynomial fitting [J]. Journal of Computer Applications, 2021, 41(9): 2699-2704.
[9]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[10]	DING Yin, SANG Nan, LI Xiaoyu, WU Feizhou. Prediction method of capacity data in telecom industry based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(8): 2373-2378.
[11]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[12]	HUANG Chengcheng, DONG Xiaoxiao, LI Zhao. Deep pipeline 5×5 convolution method based on two-dimensional Winograd algorithm [J]. Journal of Computer Applications, 2021, 41(8): 2258-2264.
[13]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[14]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[15]	HU Tianjie, HU Wenjun, WANG Shitong. Distribution entropy penalized support vector data description [J]. Journal of Computer Applications, 2021, 41(8): 2212-2218.

Stream data anomaly detection method based on long short-term memory network and sliding window

基于长短期记忆网络和滑动窗口的流数据异常检测方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics