Time series outlier detection based on sliding window prediction

doi:10.11772/j.issn.1001-9081.2014.08.2217

Journal of Computer Applications ›› 2014, Vol. 34 ›› Issue (8): 2217-2220.DOI: 10.11772/j.issn.1001-9081.2014.08.2217

Previous Articles Next Articles

Time series outlier detection based on sliding window prediction

YU Yufeng¹,ZHU Yuelong¹,WAN Dingsheng¹,GUAN Xingzhong²

1. College of Computer and Information, Hohai University, Nanjing Jiangsu 210098, China;
2. Hydrology Bureau of Jiangxi Province, Nanchang Jiangxi 330018, China

Received:2014-04-29 Revised:2014-05-07 Online:2014-08-10 Published:2014-08-01
Contact: YU Yufeng

基于滑动窗口预测的水文时间序列异常检测

余宇峰¹,朱跃龙¹,万定生¹,关兴中²

1. 河海大学计算机与信息学院，南京210098
2. 江西省水文局，南昌330018

通讯作者: 余宇峰
作者简介:余宇峰(1979-)，男，湖北黄冈人，高级实验师，博士研究生，主要研究方向：数据挖掘、数据质量、水利信息化；朱跃龙(1959-),男，江苏建湖人，教授，博士，主要研究方向：智能信息处理、数据挖掘；万定生(1963-)，男，江苏溧阳人，教授，主要研究方向：水利信息化、数据挖掘。
基金资助:
国家自然科学基金资助项目;水利部948项目

Abstract

Abstract:

To solve data quality problems for hydrological time series analysis and decision-making, a new prediction-based outlier detection algorithm was proposed. The method first split given hydrological time series into subsequences so as to build a forecasting model to predict future values, and then outliers were assumed to take place if the difference between predicted and observed values was above a certain threshold. The setup of sliding window and parameters in the detection algorithm were analyzed, and the corresponding result was validated with the real data. The experimental results show that the proposed algorithm can effectively detect the outliers in time series and improves the sensitivity and specificity to at least 80 percent and 98 percent respectively.

摘要：

针对水文时间序列分析与决策中存在的数据质量问题，提出了基于滑动窗口预测的水文时间序列异常检测算法。首先基于滑动窗口对时间序列进行子序列分割，再以子序列为基础建立预测模型对未来值进行预测，并将预测值和实测值间差异范围大于预设阈值的序列点判定为异常。探讨了算法中的滑动窗口和参数设置，并以实例数据对算法进行了验证。实验结果表明，所提算法不仅能够有效挖掘出水文时间序列中的异常点，而且将异常检测的灵敏度和特异度分别提高到80%和98%以上。

CLC Number:

YU Yufeng ZHU Yuelong WAN Dingsheng GUAN Xingzhong. Time series outlier detection based on sliding window prediction[J]. Journal of Computer Applications, 2014, 34(8): 2217-2220.

余宇峰朱跃龙万定生关兴中. 基于滑动窗口预测的水文时间序列异常检测[J]. 计算机应用, 2014, 34(8): 2217-2220.

References

［1］YANG Q, WU X. 10 challenging problems in data mining research ［J］. International Journal of Information Technology and Decision Making, 2006, 5(4): 597-604.
［2］HAN J, KAMBER M, PEI J. Data mining: concepts and tech-niques ［M］. Waltham: Morgan Kaufmann Publishers, 2006.
［3］GAO B, MA H Y, YANG Y H. HMMs (Hidden Markov Models) based on anomaly intrusion detection method ［C］// Proceedings of the 2002 International Conference on Machine Learning and Cybernetics. Piscataway: IEEE Press, 2002: 381-385.
［4］CHANDOLA V, BANERJEE A, KUMAR V. Anomaly detection: a survey ［J］. ACM Computing Surveys, 2009, 41(3): 15.
［5］RAMASWAMY S, RASTOGI R, SHIM K. Efficient algorithms for mining outliers from large data sets ［C］// ACM SIGMOD Record, 2000, 29(2): 427-438.
［6］YU H, WANG B, XIAO G, et al. Distance-based outlier detection on uncertain data ［J］. Journal of Computer Research and Development, 2010, 47(3): 474-484.(于浩,王斌,肖刚,等.基于距离的不确定离群点检测［J］.计算机研究与发展,2010,47(3):474-484.)
［7］BREUNIG M M, KRIEGEL H P, NG R T, et al. LOF: identifying density-based local outliers ［J］. ACM SIGMOD Record, 2000, 29(2): 93-104.
［8］XUE A, JU S, HE W, et al. Study on algorithms for local outlier detection ［J］. Chinese Journal of Computers, 2007, 30(8): 1455-1463.(薛安荣,鞠时光,何伟华,等.局部离群点挖掘算法研究[J].计算机学报,2007,30(8):1455-1463.)
［9］MOUNCE S R, MOUNCE R B, BOXALL J B. Novelty detection for time series data analysis in water distribution systems using support vector machines［J］. Journal of Hydroinformatics, 2011,13(4):672-686.
［10］TIAN J, GU H. Outlier one class support vector machines ［J］. Journal of Electronics and Information Technology, 2010, 32(6): 1284-1288.(田江,顾宏.孤立点一类支持向量机算法研究［J］.电子与信息学报,2010,32(6):1284-1288.)
［11］BUDALAKOTI S, SRIVASTAVA A, AKELLA R, et al. Anomaly detection in large sets of high-dimensional symbol sequences, NASA TM-2006-214553 ［R］. Moffett Field: NASA Ames Research Center, 2006.
［12］HILL D J, MINSKER B S. Anomaly detection in streaming environmental sensor data: a data-driven modeling approach ［J］. Environmental Modelling and Software, 2010, 25(9): 1014-1022.
［13］BASU S, MECKESHEIMER M. Automatic outlier detection for time series: an application to sensor data ［J］. Knowledge and Information Systems—Special Issue on Mining Low-Quality Data, 2007, 11(2): 137-154.
［14］POKRAJAC D, LAZAREVIC A, LATECKI L J. Incremental local outlier detection for data streams ［C］// CIDM 2007: Proceedings of the 2007 IEEE Symposium on Computational Intelligence and Data Mining. Piscataway: IEEE Press, 2007: 504-515.
［15］NG W W, PANU U S, LENNOX W C. Chaos based analytical techniques for daily extreme hydrological observations ［J］. Journal of Hydrology, 2007, 342(1): 17-41.
［16］FAWCETT T. An introduction to ROC analysis ［J］. Pattern Recognition Letters, 2006, 27(8): 861-874.

[1]	Jiepo FANG, Chongben TAO. Hybrid internet of vehicles intrusion detection system for zero-day attacks [J]. Journal of Computer Applications, 2024, 44(9): 2763-2769.
[2]	Hang XU, Zhi YANG, Xingyuan CHEN, Bing HAN, Xuehui DU. Coverage-guided fuzzing based on adaptive sensitive region mutation [J]. Journal of Computer Applications, 2024, 44(8): 2528-2535.
[3]	. Summary of network intrusion detection systems based on deep learning [J]. Journal of Computer Applications, 0, (): 0-0.
[4]	Zihao YAO, Yuanming LI, Ziqiang MA, Yang LI, Lianggen WEI. Multi-object cache side-channel attack detection model based on machine learning [J]. Journal of Computer Applications, 2024, 44(6): 1862-1871.
[5]	. Location privacy-preserving recommendation scheme based on federated graph neural network [J]. Journal of Computer Applications, 0, (): 0-0.
[6]	XiangJun LI, Junhong WANG, Shilu WANG, Jinxia CHEN, Jitao SUN, Jianhui WANG. Malicious traffic detection method based on multi-model parallel fusion network [J]. Journal of Computer Applications, 2023, 43(S2): 122-129.
[7]	Jingcheng XU, Xuebin CHEN, Yanling DONG, Jia YANG. DDoS attack detection by random forest fused with feature selection [J]. Journal of Computer Applications, 2023, 43(11): 3497-3503.
[8]	Shanshan HUO, Yanjun LI, Jian LIU, Yinshuang LI. Design and implementation of cipher component security criteria testing tool [J]. Journal of Computer Applications, 2023, 43(10): 3156-3161.
[9]	Xiang GUO, Wengang JIANG, Yuhang WANG. Encrypted traffic classification method based on improved Inception-ResNet [J]. Journal of Computer Applications, 2023, 43(8): 2471-2476.
[10]	Kui JIANG, Zhihang YU, Xiaolei CHEN, Yuhao LI. Design and implementation of Webshell traffic detection system based on BERT-CNN [J]. Journal of Computer Applications, 2023, 43(S1): 126-132.
[11]	Chao ZHENG, Yueting WU, Ke XIAO. Intrusion detection based on federated learning and deep residual network [J]. Journal of Computer Applications, 2023, 43(S1): 133-138.
[12]	Yiting SUN, Yue GUO, Changjin LI, Hongjun ZHANG, Kang LIU, Junjiao Liu, Limin SUN. Intrusion detection method for control logic injection attack against programmable logic controller [J]. Journal of Computer Applications, 2023, 43(6): 1861-1869.
[13]	Haiyan HU, Qiaoyan KANG, Shuo ZHAO, Jianfeng WANG, Youbin FU. Service function chain deployment optimization method based on node comprehensive importance ranking [J]. Journal of Computer Applications, 2023, 43(3): 860-868.
[14]	Jingyun CHENG, Buhong WANG, Peng LUO. Static code defect detection method based on deep semantic fusion [J]. Journal of Computer Applications, 2022, 42(10): 3170-3176.
[15]	Wenting BI, Haitao LIN, Liqun ZHANG. Moving target defense decision-making algorithm based on multi-stage evolutionary signal game model [J]. Journal of Computer Applications, 2022, 42(9): 2780-2787.

Time series outlier detection based on sliding window prediction

基于滑动窗口预测的水文时间序列异常检测

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics