Anomaly detection method based on cumulative probability fluctuation and automated clustering

doi:10.11772/j.issn.1001-9081.2024121792

Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (12): 3864-3871.DOI: 10.11772/j.issn.1001-9081.2024121792

• Data science and technology • Previous Articles Next Articles

Anomaly detection method based on cumulative probability fluctuation and automated clustering

Jun ZENG¹^,², Yinghua TONG¹^,², Defang WANG¹^,²

^1.College of Computer，Qinghai Normal University，Xining Qinghai 810008，China
^2.The State Key Laboratory of Tibetan Intelligent Information Processing and Application （Qinghai Normal University），Xining Qinghai 810008，China

Received:2024-12-19 Revised:2025-04-01 Accepted:2025-04-07 Online:2025-04-22 Published:2025-12-10
Contact: Yinghua TONG
About author:ZENG Jun， born in 1999， M. S. candidate. His research interests include anomaly detection of internet of things data.
TONG Yinghua， born in 1982， Ph. D.， associate professor. Her research interests include fault tolerance in wireless sensor networks， reliability analysis of internet of things.
WANG Defang， born in 1978， M. S.， associate professor. His research interests include data anomaly detection.
Supported by:
Applied Basic Research Project of Qinghai Province(2023-ZJ-713)

基于累积概率波动和自动化聚类的异常检测方法

曾君¹^,², 童英华¹^,², 王得芳¹^,²

^1.青海师范大学计算机学院，西宁 810008
^2.省部共建藏语智能信息处理及应用国家重点实验室（青海师范大学），西宁 810008

通讯作者: 童英华
作者简介:曾君（1999—），男，湖南株洲人，硕士研究生，主要研究方向：物联网数据异常检测
童英华（1982—），女，青海西宁人，副教授，博士，CCF会员，主要研究方向：无线传感器网络容错、物联网可靠性分析
王得芳（1978—），男，青海西宁人，副教授，硕士，主要研究方向：数据异常检测。
基金资助:
青海省应用基础研究项目(2023-ZJ-713)

Abstract

Abstract:

With the increasing complexity of multidimensional data features， the existing anomaly detection methods have limitations in capturing feature distribution. At the same time， traditional clustering and statistical methods encounter greater challenges in parameter selection， which limit the improvement of detection performance together. To address this issue， an anomaly detection method based on cumulative probability fluctuation and automated clustering was proposed. Firstly， cumulative probability fluctuation of the features was calculated to represent the Gaussian mixture distribution of the features， and the features were compression transformed according to the cumulative probability fluctuation values. Secondly， deep reinforcement learning was employed to search optimal clustering parameters in Density-Based Spatial Clustering of Applications with Noise （DBSCAN）， and the compression transformed dataset was clustered. Finally， the clustering results of the data were combined with the cumulative probability fluctuation values of the data features to determine data point anomalies. Experimental results show that the average precision， recall， F1-score， and Area Under ROC （Receiver Operating Characteristic） Curve （AUC） of the proposed method on six experimental datasets are 36.39%， 2.73%， 14.90%， and 4.84% higher than those of the best performing method among the comparison methods. It can be seen that the proposed method improves the comprehensive performance of anomaly detection for data with multi-dimensional complex features effectively without selecting parameters manually.

Key words: anomaly detection, deep reinforcement learning, data feature transformation, data clustering, cumulative probability fluctuation

摘要：

随着多维数据特征的复杂性不断增加，现有的异常检测方法在捕捉特征分布方面表现出局限性，同时传统的聚类和统计方法在参数选择上遇到了更大的挑战，这些共同导致了检测性能的提升受到限制。针对上述问题，提出一种基于累积概率波动和自动化聚类的异常检测方法。首先，计算特征的累积概率波动以表征特征的高斯混合分布，并根据累积概率波动值对特征进行压缩变换；其次，在基于密度的噪声应用空间聚类（DBSCAN）中利用深度强化学习寻找最优聚类参数，并对压缩变换后的数据集进行聚类；最后，综合数据的聚类结果与数据特征的累积概率波动值判断数据点是否异常。实验结果表明，所提方法在6个实验数据集上的平均精确率、召回率、F1分数和受试者工作特征曲线下面积（AUC）相较于对比方法中表现最好的方法分别提升了36.39%、2.73%、14.90%和4.84%。可见，所提方法在无需手动选择参数的情况下，有效提高了对多维复杂特征数据异常检测的综合性能。

关键词: 异常检测, 深度强化学习, 数据特征变换, 数据聚类, 累积概率波动

CLC Number:

TP391.4

Jun ZENG, Yinghua TONG, Defang WANG. Anomaly detection method based on cumulative probability fluctuation and automated clustering[J]. Journal of Computer Applications, 2025, 45(12): 3864-3871.

曾君, 童英华, 王得芳. 基于累积概率波动和自动化聚类的异常检测方法[J]. 《计算机应用》唯一官方网站, 2025, 45(12): 3864-3871.

Figures/Tables 12

Fig.1 Overall framework of proposed method

Fig.2 GMM fitting data distribution curves

Fig.3 Cumulative probability fluctuation values of data

Fig.4 Comparison of features before and after transformation for two datasets

Fig.5 Markov process of parameter search

Fig.6 Architecture of TD3 network

Fig.7 Recursive parameter search

Fig.8 Anomaly detection flow

Tab.1 Overview of datasets

数据集名称	特征数	异常占比/%	类别
WBC	9	4.48	Healthcare
musk	166	3.17	Chemistry
WDBC	30	2.70	Healthcare
Cardio	21	9.61	Healthcare
Wine	13	7.75	Chemistry
Aggregation	2	2.83	Shape
Compound	2	3.86	Shape

Fig.9 Comparison of metrics with different clustering parameters on Aggregation dataset

Fig.10 Comparison of performance metrics of different methods on different datasets

Fig.11 Performance comparison of ablation methods

References 22

[1]	PANG G， SHEN C， CAO L， et al. Deep learning for anomaly detection： a review［J］. ACM Computing Surveys， 2022， 54（2）： No.38.
[2]	WANG H， BAH M J， HAMMAD M. Progress in outlier detection techniques： a survey ［J］. IEEE Access， 2019， 7： 107964-108000.
[3]	ILEBERI E， SUN Y， WANG Z. A machine learning based credit card fraud detection using the GA algorithm for feature selection［J］. Journal of Big Data， 2022， 9： No.24.
[4]	SULAIMAN R BIN， SCHETININ V， SANT P. Review of machine learning approach on credit card fraud detection［J］. Human-Centric Intelligent Systems， 2022， 2（1/2）： 55-68.
[5]	HE M， WANG X， WEI P， et al. Reinforcement learning meets network intrusion detection： a transferable and adaptable framework for anomaly behavior identification［J］. IEEE Transactions on Network and Service Management， 2024， 21（2）： 2477-2492.
[6]	LI M M， HUANG K， ZITNIK M. Graph representation learning in biomedicine and healthcare［J］. Nature Biomedical Engineering， 2022， 6（12）： 1353-1369.
[7]	FERNANDO T， GAMMULLE H， DENMAN S， et al. Deep learning for medical anomaly detection： a survey［J］. ACM Computing Surveys， 2022， 54（7）： No.141.
[8]	ZAHEER M Z， MAHMOOD A， KHAN M H， et al. Generative cooperative learning for unsupervised video anomaly detection ［C］// Proceedings of the 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2022： 14724-14734.
[9]	XU R， GUO Y， HAN X， et al. OpenCDA： an open cooperative driving automation framework integrated with co-simulation［C］// Proceedings of the 2021 IEEE International Intelligent Transportation Systems Conference. Piscataway： IEEE， 2021： 1155-1162.
[10]	YANG L， CHEN J， GAO S， et al. Try with simpler： an evaluation of improved principal component analysis in log-based anomaly detection ［J］. ACM Transactions on Software Engineering and Methodology， 2024， 33（5）： No.115.
[11]	QIAO Y， WU K， JIN P. Efficient anomaly detection for high-dimensional sensing data with one-class support vector machine［J］. IEEE Transactions on Knowledge and Data Engineering， 2023， 35（1）： 404-417.
[12]	BERGMAN L， HOSHEN Y. Classification-based anomaly detection for general data ［EB/OL］. ［2024-05-05］. .
[13]	MAO J， HU Y， JIANG D， et al. CBFS： a clustering-based feature selection mechanism for network anomaly detection［J］. IEEE Access， 2020， 8： 116216-116225.
[14]	CHADHA G S， ISLAM I， SCHWUNG A， et al. Deep convolutional clustering-based time series anomaly detection ［J］. Sensors， 2021， 21（16）： No.5488.
[15]	YANG X， LATECKI L J， POKRAJAC D. Outlier detection with globally optimal exemplar based GMM ［C］// Proceedings of the 2009 SIAM International Conference on Data Mining. Philadelphia， PA： SIAM， 2009： 145-154.
[16]	ZHANG R， PENG H， DOU Y， et al. Automating DBSCAN via deep reinforcement learning ［C］// Proceedings of the 31st ACM International Conference on Information and Knowledge Management. New York： ACM， 2022： 2620-2630.
[17]	SOYDANER D. Attention mechanism in neural networks： where it comes and where it goes ［J］. Neural Computing and Applications， 2022， 34（16）： 13371-13385.
[18]	LI Z， ZHAO Y， HU X， et al. ECOD： unsupervised outlier detection using empirical cumulative distribution functions［J］. IEEE Transactions on Knowledge and Data Engineering， 2023， 35（12）： 12181-12193.
[19]	GOODGE A， HOOI B， NG S K， et al. LUNAR： unifying local outlier detection methods via graph neural networks［C］// Proceedings of the 36th AAAI Conference on Artificial Intelligence. Palo Alto： AAAI Press， 2022： 6737-6745.
[20]	PANG J， PU X， LI C. A hybrid algorithm incorporating vector quantization and one-class support vector machine for industrial anomaly detection［J］. IEEE Transactions on Industrial Informatics， 2022， 18（12）： 8786-8796.
[21]	HE Z， XU X， DENG S. Discovering cluster-based local outliers［J］. Pattern Recognition Letters， 2003， 24（9/10）： 1641-1650.
[22]	XU H， PANG G， WANG Y， et al. Deep isolation forest for anomaly detection ［J］. IEEE Transactions on Knowledge and Data Engineering， 2023， 35（12）： 12591-12604.

Anomaly detection method based on cumulative probability fluctuation and automated clustering

基于累积概率波动和自动化聚类的异常检测方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 22

Related Articles 15

Recommended Articles

Metrics

[1]	Tianyu XUE, Aiping LI, Liguo DUAN. Vehicular edge computing scheme with task offloading and resource optimization [J]. Journal of Computer Applications, 2025, 45(6): 1766-1775.
[2]	Longbo YAN, Wentao MAO, Zhihong ZHONG, Lilin FAN. Robust unsupervised multi-task anomaly detection method for defect diagnosis of urban drainage pipe network [J]. Journal of Computer Applications, 2025, 45(6): 1833-1840.
[3]	Wenpeng WANG, Yinchang QIN, Wenxuan SHI. Review of unsupervised deep learning methods for industrial defect detection [J]. Journal of Computer Applications, 2025, 45(5): 1658-1670.
[4]	Zihe CHEN, Bin CHEN. Unsupervised point cloud anomaly detection based on multi-representation fusion [J]. Journal of Computer Applications, 2025, 45(5): 1677-1685.
[5]	Pengcheng XU, Lei HE, Chuan LI, Weiqi QIAN, Tun ZHAO. Deep symbolic regression method based on Transformer [J]. Journal of Computer Applications, 2025, 45(5): 1455-1463.
[6]	Lihu PAN, Shouxin PENG, Rui ZHANG, Zhiyang XUE, Xuzhen MAO. Video anomaly detection for moving foreground regions [J]. Journal of Computer Applications, 2025, 45(4): 1300-1309.
[7]	Zhenhua XUE, Qiang LI, Chao HUANG. Vision foundation model-driven pixel-level image anomaly detection method [J]. Journal of Computer Applications, 2025, 45(3): 823-831.
[8]	Zhiqiang REN, Xuebin CHEN. FedAud： adaptive defense mechanism based on historical model updates [J]. Journal of Computer Applications, 2025, 45(2): 490-496.
[9]	Jing WANG, Xuming FANG. Intelligent joint power and channel allocation algorithm for Wi-Fi7 multi-link integrated communication and sensing [J]. Journal of Computer Applications, 2025, 45(2): 563-570.
[10]	Dixin WANG, Jiahao WANG, Min LI, Hao CHEN, Guangyao HU, Yu GONG. Abnormal attack detection for underwater acoustic communication network [J]. Journal of Computer Applications, 2025, 45(2): 526-533.
[11]	Miaolei DENG, Yupei KAN, Chuanchuan SUN, Haihang XU, Shaojun FAN, Xin ZHOU. Summary of network intrusion detection systems based on deep learning [J]. Journal of Computer Applications, 2025, 45(2): 453-466.
[12]	Huahua WANG, Liang HUANG, Jiajie CHEN, Jiening FANG. Dynamic allocation algorithm for multi-beam subcarriers of low orbit satellites based on deep reinforcement learning [J]. Journal of Computer Applications, 2025, 45(2): 571-577.
[13]	Haoqun ZHENG, Lizhi CAI, Kang YANG, Xiaoyu WANG. Intrusion detection method with multi-stage fusion for internet of medical things [J]. Journal of Computer Applications, 2025, 45(12): 3909-3915.
[14]	Lin WEI, Shihao ZHANG, Mengyang HE. Workflow task optimization and energy-efficient offloading method for computing power network [J]. Journal of Computer Applications, 2025, 45(12): 3916-3924.
[15]	Lin WEI, Jinyang LI, Yajie WANG, Mengyang HE. Highly reliable matching method based on multi-dimensional resource measurement and rescheduling in computing power network [J]. Journal of Computer Applications, 2025, 45(11): 3632-3641.