Novelty detection method based on dual autoencoders and Transformer network

doi:10.11772/j.issn.1001-9081.2021111983

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (1): 22-29.DOI: 10.11772/j.issn.1001-9081.2021111983

Special Issue: 人工智能

• Artificial intelligence • Previous Articles Next Articles

Novelty detection method based on dual autoencoders and Transformer network

ZHOU Jiahang^1,2, XING Hongjie^1,2

1.College of Mathematics and Information Science， Hebei University， Baoding Hebei 071002， China
2.Hebei Key Laboratory of Machine Learning and Computational Intelligence（Hebei University）， Baoding Hebei 071002， China

Received:2021-11-23 Revised:2022-06-06 Online:2022-06-20
Contact: XING Hongjie， born in 1976， Ph. D.， professor. His research interests include kernel methods， neural networks， novelty detection.
About author:ZHOU Jiahang，born in 1997， M.S. cadidate. His research interests include novelty detection， autoencoder；
Supported by:
This work is partially supported by National Natural Science Foundation of China （61672205）， Natural Science Foundation of Hebei Province （F2017201020）.

基于双自编码器和Transformer网络的异常检测方法

周佳航^1,2, 邢红杰^1,2

1.河北大学数学与信息科学学院，河北保定 071002
2.河北省机器学习与计算智能重点实验室（河北大学），河北保定 071002

通讯作者: 邢红杰（1976—），男，河北保定人，教授，博士，主要研究方向：核方法、神经网络、异常检测。hjxing@hbu.edu.cn
作者简介:周佳航（1997—），男，河北邢台人，硕士研究生，主要研究方向：异常检测、自编码器；；
基金资助:
国家自然科学基金资助项目（61672205）；河北省自然科学基金资助项目（F2017201020）。

Abstract

Abstract: AutoEncoder （AE） based novelty detection method utilizes reconstruction error to classify the test samples to be normal or novel data. However， the above method produces very close reconstruction errors on normal data and novel data. Therefore， some novel data are easy to be misclassified as normal data. To solve the above problem， a novelty detection method composed of two parallel AEs and one Transformer network was proposed， namely Novelty Detection based on Dual Autoencoders and Transformer Network （DATN-ND）. Firstly， the bottleneck features of input samples were used by Transformer network to generate the bottleneck features with pseudo-novel data， thereby increasing the novel data information in the training set. Secondly， the bottleneck features with novel data information were reconstructed by the dual AEs to normal data as much as possible， increasing the reconstruction error difference between novel and normal data. Compared with MemAE （Memory-augmented AE）， DATN-ND has the Area Under the Receiver Operating Characteristic curve （AUC） improved by 6.8 percentage points， 12.0 percentage points， and 2.5 percentage points respectively on MNIST， Fashion-MNIST， and CIFAR-10 datasets. Experimental results show that DATN-ND can effectively make the difference of reconstruction error between normal data and abnormal data bigger.

Key words: novelty detection, AutoEncoder (AE), reconstruction error, one-class classification, Transformer network

摘要： 基于自编码器（AE）的异常检测方法利用重构误差判断待测样本是正常数据还是异常数据。然而，上述方法在正常数据与异常数据上产生的重构误差非常接近，导致部分异常数据很容易被错分为正常数据。为解决上述问题，提出一种由两个并行的AE和一个Transformer网络组成的异常检测方法——DATN-ND。首先，Transformer网络利用输入样本的瓶颈特征生成伪异常数据的瓶颈特征，从而在训练集中增加异常数据信息；其次，双AE将带有异常数据信息的瓶颈特征尽可能地重构为正常数据，增加异常数据与正常数据的重构误差差别。与记忆增强自编码器（MemAE）相比，DATN-ND在MNIST、Fashion-MNIST、CIFAR-10数据集上ROC曲线下面积（AUC）分别提升6.8、12.0和2.5个百分点。实验结果表明，DATN-ND能够有效扩大正常数据和异常数据在重构误差上的差别。

关键词: 异常检测, 自编码器, 重构误差, 单类分类, Transformer网络

CLC Number:

TP391.4

ZHOU Jiahang, XING Hongjie. Novelty detection method based on dual autoencoders and Transformer network[J]. Journal of Computer Applications, 2023, 43(1): 22-29.

周佳航, 邢红杰. 基于双自编码器和Transformer网络的异常检测方法[J]. 《计算机应用》唯一官方网站, 2023, 43(1): 22-29.

References

1 吕浩，易鹏飞，刘瑞，等. 用于视频异常检测的时序多尺度自编码器［J］. 图学学报， 2022， 43（2）： 223-229. LYU H， YI P F， LIU R， et al. Temporal multiscale autoencoder for video anomaly detection［J］. Journal of Graphics， 2022， 43（2）： 223-229.
2 NAYAK R， PATI U C， DAS S K. A comprehensive review on deep learning-based methods for video anomaly detection［J］. Image and Vision Computing， 2021， 106： No.104078. 10.1016/j.imavis.2020.104078
3 TARASSENKO L. Novelty detection for the identification of masses in mammograms ［C］// Proceedings of the 4th International Conference on Artificial Neural Networks. Stevenage： IET， 1995： 442-447. 10.1049/cp:19950597
4 PATCHA A， PARK J-M. An overview of anomaly detection techniques： Existing solutions and latest technological trends［J］. Computer Networks， 2007， 51（12）： 3448-3470. 10.1016/j.comnet.2007.02.001
5 MARKOU M， SINGH S. A neural network-based novelty detector for image sequence analysis［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2006， 28（10）： 1664-1677. 10.1109/tpami.2006.196
6 CHALAPATHY R， CHAWLA S. Deep learning for anomaly detection： a survey［EB/OL］. （2019-01-23）［2021-11-03］.https：//arxiv.org/pdf/1901.03407.pdf. 10.1145/3394486.3406704
7 RUFF L， KAUFFMANN J， VANDERMEULEN R A， et al. A unifying review of deep and shallow anomaly detection［J］. Proceedings of the IEEE， 2021， 109（5）： 756-795. 10.1109/jproc.2021.3052449
8 GONG D， TAN M， ZHANG Y， et al. Blind image deconvolution by automatic gradient activation［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 1827-1836. 10.1109/cvpr.2016.202
9 AN J， CHO S. Variational autoencoder based anomaly detection using reconstruction probability［R/OL］. （2019-12-27）［2021-08-12］.http：//dm.snu.ac.kr/static/docs/TR/SNUDM-TR-2015-03.pdf#：~：text=Autoencoder%20based%20anomaly%20detection%20is%20a%20deviation%20based，normal%20instances%20are%20used%20to%20train%20the%20autoencoder.
10 SAKURADA M， YAIRI T. Anomaly detection using autoencoders with nonlinear dimensionality reduction ［C］// Proceedings of the 2nd Workshop on Machine Learning for Sensory Data Analysis. New York： ACM， 2014： 4-11. 10.1145/2689746.2689747
11 XIA Y， CAO X， WEN F， et al. Learning discriminative reconstructions for unsupervised outlier removal［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 1511-1519. 10.1109/iccv.2015.177
12 ZONG B， SONG Q， MIN M R， et al. Deep autoencoding Gaussian mixture model for unsupervised anomaly detection ［EB/OL］. ［2021-09-13］.https：//sites.cs.ucsb.edu/~bzong/doc/iclr18-dagmm.pdf.
13 LOCANTORE N， MARRON J， SIMPSON D， et al. Robust principal component analysis for functional data［J］. Test， 1999， 8（1）： 1-73. 10.1007/bf02595862
14 ZHOU C， PAFFENROTH R C. Anomalydetection with robust deep autoencoders［C］// Proceedings of the 23rd ACM SIGKDD International Conference on Knowledge Discovery and Data Mining， New York： ACM， 2017： 665-674. 10.1145/3097983.3098052
15 GONG D， LIU L， LE V， et al. Memorizing normality to detect anomaly： memory-augmented deep autoencoder for unsupervised anomaly detection［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway： IEEE 2019： 1705-1714. 10.1109/iccv.2019.00179
16 LAI C H， ZOU D M， LERMAN G. Robust subspace recovery layer for unsupervised anomaly detection［EB/OL］. （2019-12-24）［2021-05-09］.https：//arxiv.org/pdf/1904.00152.pdf.
17 KINGMA D P， BA J L. Adam： a method for stochastic optimization［EB/OL］. （2017-01-30）［2021-03-12］.https：//arxiv.org/pdf/1412.6980.pdf.
18 DENG L. The MNIST Database of handwritten digit images for machine learning research ［Best of the Web］［J］. IEEE Signal Processing Magazine， 2012， 29（6）： 141-142. 10.1109/msp.2012.2211477
19 XIAO H， RASUL K， VOLLGRAF R. Fashion-MNIST： a novel image dataset for benchmarking machine learning algorithms［EB/OL］. ［2021-03-17］. https：//arxiv.org/pdf/1708.07747v1.pdf.
20 THAKKAR V， TEWARY S， CHAKRABORTY C. Batch normalization in convolutional neural networks—A comparative study with CIFAR-10 data［C］// Proceedings of the 5th International Conference on Emerging Applications of Information Technology. Piscataway： IEEE， 2018： 1-5. 10.1109/eait.2018.8470438
21 CHEN Y， ZHOU X S， HUANG T S. One-class SVM for learning in image retrieval ［C］// Proceedings of the 2001 International Conference on Image Processing. Piscataway： IEEE， 2001， 1： 34-37.
22 RUFF L， VANDERMEULEN R A， G?RNITZ N， et al. Deep one-class classification［C］// Proceedings of the 35th International Conference on Machine Learning. New York： JMLR.org， 2018： 4393-4402.
23 ZHANG Z， DENG X. Anomaly detection using improved deep SVDD model with data structure preservation［J］. Pattern Recognition Letters， 2021， 148： 1-6. 10.1016/j.patrec.2021.04.020
24 ZHOU Y， LIANG X， ZHANG W， et al. VAE-based deep SVDD for anomaly detection ［J］. Neurocomputing， 2021， 453： 131-140. 10.1016/j.neucom.2021.04.089
25 SCHLEGL T， SEEB?CK P， WALDSTEIN S M， et al. f-AnoGAN： fast unsupervised anomaly detection with generative adversarial networks ［J］. Medical Image Analysis， 2019， 54： 30-44. 10.1016/j.media.2019.01.010
26 AKCAY S， ATAPOUR-ABARGHOUEI A， BRECKON T P. GANomaly： semi-supervised anomaly detection via adversarial training ［C］// Proceedings of the 2018 Asian Conference on Computer Vision. Cham： Springer， 2019： 622-637. 10.1007/978-3-030-20893-6_39
27 YANG Z I， ZHANG T， BOZCHALOOI I S， et al. Memory-augmented generative adversarial networks for anomaly detection ［J］. IEEE Transactions on Neural Networks and Learning Systems， 2022， 33（6）： 2324-2334. 10.1109/tnnls.2021.3132928
28 ZHU Y， YANG Y， LI Y， et al. Representation learning with dual autoencoder for multi-label classification ［J］. IEEE Access， 2021， 9： 98939-98947. 10.1109/access.2021.3096194
29 DUA D， GRAFF C. UCI machine learning repository［EB/OL］. ［2021-04-22］.http：//archive.ics.uci.edu/ml.
30 YANG B， FU X， SIDIROPOULOS N D， et al. Towards K-means-friendly spaces： simultaneous deep learning and clustering［C］// Proceedings of the 34th International Conference on Machine Learning. New York： JMLR.org， 2017： 3861-3870.
31 ZHAI S F， CHENG Y， LU W N， et al. Deep structured energy based models for anomaly detection［C］// Proceedings of the 33rd International Conference on Machine Learning. New York： JMLR.org， 2016： 1100-1109.

[1]	Chunyong YIN, Liwen ZHOU. Unsupervised time series anomaly detection model based on re-encoding [J]. Journal of Computer Applications, 2023, 43(3): 804-811.
[2]	YUAN Lining, LIU Zhao. Graph representation learning by autoencoder with one-shot aggregation [J]. Journal of Computer Applications, 2023, 43(1): 8-14.
[3]	Yiyang GUO, Jiong YU, Xusheng DU, Shaozhi YANG, Ming CAO. Outlier detection algorithm based on autoencoder and ensemble learning [J]. Journal of Computer Applications, 2022, 42(7): 2078-2087.
[4]	Xiangzhou QI, Hongjie XING. Centered kernel alignment based multiple kernel one-class support vector machine [J]. Journal of Computer Applications, 2022, 42(2): 349-356.
[5]	Xin LI, Tao JIA. Deep fusion model for predicting differential gene expression by histone modification data [J]. Journal of Computer Applications, 2022, 42(11): 3404-3412.
[6]	LIN Zhixing, WANG Like. Network situation prediction method based on deep feature and Seq2Seq model [J]. Journal of Computer Applications, 2020, 40(8): 2241-2247.
[7]	LUO Shiqi, TIAN Shengwei, YU Long, YU Jiong, SUN Hua. Android malware detection based on texture fingerprint and malware activity vector space [J]. Journal of Computer Applications, 2018, 38(4): 1058-1063.
[8]	WANG Hong, GE Lina, WANG Suqing, WANG Liying, ZHANG Yipeng, LIANG Juncheng. Improvement of differential privacy protection algorithm based on OPTICS clustering [J]. Journal of Computer Applications, 2018, 38(1): 73-78.
[9]	HUANG Xiujie, CHEN Jing, ZHANG Yunchao. Optimized vector of locally aggregated descriptor algorithm in image retrieval based on minimized reconstruction error [J]. Journal of Computer Applications, 2016, 36(6): 1682-1687.
[10]	LIU Shangwang, GAO Liuyang. Target tracking based on improved sparse representation model [J]. Journal of Computer Applications, 2016, 36(11): 3152-3160.
[11]	Jian-hua YUAN. Adaptive regularization for super-resolution image reconstruction based on local structures [J]. Journal of Computer Applications, 2009, 29(11): 3008-3010.
[12]	. An anomaly intrusion detection algorithm based on semisupervised clustering [J]. Journal of Computer Applications, 2006, 26(7): 1640-1642.

Novelty detection method based on dual autoencoders and Transformer network

基于双自编码器和Transformer网络的异常检测方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 12

Recommended Articles

Metrics