基于外观和动作特征双预测模型的视频异常行为检测

doi:10.11772/j.issn.1001-9081.2020121906

计算机应用 ›› 2021, Vol. 41 ›› Issue (10): 2997-3003.DOI: 10.11772/j.issn.1001-9081.2020121906

所属专题：多媒体计算与计算机仿真

• 多媒体计算与计算机仿真 • 上一篇下一篇

基于外观和动作特征双预测模型的视频异常行为检测

李自强¹, 王正勇¹, 陈洪刚¹, 李林怡², 何小海¹

1. 四川大学电子信息学院, 成都 610065;
2. 中国民航局第二研究所, 成都 610041

收稿日期:2020-12-08 修回日期:2021-04-26 出版日期:2021-10-10 发布日期:2021-07-16
通讯作者: 陈洪刚
作者简介:李自强(1998-),男,江西九江人,硕士研究生,主要研究方向:计算机视觉、深度学习、异常行为检测;王正勇(1969-),女,四川达州人,副教授,博士,主要研究方向:图像处理、模式识别;陈洪刚(1991-),男,四川达州人,助理研究员,博士,主要研究方向:图像处理;李林怡(1996-),女,四川内江人,主要研究方向:智能运行控制;何小海(1964-),男,四川绵阳人,教授,博士,主要研究方向:图像处理、模式识别、图像通信。
基金资助:
国家自然科学基金资助项目（61871278）；四川省科技计划项目（2019YFH0034）。

Video abnormal behavior detection based on dual prediction model of appearance and motion features

LI Ziqiang¹, WANG Zhengyong¹, CHEN Honggang¹, LI Linyi², HE Xiaohai¹

1. College of Electronics and Information Engineering, Sichuan University, Chengdu Sichuan 610065, China;
2. The Second Research Institute of Civil Aviation Administration of China, Chengdu Sichuan 610041, China

Received:2020-12-08 Revised:2021-04-26 Online:2021-10-10 Published:2021-07-16
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61871278), the Sichuan Science and Technology Program (2019YFH0034).

摘要/Abstract

摘要： 为了在视频异常行为检测中更加充分地运用外观和动作信息，设计出了一种能同时捕捉外观和动作信息的孪生网络模型。该网络的两个分支采用相同的自编码器结构，其中的外观子网络以连续几帧RGB图作为输入来预测下一帧，而动作子网络则输入RGB帧差图来预测未来帧差图。此外，考虑到影响基于预测的方法的检测效果的原因之一，即正常样本的多样性以及自编码器网络强大的“生成”能力，即对部分异常样本也有很好的预测效果，因此在编码器与解码器之间加入一个学习并存储正常样本的“原型”特征的记忆增强模块，从而使异常样本能获得更大的预测误差。在Avenue、UCSD-ped2和ShanghaiTech三个公共的异常数据集上进行了广泛的实验。实验结果表明，相较于其他基于重建或预测的视频异常行为检测方法，所提方法取得了更优异的表现。具体来说，该方法在Avenue、UCSD-ped2和ShanghaiTech数据集上的平均曲线下面积（AUC）分别达到了88.2%、97.5%和73.0%。

关键词: 异常行为检测, 视频监控, 自编码器, 记忆增强, 孪生网络

Abstract: In order to make full use of appearance and motion information in video abnormal behavior detection, a Siamese network model that can capture appearance and motion information at the same time was proposed. The two branches of the network were composed of the same autoencoder structure. Several consecutive frames of RGB images were used as the input of the appearance sub-network to predict the next frame, while RGB frame difference image was used as the input of the motion sub-network to predict the future frame difference. In addition, considering one of the reasons that affected the detection effect of the prediction-based method, that is the diversity of normal samples, and the powerful "generation" ability of the autoencoder network, that is it has a good prediction effect on some abnormal samples. Therefore, a memory enhancement module that learns and stores the "prototype" features of normal samples was added between the encoder and the decoder, so that the abnormal samples were able to obtain greater prediction error. Extensive experiments were conducted on three public anomaly detection datasets Avenue, UCSD-ped2 and ShanghaiTech. Experimental results show that, compared with other video abnormal behavior detection methods based on reconstruction or prediction, the proposed method achieves better performance. Specifically, the average Area Under Curve (AUC) of the proposed method on Avenue, UCSD-ped2 and ShanghaiTech datasets reach 88.2%, 97.5% and 73.0% respectively.

Key words: abnormal behavior detection, video surveillance, autoencoder, memory enhancement, Siamese network

中图分类号:

TP391.41

李自强, 王正勇, 陈洪刚, 李林怡, 何小海. 基于外观和动作特征双预测模型的视频异常行为检测[J]. 计算机应用, 2021, 41(10): 2997-3003.

LI Ziqiang, WANG Zhengyong, CHEN Honggang, LI Linyi, HE Xiaohai. Video abnormal behavior detection based on dual prediction model of appearance and motion features[J]. Journal of Computer Applications, 2021, 41(10): 2997-3003.

参考文献

[1] PANG G, SHEN C, CAO L, et al. Deep learning for anomaly detection:a review[J]. ACM Computing Surveys, 2021, 54(2):No. 38.
[2] SULTANI W, CHEN C, SHAH M. Real-world anomaly detection in surveillance videos[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:6479-6488.
[3] WAN B, FANG Y, XIA X, et al. Weakly supervised video anomaly detection via center-guided discriminative learning[C]//Proceedings of the 2020 IEEE International Conference on Multimedia and Expo. Piscataway:IEEE, 2020:1-6.
[4] GONG D, LIU L, LE V, et al. Memorizing normality to detect anomaly:memory-augmented deep autoencoder for unsupervised anomaly detection[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE, 2019:1705-1714.
[5] ABATI D, PORRELLO A, CALDERARA S, et al. Latent space autoregression for novelty detection[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:481-490.
[6] 朱松豪, 赵云斌. 基于半监督生成式对抗网络的异常行为检测[J]. 南京邮电大学学报(自然科学版), 2020, 40(4):50-56. (ZHU S H, ZHAO Y B. Anomaly detection based on semisupervised generative adversarial networks[J]. Journal of Nanjing University of Posts and Telecommunications (Natural Science Edition), 2020, 40(4):50-56.)
[7] MORAIS R, LE V, TRAN T, et al. Learning regularity in skeleton trajectories for anomaly detection in videos[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:11988-11996.
[8] LU Y W, KUMAR K M, NABAVI S S, et al. Future frame prediction using convolutional VRNN for anomaly detection[C]//Proceedings of 16th IEEE International Conference on Advanced Video and Signal Based Surveillance. Piscataway:IEEE, 2019:1-8.
[9] LIU W, LUO W X, LIAN D Z, et al. Future frame prediction for anomaly detection-a new baseline[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:6536-6545.
[10] NGUYEN T N, MEUNIER J. Anomaly detection in video sequence with appearance-motion correspondence[C]//Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision. Piscataway:IEEE, 2019:1273-1283.
[11] WANG T, QIAO M N, DENG Y J, et al. Abnormal event detection based on analysis of movement information of video sequence[J]. Optik, 2018, 152:50-60.
[12] YUAN Y, FENG Y C, LU X Q. Statistical hypothesis detector for abnormal event detection in crowded scenes[J]. IEEE Transactions on Cybernetics, 2017, 47(11):3597-3608.
[13] LU C W, SHI J P, JIA J Y. Abnormal event detection at 150 FPS in Matlab[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2013:2720-2727.
[14] KIM J, GRAUMAN K. Observe locally, infer globally:a spacetime MRF for detecting abnormal activities with incremental updates[C]//Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2009:2921-2928.
[15] MAHADEVAN V, LI W X, BHALODIA V, et al. Anomaly detection in crowded scenes[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2010:1975-1981.
[16] 胡学敏, 易重辉, 陈钦, 等. 基于运动显著图的人群异常行为检测[J]. 计算机应用, 2018, 38(4):1164-1169.(HU X M, YI C H, CHNE Q, et al. Abnormal crowd behavior detection based motion saliency map[J]. Journal of Computer Applications, 2018, 38(4):1164-1169.)
[17] PARK H, NOH J, HAM B. Learning memory-guided normality for anomaly detection[C]//Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2020:14360-14369.
[18] LEI Z, DENG F, YANG X D. Spatial temporal balanced generative adversarial autoencoder for anomaly detection[C]//Proceedings of the 2019 International Conference on Image, Video and Signal Processing. New York:ACM, 2019:1-7.
[19] HASAN M, CHOI J, NEUMANN J, et al. Learning temporal regularity in video sequences[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:733-742.
[20] LUO W X, LIU W, GAO S H. Remembering history with convolutional LSTM for anomaly detection[C]//Proceedings of the 2017 IEEE International Conference on Multimedia and Expo. Piscataway:IEEE, 2017:439-444.
[21] FAN Y X, WEN G J, LI D R, et al. Video anomaly detection and localization via Gaussian mixture fully convolutional variational autoencoder[J]. Computer Vision and Image Understanding, 2020, 195:No. 102920.
[22] ILG E, MAYER N, SAIKIA T, et al. FlowNet 2.0:evolution of optical flow estimation with deep networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:1647-1655.
[23] WANG L M, XIONG Y J, WANG Z, et al. Temporal segment networks for action recognition in videos[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2019, 41(11):2740-2755.
[24] IONESCU R T, KHAN F S, GEORGESCU M I, et al. Objectcentric auto-encoders and dummy anomalies for abnormal event detection in video[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:7834-7843.
[25] LUO W X, LIU W, GAO S H. A revisit of sparse coding based anomaly detection in stacked RNN framework[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:341-349.
[26] PRAWIRO H, PENG J W, PAN T Y, et al. Abnormal event detection in surveillance videos using two-stream decoder[C]//Proceedings of the 2020 IEEE International Conference on Multimedia and Expo Workshops. Piscataway:IEEE, 2020:1-6.

基于外观和动作特征双预测模型的视频异常行为检测

Video abnormal behavior detection based on dual prediction model of appearance and motion features

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	姬张建, 任兴旺. 带旋转与尺度估计的全卷积孪生网络目标跟踪算法[J]. 计算机应用, 2021, 41(9): 2705-2711.
[2]	张元钧, 张曦煌. 基于图卷积与长短期记忆网络的动态网络表示学习模型[J]. 计算机应用, 2021, 41(7): 1857-1864.
[3]	孙鹤立, 孙玉柱, 张晓云. 基于生成对抗网络的事件描述生成[J]. 计算机应用, 2021, 41(5): 1256-1261.
[4]	樊玮, 王慧敏, 邢艳. 基于自编码器的多视图属性网络表示学习模型[J]. 计算机应用, 2021, 41(4): 1064-1070.
[5]	钟莎, 黄玉清. 基于孪生区域候选网络的无人机指定目标跟踪[J]. 计算机应用, 2021, 41(2): 523-529.
[6]	黄晓祥, 胡咏梅, 吴丹, 任力杰. 基于变分自编码器的异常颈动脉早期识别和预测[J]. 计算机应用, 2021, 41(10): 3082-3088.
[7]	屠红艳, 张挺, 夏鹏飞, 杜奕. 基于改进型变分自编码器的不确定性空间信息重建方法[J]. 计算机应用, 2021, 41(10): 2959-2963.
[8]	罗俊, 陈黎飞. 基于BERT的不完全数据情感分类[J]. 计算机应用, 2021, 41(1): 139-144.
[9]	熊昌镇, 李言. 动态的加权孪生网络跟踪算法[J]. 计算机应用, 2020, 40(8): 2214-2218.
[10]	霍纬纲, 王慧芳. 基于自编码器和隐马尔可夫模型的时间序列异常检测方法[J]. 计算机应用, 2020, 40(5): 1329-1334.
[11]	朱喆, 许少华. 降噪自编码器深度卷积过程神经网络及在时变信号分类中的应用[J]. 计算机应用, 2020, 40(3): 698-703.
[12]	傅洪亮, 雷沛之. 基于去噪自编码器和长短时记忆网络的语音测谎算法[J]. 计算机应用, 2020, 40(2): 589-594.
[13]	王艺皓, 丁洪伟, 李波, 保利勇, 张颖婕. 基于深度学习的蛋白质亚细胞定位预测[J]. 计算机应用, 2020, 40(11): 3393-3399.
[14]	杨辉华, 张天宇, 李灵巧, 潘细朋. 基于MobileNet的移动端城管案件目标识别算法[J]. 计算机应用, 2019, 39(8): 2475-2479.
[15]	杨宏宇, 李博超. 基于逆向习得推理的网络异常行为检测模型[J]. 计算机应用, 2019, 39(7): 1967-1972.