基于视频深度学习的时空双流人物动作识别模型

doi:10.11772/j.issn.1001-9081.2017071740

计算机应用 ›› 2018, Vol. 38 ›› Issue (3): 895-899.DOI: 10.11772/j.issn.1001-9081.2017071740

• 应用前沿、交叉与综合 • 上一篇下一篇

基于视频深度学习的时空双流人物动作识别模型

杨天明¹, 陈志¹, 岳文静²

1. 南京邮电大学计算机学院, 南京 210023;
2. 南京邮电大学通信与信息工程学院, 南京 210003

收稿日期:2017-07-14 修回日期:2017-09-07 出版日期:2018-03-10 发布日期:2018-03-07
通讯作者: 陈志
作者简介:杨天明(1993-),男,江苏南通人,硕士研究生,主要研究方向:机器学习、视频数据挖掘;陈志(1978-),男,江苏淮安人,教授,硕士生导师,博士,CCF会员,主要研究方向:传感器网络、信息物理融合系统、机器学习、数据挖掘、Agent和多Agent系统;岳文静(1982-),女,山西应县人,副教授,博士,主要研究方向:认知无线电网络、数据挖掘。
基金资助:
国家自然科学基金资助项目（61501253）；江苏省自然科学基金资助项目（BK20151506）；江苏省"六大人才高峰"第十一批高层次人才选拔培养资助项目（XXRJ-009）；江苏省重点研发计划（社会发展）项目（BE2016778）；南京邮电大学科研项目（NY217054）。

Spatio-temporal two-stream human action recognition model based on video deep learning

YANG Tianming¹, CHEN Zhi¹, YUE Wenjing²

1. College of Computer, Nanjing University of Posts and Telecommunications, Nanjing Jiangsu 210023, China;
2. College of Communication and Information Technology, Nanjing University of Posts and Telecommunications, Nanjing Jiangsu 210003, China

Received:2017-07-14 Revised:2017-09-07 Online:2018-03-10 Published:2018-03-07
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61501253), the Basic Research Program of Jiangsu Province (Natural Science Foundation) (BK20151506), the 11th Six Talent Peaks Program of Jiangsu Province (XXRJ-009), the Key Research and Development Program (Social Development) of Jiangsu Province (BE2016778), the Scientific Research Foundation of Nanjing University of Posts and Telecommunications (NY217054).

摘要/Abstract

摘要： 深度学习在人物动作识别方面已取得较好的成效，但当前仍然需要充分利用视频中人物的外形信息和运动信息。为利用视频中的空间信息和时间信息来识别人物行为动作，提出一种时空双流视频人物动作识别模型。该模型首先利用两个卷积神经网络分别抽取视频动作片段空间和时间特征，接着融合这两个卷积神经网络并提取中层时空特征，最后将提取的中层特征输入到3D卷积神经网络来完成视频中人物动作的识别。在数据集UCF101和HMDB51上，进行视频人物动作识别实验。实验结果表明，所提出的基于时空双流的3D卷积神经网络模型能够有效地识别视频人物动作。

关键词: 人物动作识别, 时空模型, 深度学习, 卷积神经网络, 视频挖掘

Abstract: Deep learning has achieved good results in human action recognition, but it still needs to make full use of video human appearance information and motion information. To recognize human actions by using spatial information and temporal information in video, a video human action recognition model based on spatio-temporal two-stream was proposed. Two convolutional neural networks were used to extract spatial and temporal features of video sequences respectively in the proposed model, and then the two neural networks were merged to extract the middle spatio-temporal features, finally the video human action recognition was completed by inputting the extracted features into a 3D convolutional neural network. The video human action recognition experiments were carried out on the data set UCF101 and HMDB51. Experimental results show that the proposed 3D convolutional neural network model based on the spatio-temporal two-stream can effectively recognize the video human actions.

Key words: human action recognition, spatio-temporal model, deep learning, Convolution Neural Network (CNN), video mining

中图分类号:

TP391

杨天明, 陈志, 岳文静. 基于视频深度学习的时空双流人物动作识别模型[J]. 计算机应用, 2018, 38(3): 895-899.

YANG Tianming, CHEN Zhi, YUE Wenjing. Spatio-temporal two-stream human action recognition model based on video deep learning[J]. Journal of Computer Applications, 2018, 38(3): 895-899.

参考文献

[1] 唐宋, 陈利娟, 陈志贤, 等. 基于目标域局部近邻几何信息的域自适应图像分类方法[J]. 计算机应用, 2017, 37(4):1164-1168.(TANG S, CHEN L J, CHEN Z X, et al. Domain adaptation image classification based on target local-neighbor geometrical information[J]. Journal of Computer Applications, 2017, 37(4):1164-1168.)
[2] XIONG H, YU W, YANG X, et al. Learning the conformal transformation kernel for image recognition[J]. IEEE Transactions on Neural Networks and Learning Systems, 2017, 28(1):149-163.
[3] SZEGEDY C, LIU W, JIA Y, et al. Going deeper with convolutions[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:1-9.
[4] SCHROFF F, KALENICHENKO D, PHILBIN J. FaceNet:a unified embedding for face recognition and clustering[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:815-823.
[5] TOMPSON J, GOROSHIN R, JAIN A, et al. Efficient object localization using convolutional networks[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:648-656.
[6] ZHANG J, HAN Y, TANG J, et al. Semi-supervised image-to-video adaptation for video action recognition[J]. IEEE Transactions on Cybernetics, 2016, 47(4):960-973.
[7] LIU L, SHAO L, LI X, et al. Learning spatio-temporal representations for action recognition:a genetic programming approach[J]. IEEE Transactions on Cybernetics, 2016, 46(1):158-170.
[8] HUSAIN F, DELLEN B, TORRAS C. Action recognition based on efficient deep feature learning in the spatio-temporal domain[J]. IEEE Robotics and Automation Letters, 2016, 1(2):984-991.
[9] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[EB/OL].[2017-05-06]. http://www.datascienceassn.org/sites/default/files/Two-Stream%20Convolutional%20Networks%20for%20Action%20Recognition%20in%20Videos.pdf.
[10] JI S, YANG M, YU K, et al. 3D convolutional neural networks for human action recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2013, 35(1):221-231.
[11] KARPATHY A, TODERICI G, SHETTY S, et al. Large-scale video classification with convolutional neural networks[C]//CVPR'14:Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2014:1725-1732.
[12] TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]//ICCV'15:Proceedings of the 2015 IEEE International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2015:4489-4497.
[13] DONAHUE J, JIA Y, VINYALS O, et al. DeCAF:a deep convolutional activetion feature for generic visual recognition[EB/OL].[2017-05-09]. https://people.eecs.berkeley.edu/~nzhang/papers/icml14_decaf.pdf.
[14] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[EB/OL].[2017-05-07]. http://xanadu.cs.sjsu.edu/~drtylin/classes/cs267_old/ImageNet%20DNN%20NIPS2012(2).pdf.
[15] KUEHNE H, JHUANG H, GARROTE E, et al. HMDB:a large video database for human motion recognition[C]//ICCV'11:Proceedings of the 2011 International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2011:2556-2563.
[16] DONAHUE J, HENDRICKS L A, GUADARRAMA S, et al. Long-term recurrent convolutional networks for visual recognition and description[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2015:2625-2634.
[17] SUN L, JIA K, YEUNG D Y, et al. Human action recognition using factorized spatio-temporal convolutional networks[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2015:4597-4605.
[18] NAHA S, WANG Y. Beyond verbs:understanding actions in videos with text[C]//Proceedings of the 201623rd International Conference on Pattern Recognition. Piscataway, NJ:IEEE, 2016:1833-1838.
[19] HU R, XU H, ROHRBACH M, et al. Natural language object retrieval[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2016:4555-4564.

基于视频深度学习的时空双流人物动作识别模型

Spatio-temporal two-stream human action recognition model based on video deep learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	陈成瑞, 孙宁, 何世彪, 廖勇. 面向C-V2X通信的基于深度学习的联合信道估计与均衡算法[J]. 计算机应用, 2021, 41(9): 2687-2693.
[2]	谢德峰, 吉建民. 融入句法感知表示进行句法增强的语义解析[J]. 计算机应用, 2021, 41(9): 2489-2495.
[3]	代雨柔, 杨庆, 张凤荔, 周帆. 基于自监督学习的社交网络用户轨迹预测模型[J]. 计算机应用, 2021, 41(9): 2545-2551.
[4]	宋中山, 梁家锐, 郑禄, 刘振宇, 帖军. 基于双向门控尺度特征融合的遥感场景分类[J]. 计算机应用, 2021, 41(9): 2726-2735.
[5]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[6]	张永斌, 常文欣, 孙连山, 张航. 基于字典的域名生成算法生成域名的检测方法[J]. 计算机应用, 2021, 41(9): 2609-2614.
[7]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[8]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[9]	牟长宁, 王海鹏, 周丕宇, 侯鑫行. 基于图卷积神经网络的串联质谱从头测序[J]. 计算机应用, 2021, 41(9): 2773-2779.
[10]	王贺兵, 张春梅. 基于非对称卷积-压缩激发-次代残差网络的人脸关键点检测[J]. 计算机应用, 2021, 41(9): 2741-2747.
[11]	郑志强, 胡鑫, 翁智, 王雨禾, 程曦. 基于改进DenseNet的牛眼图像特征提取方法[J]. 计算机应用, 2021, 41(9): 2780-2784.
[12]	何正海, 线岩团, 王蒙, 余正涛. 融合句法指导与字符注意力机制的案情阅读理解方法[J]. 计算机应用, 2021, 41(8): 2427-2431.
[13]	曾祥银, 郑伯川, 刘丹. 基于深度卷积神经网络和聚类的左右轨道线检测[J]. 计算机应用, 2021, 41(8): 2324-2329.
[14]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 计算机应用, 2021, 41(8): 2273-2287.
[15]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.