基于RGB和关节点数据融合模型的双人交互行为识别

doi:10.11772/j.issn.1001-9081.2019040633

计算机应用 ›› 2019, Vol. 39 ›› Issue (11): 3349-3354.DOI: 10.11772/j.issn.1001-9081.2019040633

• 虚拟现实与多媒体计算 • 上一篇下一篇

基于RGB和关节点数据融合模型的双人交互行为识别

姬晓飞, 秦琳琳, 王扬扬

沈阳航空航天大学自动化学院, 沈阳 110168

收稿日期:2019-04-15 修回日期:2019-07-26 出版日期:2019-11-10 发布日期:2019-08-26
通讯作者: 姬晓飞
作者简介:姬晓飞(1978-),女,辽宁鞍山人,副教授,博士,主要研究方向:视频分析与处理、模式识别;秦琳琳(1994-),女,山东菏泽人,硕士研究生,主要研究方向:视频分析与处理、生物特征与行为分析;王扬扬(1979-),女,辽宁沈阳人,工程师,博士,主要研究方向:视频分析与处理。
基金资助:
国家自然科学基金资助项目（61602321）；辽宁省教育厅科学研究服务地方项目（L201708）；辽宁省教育厅科学研究青年项目（L201745）。

Human interaction recognition based on RGB and skeleton data fusion model

JI Xiaofei, QIN Linlin, WANG Yangyang

College of Automation, Shenyang Aerospace University, Shenyang Liaoning 110136, China

Received:2019-04-15 Revised:2019-07-26 Online:2019-11-10 Published:2019-08-26
Supported by:
This work is partially supported by National Natural Science Foundation of China (61602321), the Local Project of Scientific Research Service of Liaoning Education Department (L201708), the Scientific Research Youth Project of Liaoning Education Department (L201745).

摘要/Abstract

摘要： 基于RGB视频序列的双人交互行为识别已经取得了重大进展，但因缺乏深度信息，对于复杂的交互动作识别不够准确。深度传感器（如微软Kinect）能够有效提高全身各关节点的跟踪精度，得到准确的人体运动及变化的三维关节点数据。依据RGB视频和关节点数据的各自特性，提出一种基于RGB和关节点数据双流信息融合的卷积神经网络（CNN）结构模型。首先，利用Vibe算法获得RGB视频在时间域的感兴趣区域，之后提取关键帧映射到RGB空间，以得到表示视频信息的时空图，并把图送入CNN提取特征；然后，在每帧关节点序列中构建矢量，以提取余弦距离（CD）和归一化幅值（NM）特征，将单帧中的余弦距离和关节点特征按照关节点序列的时间顺序连接，馈送入CNN学习更高级的时序特征；最后，将两种信息源的softmax识别概率矩阵进行融合，得到最终的识别结果。实验结果表明，将RGB视频信息和关节点信息结合可以有效地提高双人交互行为识别结果，在国际公开的SBU Kinect interaction数据库和NTU RGB+D数据库中分别达到92.55%和80.09%的识别率，证明了提出的模型对双人交互行为识别的有效性。

关键词: RGB视频, 关节点数据, 卷积神经网路, softmax, 融合, 双人交互行为识别

Abstract: In recent years, significant progress has been made in human interaction recognition based on RGB video sequences. Due to its lack of depth information, it cannot obtain accurate recognition results for complex interactions. The depth sensors (such as Microsoft Kinect) can effectively improve the tracking accuracy of the joint points of the whole body and obtain three-dimensional data that can accurately track the movement and changes of the human body. According to the respective characteristics of RGB and joint point data, a convolutional neural network structure model based on RGB and joint point data dual-stream information fusion was proposed. Firstly, the region of interest of the RGB video in the time domain was obtained by using the Vibe algorithm, and the key frames were extracted and mapped to the RGB space to obtain the spatial-temporal map representing the video information. The map was sent to the convolutional neural network to extract features. Then, a vector was constructed in each frame of the joint point sequence to extract the Cosine Distance (CD) and Normalized Magnitude (NM) features. The cosine distance and the characteristics of the joint nodes in each frame were connected in time order of the joint point sequence, and were fed into the convolutional neural network to learn more advanced temporal features. Finally, the softmax recognition probability matrixes of the two information sources were fused to obtain the final recognition result. The experimental results show that combining RGB video information with joint point information can effectively improve the recognition result of human interaction behavior, and achieves 92.55% and 80.09% recognition rate on the international public SBU Kinect interaction database and NTU RGB+D database respectively, verifying the effectiveness of the proposed model for the identification of interaction behaviour between two people.

Key words: RGB video, skeleton data, Convolutional Neural Network (CNN), softmax, fusion, human interaction recognition

中图分类号:

TP391

姬晓飞, 秦琳琳, 王扬扬. 基于RGB和关节点数据融合模型的双人交互行为识别[J]. 计算机应用, 2019, 39(11): 3349-3354.

JI Xiaofei, QIN Linlin, WANG Yangyang. Human interaction recognition based on RGB and skeleton data fusion model[J]. Journal of Computer Applications, 2019, 39(11): 3349-3354.

参考文献

[1] 王世刚,孙爱朦,赵文婷,等. 基于时空兴趣点的单人行为及交互行为识别[J]. 吉林大学学报(工学版), 2015, 45(1):304-308.(WANG S G, SUN A M, ZHAO W T, et al. Single and interactive human behavior recognition algorithm based on spatio-temporal interest point[J]. Journal of Jilin University (Engineering and Technology Edition), 2015, 45(1):304-308.)
[2] GAVRILA D M, DAVIS L S. 3-D model-based tracking of humans in action:a multi-view approach[C]//Proceedings of the 1996 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 1996:73-80.
[3] 赵海勇,刘志镜,张浩. 基于轮廓特征的人体行为识别[J]. 光电子·激光, 2010, 21(10):1547-1551. (ZHAO H Y, LIU Z J, ZHANG H. Human action recognition based on image contour[J]. Journal of Photoelectron·Laser, 2010, 21(10):1547-1551)
[4] 韩磊,李军峰,贾云得. 基于时空单词的双人交互行为识别方法[J].计算机学报, 2010, 33(4):776-784. (HAN L, LI J F, JIA Y D. Human interaction recognition method using spatio-temporal words[J]. Chinese Journal of Computers, 2010, 33(4):776-784.)
[5] LI N, CHENG X, GUO H, et al. Recognizing human interactions by genetic algorithm-based random forest spatio-temporal correlation[J]. Pattern Analysis and Applications, 2016, 19(1):267-282.
[6] YUN K, HONORIO J, CHATTOPADHYAY D, et al. Two-person interaction detection using body-pose features and multiple instance learning[C]//Proceedings of the 2012 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE, 2012:28-35.
[7] SLAMA R, WANNOUS H, DAOUDI M, et al. Accurate 3D action recognition using learning on the Grassmann manifold[J]. Pattern Recognition, 2015, 48(2):556-567.
[8] GHORBEL E, BOUTTEAU R, BOONAERT J, et al. 3D real-time human action recognition using a spline interpolation approach[C]//Proceedings of the 2015 International Conference on Image Processing Theory, Tools and Applications. Piscataway:IEEE, 2015:61-66.
[9] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2014:568-576.
[10] LI C, ZHONG Q, XIE D, et al. Skeleton-based action recognition with convolutional neural networks[C]//Proceedings of the 2017 IEEE International Conference on Multimedia & Expo Workshops. Piscataway:IEEE, 2017:597-600.
[11] LIU J, WANG G, DUAN L, et al. Skeleton-based human action recognition with global context-aware attention LSTM networks[J]. IEEE Transactions on Image Processing, 2018, 27(4):1586-1599.
[12] KE Q, BENNAMOUN M, AN S, et al. Learning clip representations for skeleton-based 3D action recognition[J]. IEEE Transactions on Image Processing, 2018, 27(6):2842-2855.
[13] LIU J, SHAHROUDY A, XU D, et al. Spatio-temporal LSTM with trust gates for 3D human act in recognition[C]//Proceedings of the 2016 European Conference on Computer Vision, LNCS 9907. Berlin:Springer, 2016:816-833.
[14] LI C, ZHONG Q, XIE D, et al. Co-occurrence feature learning from skeleton data for action recognition and detection with hierarchical aggregation[EB/OL].[2019-03-20].http://arxiv.org/pdf/1804.06055.
[15] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-01-10].https://arxiv.org/pdf/1409.1556.pdf.

基于RGB和关节点数据融合模型的双人交互行为识别

Human interaction recognition based on RGB and skeleton data fusion model

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	王伟, 赵尔平, 崔志远, 孙浩. 基于HowNet义原和Word2vec词向量表示的多特征融合消歧方法[J]. 计算机应用, 2021, 41(8): 2193-2198.
[2]	周险兵, 樊小超, 任鸽, 杨勇. 基于多层次语义特征的英文作文自动评分方法[J]. 计算机应用, 2021, 41(8): 2205-2211.
[3]	张林发, 张榆锋, 王琨, 李支尧. 基于直觉模糊集和亮度增强的医学图像融合[J]. 计算机应用, 2021, 41(7): 2082-2091.
[4]	李朝, 兰海, 魏宪. 基于注意力的毫米波-激光雷达融合目标检测[J]. 计算机应用, 2021, 41(7): 2137-2144.
[5]	吴丽丹, 薛雨阳, 童同, 杜民, 高钦泉. 基于前景语义信息的图像着色算法[J]. 计算机应用, 2021, 41(7): 2048-2053.
[6]	杜炎, 吕良福, 焦一辰. 基于模糊推理的模糊原型网络[J]. 计算机应用, 2021, 41(7): 1885-1890.
[7]	牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲. 基于深度学习的双通道夜视图像复原方法[J]. 计算机应用, 2021, 41(6): 1775-1784.
[8]	章荪, 尹春勇. 基于多任务学习的时序多模态情感分析模型[J]. 计算机应用, 2021, 41(6): 1631-1639.
[9]	栾佳宁, 张伟, 孙伟, 张奥, 韩冬. 基于二维码视觉与激光雷达融合的高精度定位算法[J]. 计算机应用, 2021, 41(5): 1484-1491.
[10]	赖雪梅, 唐宏, 陈虹羽, 李珊珊. 基于注意力机制的特征融合-双向门控循环单元多模态情感分析[J]. 计算机应用, 2021, 41(5): 1268-1274.
[11]	陆鑫伟, 余鹏飞, 李海燕, 李红松, 丁文谦. 基于注意力自身线性融合的弱监督细粒度图像分类算法[J]. 计算机应用, 2021, 41(5): 1319-1325.
[12]	曹建芳, 田晓东, 贾一鸣, 闫敏敏. 改进DeepLabV3+模型在壁画分割中的应用[J]. 计算机应用, 2021, 41(5): 1471-1476.
[13]	傅卓鑫, 孙昊, 陈建文, 郭悦, 陈金. 基于信息物理融合的骨折复位机器人系统构建[J]. 计算机应用, 2021, 41(5): 1533-1538.
[14]	卞鹏程, 郑忠龙, 李明禄, 何依然, 王天翔, 张大伟, 陈丽媛. 基于注意力融合网络的视频超分辨率重建[J]. 计算机应用, 2021, 41(4): 1012-1019.
[15]	杨丰瑞, 霍娜, 张许红, 韦巍. 基于注意力机制的主题扩展情感对话生成[J]. 计算机应用, 2021, 41(4): 1078-1083.