Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (5): 1450-1457.DOI: 10.11772/j.issn.1001-9081.2020081178

Special Issue: 多媒体计算与计算机仿真

• Virtual reality and multimedia computing • Previous Articles     Next Articles

Human behavior recognition algorithm based on skeletal temporal divergence feature

TIAN Zhiqiang1,2,3, DENG Chunhua1,2,3, ZHANG Junwen1,2,3   

  1. 1. School of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan Hubei 430065, China;
    2. Institute of Big Data Science and Engineering, Wuhan University of Science and Technology, Wuhan Hubei 430065, China;
    3. Hubei Province Key Laboratory of Intelligent Information Processing and Real-Time Industrial System(Wuhan University of Science and Technology), Wuhan Hubei 430065, China
  • Received:2020-08-06 Revised:2020-11-15 Online:2021-05-10 Published:2020-12-09
  • Supported by:
    This work is partially supported by the National Natural Science Foundation of China (61806150), the Program of Science and Technology Department of Hubei Province (2018CFB195), the Young Talent Program of Science and Technology Research Plan of Education Department of Hubei Province (Q20181104), the Open Foundation of Hubei Province Key Laboratory of Intelligent Information Processing and Real-Time Industrial System (znxx2018QN09),the National Defense Advanced Research Foundation of Wuhan University of Science and Technology (GF201814).

基于骨骼时序散度特征的人体行为识别算法

田志强1,2,3, 邓春华1,2,3, 张俊雯1,2,3   

  1. 1. 武汉科技大学 计算机科学与技术学院, 武汉 430065;
    2. 武汉科技大学 大数据科学与工程研究院, 武汉 430065;
    3. 智能信息处理与实时工业系统湖北省重点实验室(武汉科技大学), 武汉 430065
  • 通讯作者: 邓春华
  • 作者简介:田志强(1996-),男,湖北武汉人,硕士研究生,主要研究方向:计算机视觉、机器学习;邓春华(1984-),男,湖南郴州人,副教授,博士,主要研究方向:计算机视觉、机器学习;张俊雯(1997-),女,湖北荆门人,硕士研究生,主要研究方向:计算机视觉、机器学习。
  • 基金资助:
    国家自然科学基金资助项目(61806150);湖北省科技厅计划项目(2018CFB195);湖北省教育厅科学技术研究计划青年人才项目(Q20181104);智能信息处理与实时工业系统湖北省重点实验室开放基金资助项目(znxx2018QN09);武汉科技大学国防预研基金资助项目(GF201814)。

Abstract: Human behavior recognition is an important basic technology in the fields such as intelligent monitoring, human-computer interaction and robotics. Graph Convolutional Neural Network (GCN) achieve excellent performance in skeleton-based human behavior recognition. The following problems exist in the research of human behavior recognition using GCNs:1) the human skeleton points are represented by coordinates, which lacks detailed information about the movement of the skeleton points; 2) in some videos, the motion amplitude of the human skeleton is too small, so that the representation information of the key skeleton points is not obvious. Aiming at the above problems, firstly, a temporal divergence model of skeleton points was designed to describe the movement states of the skeleton points, which amplified the between-class variances of different human behaviors. In addition, the attention mechanism of temporal divergence features was designed to highlight the key skeleton points and further expand the between-class variances. Finally, a two-stream fusion model was constructed based on the complementarity between the spatial data characteristics of the original skeleton and the temporal divergence characteristics. The proposed algorithm achieved the accuracy of 82.9% and 83.7% under two partitioning strategies of authoritative human behavior dataset NTU-RGB+D respectively, which were 1.3 percentage points and 0.5 percentage points higher than those of Adaptive Graph Convolutional Network (AGCN) respectively. The improvement of the accuracy of the proposed algorithm on the dataset proves the effectiveness of this algorithm.

Key words: skeleton, behavior recognition, graph convolution, temporal divergence, attention

摘要: 人体行为识别是智能监控、人机交互、机器人等领域的一项重要的基础技术。图卷积神经网络(GCN)在基于骨骼的人体行为识别上取得了卓越的性能。不过GCN在人体行为识别研究中存在以下问题:1)人体骨架的骨骼点采用坐标表示,缺乏骨骼点的运动细节信息;2)在某些视频中,人体骨架的运动幅度太小导致关键骨骼点的表征信息不明显。针对上述问题,首先提出骨骼点的时序散度模型来描述骨骼点的运动状态,从而放大了不同人体行为的类间方差。并进一步提出了时序散度特征的注意力机制,以突显关键骨骼点,进一步扩大类间方差。最后根据原始骨架的空间数据特征和时序散度特征的互补性构建了双流融合模型。所提算法在权威的人体行为数据集NTU-RGB+D的两种划分策略下分别达到了82.9%和83.7%的准确率,相比自适应图卷积网络(AGCN)提高了1.3个百分点和0.5个百分点,准确率的提升证明了所提算法的有效性。

关键词: 骨骼, 行为识别, 图卷积, 时序散度, 注意力

CLC Number: