Abstract: In consideration of the problems of low utilization rate of action information and insufficient attention of temporal information in video human action recognition, a human action recognition model based on tightly coupled spatiotemporal two-stream convolutional neural network was proposed. Firstly, two 2D convolutional neural networks were used to separately extract the spatial and temporal features in the video. Then, the forget gate module in the Long Short-Term Memory (LSTM) network was used to establish the feature-level tightly coupled connections between different sampled segments to achieve the transfer of information flow. After that, the Bi-directional Long Short-Term Memory (Bi-LSTM) network was used to evaluate the importance of each sampled segment and assign adaptive weight to it. Finally, the spatiotemporal two-stream features were combined to complete the human action recognition. The accuracy rates of this model on the datasets UCF101 and HMDB51 selected for the experiment and verification were 94.2% and 70.1% respectively. Experimental results show that the proposed model can effectively improve the utilization rate of temporal information and the ability of overall action representation, thus significantly improving the accuracy of human action recognition.

Key words: human action recognition, spatiotemporal model, Convolutional Neural Network (CNN), forget gate, feature fusion

摘要： 针对视频人体动作识别中动作信息利用率不高、时间信息关注度不足等问题，提出了一种基于紧耦合时空双流卷积神经网络的人体动作识别模型。首先，采用两个2D卷积神经网络分别提取视频中的空间特征和时间特征；然后，利用长短期记忆（LSTM）网络中的遗忘门模块在各采样片段之间建立特征层次的紧耦合连接以实现信息流的传递；接着，利用双向长短期记忆（Bi-LSTM）网络评估各采样片段的重要性并为其分配自适应权重；最后，结合时空双流特征以完成人体动作识别。在数据集UCF101和HMDB51上进行实验验证，该模型在这两个数据集上的准确率分别为94.2%和70.1%。实验结果表明，所提出的紧耦合时空双流卷积网络模型能够有效提高时间信息利用率和动作整体表达能力，由此明显提升人体动作识别的准确度。

关键词: 人体动作识别, 时空模型, 卷积神经网络, 遗忘门, 特征融合