Hidden state initialization method for recurrent neural network-based human motion model

doi:10.11772/j.issn.1001-9081.2022020175

Journal of Computer Applications ›› 2023, Vol. 43 ›› Issue (3): 723-727.DOI: 10.11772/j.issn.1001-9081.2022020175

• Artificial intelligence • Previous Articles

Hidden state initialization method for recurrent neural network-based human motion model

Nanfan LI¹, Wenwen SI¹, Siyuan DU¹, Zhiyong WANG²^,³, Chongyang ZHONG²^,³(), Shihong XIA²^,³

^1.State Grid Beijing Urban Power Supply Company，Beijing 100034，China
^2.Institute of Computing Technology，Chinese Academy of Sciences，Beijing 100190，China
^3.University of Chinese Academy of Sciences，Beijing 100049，China

Received:2022-02-18 Revised:2022-05-12 Accepted:2022-05-13 Online:2022-08-16 Published:2023-03-10
Contact: Chongyang ZHONG
About author:LI Nanfan， born in 1993， M. S.， engineer. His research interests include human motion simulation.
SI Wenwen， born in 1984， M. S.， senior engineer. Her research interests include human motion simulation.
DU Siyuan， born in 1994， assistant engineer. His research interests include human motion simulation.
WANG Zhiyong， born in 1990， Ph. D. His research interests include computer graphics.
XIA Shihong， born in 1974， Ph. D.， research fellow. His research interests include computer graphics， virtual reality， artificial intelligence.
Supported by:
Special Project of Science and Technology for Winter Olympics of National Key Research and Development Program of China(2020YFF0304701);Science and Technology Project of Beijing Electric Power Company(52020220004B)

基于循环神经网络的人体运动模型的隐状态初始化方法

李南帆¹, 司文文¹, 杜思远¹, 王志勇²^,³, 钟重阳²^,³(), 夏时洪²^,³

^1.国网北京城区供电公司，北京 100034
^2.中国科学院计算技术研究所，北京 100190
^3.中国科学院大学，北京 100049

通讯作者: 钟重阳
作者简介:李南帆（1993—），男，北京人，工程师，硕士，主要研究方向：人体运动仿真
司文文（1984—），女，山东济南人，高级工程师，硕士，主要研究方向：人体运动仿真
杜思远（1994—），男，北京人，助理工程师，主要研究方向：人体运动仿真
王志勇（1990—），男，天津人，博士，主要研究方向：计算机图形学
钟重阳（1994—），男，重庆人，博士，主要研究方向：计算机图形学
夏时洪（1974—），男，四川平昌人，研究员，博士，主要研究方向：计算机图形学、虚拟现实、人工智能。
基金资助:
国家重点研发计划科技冬奥重点专项(2020YFF0304701);北京市电力公司科技项目(52020220004B)

Abstract

Abstract:

Aiming at the problem of the jump existed in the first frame of human motion synthesis method based on Recurrent Neural Network （RNN）， which affects the quality of generated motion， a human motion synthesis method with hidden state initialization was proposed. The initial hidden state was used as independent variable， the objective function of the neural network was used as optimization goal， and the gradient descent method was used to optimize and solve the problem to obtain a suitable initial hidden state. Compared with Encoder-Recurrent-Decoder （ERD） model and Residual Gate Recurrent Unit （RGRU） model， the proposed method with initial hidden state estimation reduces the prediction error of the first frame by 63.51% and 6.90% respectively， and decreases the total error of 10 frames by 50.00% and 4.89% respectively. Experimental results show that the proposed method is better than the method without initial hidden state estimation in both motion synthesis quality and motion prediction accuracy. And the proposed method accurately estimates the hidden state of the first frame of RNN-based human motion model， which improves the quality of motion synthesis and provides reliable data support for action recognition model in real-time security monitoring.

Key words: human motion synthesis, Recurrent Neural Network (RNN), hidden state estimation, action recognition, motion model

摘要：

针对基于循环神经网络（RNN）的人体运动合成方法存在首帧跳变，进而影响生成运动的质量的问题，提出一种带有隐状态初始化的人体运动合成方法，将初始隐状态作为自变量，利用神经网络的目标函数作为优化目标，并使用梯度下降的方法进行优化求解，以得到一个合适的初始隐状态。相较于编码器-循环-解码器（ERD）、残差门控循环单元（RGRU）模型，所提方法在首帧的预测误差分别减小63.51%和6.90%，10帧的总误差分别减小50.00%和4.89%。实验结果表明，该方法无论是运动合成质量还是运动预测精度都优于不进行初始隐状态估计的方法；它通过准确估计基于RNN的人体运动模型的首帧隐状态可提升运动合成的质量，并且为实时安全监测中的动作识别模型提供可靠的数据支持。

关键词: 人体运动合成, 循环神经网络, 隐状态估计, 动作识别, 运动模型

CLC Number:

TP181

Nanfan LI, Wenwen SI, Siyuan DU, Zhiyong WANG, Chongyang ZHONG, Shihong XIA. Hidden state initialization method for recurrent neural network-based human motion model[J]. Journal of Computer Applications, 2023, 43(3): 723-727.

李南帆, 司文文, 杜思远, 王志勇, 钟重阳, 夏时洪. 基于循环神经网络的人体运动模型的隐状态初始化方法[J]. 《计算机应用》唯一官方网站, 2023, 43(3): 723-727.

Figures/Tables 8

Fig. 1 Discontinuities in motion synthesis

Fig. 2 Structure of motion model

Fig. 3 Hidden state estimation model

Fig. 4 Comparison of motion synthesis results

Fig. 5 Comparison of ankle joint velocity and hidden state curves

Tab. 1 Comparison of joint angle prediction errors for walking motion

帧序号	ERD	RGRU	本文方法
1	0.74	0.29	0.27
2	0.93	0.34	0.33
4	1.18	0.60	0.52
8	1.59	0.95	0.92
10	1.78	1.09	1.07

Fig. 6 Comparison of motion prediction results

Fig. 7 Motion interpolation results

References 17

1	WEN Y H， GAO L， FU H B， et al. Graph CNNs with motif and variable temporal block for skeleton-based action recognition［C］// Proceedings of the 33rd AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2019：8989-8996. 10.1609/aaai.v33i01.33018989
2	苏乐，柴金祥，夏时洪. 基于局部姿态先验的深度图像3D人体运动捕获方法［J］. 软件学报， 2016， 27（S2）： 172-183.
	SU L， CHAI J X， XIA S H. Local pose prior based 3D human motion capture from depth camera［J］. Journal of Software， 2016， 27（S2）： 172-183.
3	XIA S H， GAO L， LAI Y K， et al. A survey on human performance capture and animation［J］. Journal of Computer Science and Technology， 2017， 32（3）：536-554. 10.1007/s11390-017-1742-y
4	FRAGKIADAKI K， LEVINE S， FELSEN P， et al. Recurrent network models for human dynamics［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 4346-4354. 10.1109/iccv.2015.494
5	JAIN A， ZAMIR A R， SAVARESE S， et al. Structural-RNN： deep learning on spatio-temporal graphs［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 5308-5317. 10.1109/cvpr.2016.573
6	MARTINEZ J， BLACK M J， ROMERO J. On human motion prediction using recurrent neural networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 4674-4683. 10.1109/cvpr.2017.497
7	BARSOUM E， KENDER J， LIU Z C. HP-GAN： probabilistic 3D human motion prediction via GAN［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2018： 1499-1508. 10.1109/cvprw.2018.00191
8	ZHOU Y， LI Z M， XIAO S J， et al. Auto-conditioned recurrent networks for extended complex human motion synthesis［EB/OL］. ［2022-01-23］..
9	HOLDEN D， SAITO J， KOMURA T. A deep learning framework for character motion synthesis and editing［J］. ACM Transactions on Graphics， 2016， 35（4）： No.138. 10.1145/2897824.2925975
10	HOLDEN D， KOMURA T， SAITO J. Phase-functioned neural networks for character control［J］. ACM Transactions on Graphics， 2017， 36（4）： No.42. 10.1145/3072959.3073663
11	ZHANG H， STARKE S， KOMURA T， et al. Mode-adaptive neural networks for quadruped motion control［J］. ACM Transactions on Graphics， 2018， 37（4）： No.145. 10.1145/3197517.3201366
12	HARVEY F G， ROY J， KANAA D， et al. Recurrent semi-supervised classification and constrained adversarial generation with motion capture data［J］. Image and Vision Computing， 2018， 78： 42-52. 10.1016/j.imavis.2018.07.001
13	LEE K， LEE S， LEE J. Interactive character animation by learning multi-objective control［J］. ACM Transactions on Graphics， 2018， 37（6）： No.180. 10.1145/3272127.3275071
14	WANG Z Y， CHAI J X， XIA S H. Combining recurrent neural networks and adversarial training for human motion synthesis and control［J］. IEEE Transactions on Visualization and Computer Graphics， 2021， 27（1）： 14-28. 10.1109/tvcg.2019.2938520
15	GRAVES A. Generating sequences with recurrent neural networks［EB/OL］. ［2022-01-23］..
16	WERBOS P J. Backpropagation through time： what it does and how to do it［J］. Proceedings of the IEEE， 1990， 78（10）： 1550-1560. 10.1109/5.58337
17	HABIBIE I， HOLDEN D， SCHWARZ J， et al. A recurrent variational autoencoder for human motion synthesis［C］// Proceedings of the 2017 British Machine Vision Conference. Durham： BMVA Press， 2017： No.119. 10.5244/c.31.119

[1]	Ranyan NI, Yi ZHANG. Action recognition method based on video spatio-temporal features [J]. Journal of Computer Applications, 2023, 43(2): 521-528.
[2]	Meng YU, Wentao HE, Xuchuan ZHOU, Mengtian CUI, Keqi WU, Wenjie ZHOU. Review of recommendation system [J]. Journal of Computer Applications, 2022, 42(6): 1898-1913.
[3]	Xueqin CHEN, Tao TAO, Zhongwang ZHANG, Yilei WANG. Long short-term session-based recommendation algorithm combining paired coding scheme and two-dimensional conventional neural network [J]. Journal of Computer Applications, 2022, 42(5): 1347-1354.
[4]	Tingxiu CHEN, Jianqin YIN. Audio visual joint action recognition based on key frame selection network [J]. Journal of Computer Applications, 2022, 42(3): 731-735.
[5]	Yongkang HUANG, Meiyu LIANG, Xiaoxiao WANG, Zheng CHEN, Xiaowen CAO. Multi-person classroom action recognition in classroom teaching videos based on deep spatiotemporal residual convolution neural network [J]. Journal of Computer Applications, 2022, 42(3): 736-742.
[6]	Jiana MENG, Pin LYU, Yuhai YU, Shichang SUN, Hongfei LIN. Aspect-level cross-domain sentiment analysis based on capsule network [J]. Journal of Computer Applications, 2022, 42(12): 3700-3707.
[7]	Jialiang HAN, Yudong HAN, Xuanzhe LIU, Yaoshuai ZHAO, Di FENG. Personalized recommendation service system based on cloud-client-convergence [J]. Journal of Computer Applications, 2022, 42(11): 3506-3512.
[8]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[9]	LIU Zichen, LI Xiaojuan, WEI Wei. Automatic patent price evaluation based on recurrent neural network [J]. Journal of Computer Applications, 2021, 41(9): 2532-2538.
[10]	LI Yangzhi, YUAN Jiazheng, LIU Hongzhe. Human skeleton-based action recognition algorithm based on spatiotemporal attention graph convolutional network model [J]. Journal of Computer Applications, 2021, 41(7): 1915-1921.
[11]	ZHAO Xiaohu, LI Xiao. Image captioning algorithm based on multi-feature extraction [J]. Journal of Computer Applications, 2021, 41(6): 1640-1646.
[12]	NI Shuiping, LI Huifang. Battery state-of-charge prediction method based on one-dimensional convolutional neural network combined with long short-term memory network [J]. Journal of Computer Applications, 2021, 41(5): 1514-1521.
[13]	GUO Tianxiao, HU Qingrui, LI Jianwei, SHEN Yanfei. Fitness action recognition method based on human skeleton feature encoding [J]. Journal of Computer Applications, 2021, 41(5): 1458-1464.
[14]	CHE Bingqian, ZHOU Dong. Tag recommendation method combining network structure information and text content [J]. Journal of Computer Applications, 2021, 41(4): 976-983.
[15]	JIANG Li, HUANG Shijian, YAN Wenjuan. Human action recognition method based on low-rank action information and multi-scale convolutional neural network [J]. Journal of Computer Applications, 2021, 41(3): 721-726.

Hidden state initialization method for recurrent neural network-based human motion model

基于循环神经网络的人体运动模型的隐状态初始化方法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 8

References 17

Related Articles 15

Recommended Articles

Metrics