Kinect-based human pose estimation optimization and animation generation

doi:10.11772/j.issn.1001-9081.2021061043

Abstract

Abstract:

In order to generate more accurate and smooth virtual human animation， the Kinect device was used to capture the 3D human body pose data， and the monocular 3D human body pose estimation algorithm was used to reason the skeleton points in the color information of the Kinect at the same time， thereby optimizing the human pose estimation effect at real time， and driving the virtual character model to generate animation. Firstly， a spatio-temporal optimization method of skeleton point data processing was proposed to improve the stability of monocular estimation of the 3D human body pose. Secondly， a human pose estimation method based on the fusion of Kinect and Occlusion-Robust Pose-Maps （ORPM） algorithm was proposed to solve the occlusion problem of Kinect. Finally， a virtual human animation system based on quaternion vector interpolation and inverse kinematics constraints was developed， which was able to perform motion simulation and real-time animation generation. Compared with the animation generation method that only uses Kinect to capture human motion， the proposed method has the human body estimation data more robust， and has a certain anti-occlusion ability. The animation frame rate generated by this method is two times higher compared to that of the ORPM-based animation generation method， so that the effect of the animation generated by the proposed method is more realistic and smooth.

Key words: human pose estimation, Kinect, virtual human animation, simulation, anti-occlusion

摘要：

为了生成更准确流畅的虚拟人动画，采用Kinect设备捕获三维人体姿态数据的同时，使用单目人体三维姿态估计算法对Kinect的彩色信息进行骨骼点数据推理，从而实时优化人体姿态估计效果，并驱动虚拟人物模型生成动画。首先，提出了一种时空优化的骨骼点数据处理方法，以提高单目估计人体三维姿态的稳定性；其次，提出了一种Kinect和遮挡鲁棒姿势图（ORPM）算法融合的人体姿态估计方法来解决Kinect的遮挡问题；最后，研制了基于四元数向量插值和逆向运动学约束的虚拟人动画系统，其能够进行运动仿真和实时动画生成。与仅利用Kinect捕获人体运动来生成动画的方法相比，所提方法的人体姿态估计数据鲁棒性更强，具备一定的防遮挡能力，而与基于ORPM算法的动画生成方法相比，所提方法生成的动画在帧率上提高了两倍，效果更真实流畅。

关键词: 人体姿态估计, Kinect, 虚拟人动画, 仿真, 防遮挡

CLC Number:

TP391.4

Wei ZHAO, Yi LI. Kinect-based human pose estimation optimization and animation generation[J]. Journal of Computer Applications, 2022, 42(9): 2830-2837.

赵威, 李毅. 基于Kinect的人体姿态估计优化和动画生成[J]. 《计算机应用》唯一官方网站, 2022, 42(9): 2830-2837.

Figures/Tables 12

Fig. 1 Human skeleton tree coding and sequence

Fig. 2 Sampling sequences of different algorithms

Fig. 3 Motion sequence processing

Tab. 1 Least squares distance between frames

序列/帧	最小二乘距离/cm	约束后的最小二乘距离/cm
2	0.587 3	0.587 3
3	3.721 0	0.7112
4	1.290 0	0.6739
5	4.146 6	1.8537
6	5.340 2	1.9203

Fig. 4 Coordinate changes of joint points in motion sequence

Fig. 5 Experimental scene

Fig. 6 Generated animation effects

Fig. 7 Changes in angles of right shoulder and right elbow during waving motion

Fig. 8 Changes in angles of right elbow and right wrist in sideways punching motion

Fig. 9 Flowchart of virtual human animation system

Fig. 10 Part of performer’s actions and corresponding animation effects

Tab. 2 Comparison of different animation generation methods

方法	动画帧率/FPS	是否插值	适应场景
本文方法	60	是	正常光、弱光
文献［14］方法	20	否	正常光
欧拉角方法	20	否

References 15

1	MEHTA D， SOTNYCHENKO O， MUELLER F， et al. Single-shot multi-person 3D pose estimation from monocular RGB［C］// Proceedings of the 2018 International Conference on 3D Vision. Piscataway： IEEE， 2018： 120-130. 10.1109/3dv.2018.00024
2	SHOTTON J， GIRSHICK R， FITZGIBBON A， et al. Efficient human pose estimation from single depth images［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2013， 35（12）： 2821-2840. 10.1109/tpami.2012.241
3	CAO Z， SIMON T， WEI S E， et al. Realtime multi-person 2D pose estimation using part affinity fields［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1302-1310. 10.1109/cvpr.2017.143
4	NEWELL A， HUANG Z A， DENG J. Associative embedding： end-to-end learning for joint detection and grouping［C］// Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2017：2274-2284.
5	LI S J， CHAN A B. 3D human pose estimation from monocular images with deep convolutional neural network［C］// Proceedings of the 2014 Asian Conference on Computer Vision， LNCS 9004. Cham： Springer， 2015： 332-347.
6	MEHTA D， SRIDHAR S， SOTNYCHENKO O， et al. VNect： real-time 3D human pose estimation with a single RGB camera［J］. ACM Transactions on Graphics， 2017， 36（4）： No.44. 10.1145/3072959.3073596
7	YEUNG L F， CHENG K C， FONG C H， et al. Evaluation of the Microsoft Kinect as a clinical assessment tool of body sway［J］. Gait and Posture， 2014， 40（4）：532-538. 10.1016/j.gaitpost.2014.06.012
8	MOON S， PARK Y， KO D W， et al. Multiple Kinect sensor fusion for human skeleton tracking using Kalman filtering［J］. International Journal of Advanced Robotic Systems， 2016， 13（2）： No.62415. 10.5772/62415
9	李琪，王向东，李华. 基于双Kinect传感器的三维人体姿态跟踪方法［J］. 系统仿真学报， 2020， 32（8）：1446-1454.
	LI Q， WANG X D， LI H. 3D human pose tracking approach based on double Kinect sensors［J］. Journal of System Simulation， 2020， 32（8）：1446-1454.
10	IONESCU C， PAPAVA D， OLARU V， et al. Human3.6M： large scale datasets and predictive methods for 3D human sensing in natural environments［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2014， 36（7）：1325-1339. 10.1109/tpami.2013.248
11	ZHANG Z Y. A flexible new technique for camera calibration［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2000， 22（11）： 1330-1334. 10.1109/34.888718
12	黄漫国，樊尚春，郑德智，等. 多传感器数据融合技术研究进展［J］. 传感器与微系统， 2010， 29（3）： 5-8， 12.
	HUANG M G， FAN S C， ZHENG D Z， et al. Research progress of multi-sensor data fusion technology［J］. Transducer and Microsystem Technologies， 2010， 29（3）： 5-8， 12.
13	ARUN K S， HUANG T S， BLOSTEIN S D. Least-squares fitting of two 3-D point sets［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 1987， PAMI-9（5）： 698-700. 10.1109/tpami.1987.4767965
14	刘洁，李毅，朱江平. 基于双相机捕获面部表情及人体姿态生成三维虚拟人动画［J］. 计算机应用， 2021， 41（3）：839-844.
	LIU J， LI Y， ZHU J P. 3D virtual human animation generation based on dual-camera capture of facial expression and human pose［J］. Journal of Computer Applications， 2021， 41（3）： 839-844.
15	LOPER M， MAHMOOD N， ROMERO J， et al. SMPL： a skinned multi-person linear model［J］. ACM Transactions on Graphics， 2015， 34（6）： No.248. 10.1145/2816795.2818013

[1]	Kun LI, Qing HOU. Lightweight human pose estimation based on attention mechanism [J]. Journal of Computer Applications, 2022, 42(8): 2407-2414.
[2]	Feiyu YANG, Zhan SONG, Zhenzhong XIAO, Yaoyang MO, Yu CHEN, Zhe PAN, Min ZHANG, Yao ZHANG, Beibei QIAN, Chaowei TANG, Wu JIN. Rethinking errors in human pose estimation heatmap [J]. Journal of Computer Applications, 2022, 42(8): 2548-2555.
[3]	Qixiang SUN, Ning HE, Jingzun ZHANG, Chen HONG. Lightweight human pose estimation method based on non-local high-resolution network [J]. Journal of Computer Applications, 2022, 42(5): 1398-1406.
[4]	Jianmin ZHAO, Cheng ZHAO, Haiguang XIA. Cattle body size measurement method based on Kinect v4 [J]. Journal of Computer Applications, 2022, 42(5): 1598-1606.
[5]	LI Jin, WANG Feng, YANG Shenyu. Freight routing optimization model and algorithm of battery-swapping electric vehicle [J]. Journal of Computer Applications, 2021, 41(6): 1792-1798.
[6]	LI Yan, CHEN Qiaoping. Internet rumor propagation model considering non-supportive comments [J]. Journal of Computer Applications, 2021, 41(4): 1128-1135.
[7]	LIU Jie, LI Yi, ZHU Jiangping. 3D virtual human animation generation based on dual-camera capture of facial expression and human pose [J]. Journal of Computer Applications, 2021, 41(3): 839-844.
[8]	WANG Jiangkun, HE Kunjin, CAO Hongfei, WANG Jinqiang, ZHANG Yan. Interactive water flow heating simulation based on smoothed particle hydrodynamics method [J]. Journal of Computer Applications, 2020, 40(5): 1409-1414.
[9]	Yizhuo ZHANG, Sen GE, Liangjun WANG, Jiang XIE, Jie CAO, Wu ZHANG. Parallel computing of bifurcation stenosis flows of carotid artery based on lattice Boltzmann method and large eddy simulation model [J]. Journal of Computer Applications, 2020, 40(2): 404-409.
[10]	Tian LI, Shumei ZHANG, Junli ZHAO. Design and implementation of intelligent flow field pathfinding algorithm for real-time strategy game [J]. Journal of Computer Applications, 2020, 40(2): 602-607.
[11]	SUN Tianqi, HU Jianpeng, HUANG Juan, FAN Ying. Bandwidth resource prediction and management of Web applications hosted on cloud [J]. Journal of Computer Applications, 2020, 40(1): 181-187.
[12]	HUANG Dongjin, DUAN Siwen, LEI Xue, LIANG Jingkun. Group formation control method based on Voronoi diagram [J]. Journal of Computer Applications, 2019, 39(6): 1799-1803.
[13]	ZHAO Shicao, XIAO Yonghao, DUAN Bowen, LI Yufeng. HSWAP: numerical simulation workflow management platform suitable for high performance computing environment [J]. Journal of Computer Applications, 2019, 39(6): 1569-1576.
[14]	ZHANG Cheng, CHEN Fulong, LIU Chao, QI Xuemei. XML-based component modeling and stimulation of cyber physical system [J]. Journal of Computer Applications, 2019, 39(6): 1842-1848.
[15]	YU Lu, HU Jianfeng, YAO Leiyue. Detection method of non-standard deep squat posture based on human skeleton [J]. Journal of Computer Applications, 2019, 39(5): 1448-1452.