Fitness action recognition method based on human skeleton feature encoding

doi:10.11772/j.issn.1001-9081.2020071113

Abstract

Abstract: Fitness action recognition is the core of the intelligent fitness system. In order to improve the accuracy and speed of fitness action recognition algorithm, and reduce the influence of the global displacement of fitness actions on the recognition results, a fitness action recognition method based on human skeleton feature encoding was proposed which included three steps:firstly, the simplified human skeleton model was constructed, and the information of skeleton model's joint point coordinates was extracted through the human pose estimation technology; secondly, the action feature region was extracted by using the human central projection method in order to eliminate the influence of the global displacement on action recognition; finally, the feature region was encoded as the feature vector and input to a multi-classifier to realize the action recognition, at the same time the length of the feature vector was optimized for improving the recognition rate and speed. Experiment results showed that the proposed method achieved the recognition rate of 97.24% on the self-built fitness dataset with 28 types of fitness actions, which verified the effectiveness of this method to recognize different types of fitness actions; on the public KTH and Weizmann datasets, the recognition rates of the proposed method were 91.67% and 90% respectively, higher than those of other similar methods.

Key words: computer vision, action recognition, intelligent fitness, skeleton information, pose estimation

摘要： 健身动作识别是智能健身系统的核心环节。为了提高健身动作识别算法的精度和速度，并减少健身动作中人体整体位移对识别结果的影响，提出了一种基于人体骨架特征编码的健身动作识别方法。该方法包括三个步骤：首先，构建精简的人体骨架模型，并利用人体姿态估计技术提取骨架模型中各关节点的坐标信息；其次，利用人体中心投影法提取动作特征区域以消除人体整体位移对动作识别的影响；最后，将特征区域编码作为特征向量并输入多分类器进行动作识别，同时通过优化特征向量长度使识别率和速度达到最优。实验结果表明，本方法在包含28种动作的自建健身数据集上的动作识别率为97.24%，证明该方法能够有效识别各类健身动作；在公开的KTH和Weizmann数据集上，所提方法的动作识别率分别为91.67%和90%，优于其他同类型方法。

关键词: 计算机视觉, 动作识别, 智能健身, 骨架信息, 姿态估计

CLC Number:

TP391.41

GUO Tianxiao, HU Qingrui, LI Jianwei, SHEN Yanfei. Fitness action recognition method based on human skeleton feature encoding[J]. Journal of Computer Applications, 2021, 41(5): 1458-1464.

郭天晓, 胡庆锐, 李建伟, 沈燕飞. 基于人体骨架特征编码的健身动作识别方法[J]. 计算机应用, 2021, 41(5): 1458-1464.

References

[1] TING H Y,SIM K S,ABAS F S. Automatic badminton action recognition using RGB-D sensor[J]. Advanced Materials Research,2014,1042:89-93.
[2] KUO Y M,LEE J S,CHUNG P C. A visual context-awarenessbased sleeping-respiration measurement system[J]. IEEE Transactions on Information Technology in Biomedicine,2010,14(2):255-265.
[3] VANDO S,HADDAD M,MASALA D,et al. Visual feedback training in young karate athletes[J]. Muscles,Ligaments and Tendons Journal,2014,4(2):137-140.
[4] JONES C M,GRIFFITHS P C,MELLALIEU S D. Training load and fatigue marker associations with injury and illness:a systematic review of longitudinal studies[J]. Sports Medicine,2017,47(5):943-974.
[5] HALSON S L. Monitoring training load to understand fatigue in athletes[J]. Sports Medicine,2014,44(S2):139-147.
[6] 吕咏. 从符号互动理论看当今社会运动健身类APP热的现象——以keep app为例[C]//第十一届全国体育科学大会论文摘要汇编. 南京:中国体育科学学会,2019:4330-4332.(LYU Y. On the phenomenon of sports and fitness apps in current society from the perspective of symbolic interaction theory-taking Keep APP as an example[C]//Proceedings of the 11th National Convention on Sports Science of China-Abstracts. Nanjing:China Sport Science Society,2019:4330-4332.)
[7] SIGRIST R,RAUTER G,RIENER R,et al. Augmented visual, auditory,haptic,and multimodal feedback in motor learning:a review[J]. Psychonomic Bulletin and Review,2013,20(1):21-53.
[8] 王瑞元, 苏全生. 运动生理学[M]. 北京:人民体育出版社, 2012:295-296.(WANG R U,SU Q S. Sports Physiology[M]. Beijing:People's Sports Press,2012:295-296.)
[9] LUCAS T. Exploring the effect of realism at the cognitive stage of complex motor skill learning[J]. E-Learning and Digital Media, 2019,16(4):242-266.
[10] HUANG C C,LIU H M,HUANG C L. Intelligent scheduling of execution for customized physical fitness and healthcare system[J]. Technology and Health Care,2016,24(S1):S385-S392.
[11] HUANG C C,HUANG C L,LIU H M. Fool-proofing design and crisis management for customized intelligent physical fitness and healthcare system[J]. Technology and Health Care,2016,24(S1):407-413.
[12] QI J,YANG P,HANNEGHAN M,et al. A hybrid hierarchical framework for gym physical activity recognition and measurement using wearable sensors[J]. IEEE Internet of Things Journal, 2019,6(2):1384-1393.
[13] HAUSBERGER P,FERNBACH A,KASTNER W. IMU-based smart fitness devices for weight training[C]//Proceedings of the 42nd Annual Conference of the IEEE Industrial Electronics Society. Piscataway:IEEE,2016:5182-5189.
[14] ÖRÜCÜ S,SELEK M. Design and validation of rule-based expert system by using Kinect V2 for real-time athlete support[J]. Applied Sciences,2020,10(2):No. 611.
[15] SHIH H C. A survey of content-aware video analysis for sports[J]. IEEE Transactions on Circuits and Systems for Video Technology,2018,28(5):1212-1231.
[16] TRAN D, BOURDEV L, FERGUS R, et al. Learning spatiotemporal features with 3D convolutional networks[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE,2015:4489-4497.
[17] SIMONYAN K, ZISSERMAN A. Two-stream convolutional networks for action recognition in videos[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:568-576.
[18] KARPATHY A,TODERICI G,SHETTY S,et al. Large-scale video classification with convolutional neural networks[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2014:1725-1732.
[19] WANG L, XIONG Y, WANG Z, et al. Temporal segment networks:towards good practices for deep action recognition[C]//Proceedings of the 2016 European Conference on Computer Vision,LNCS 9912. Cham:Springer,2016:20-36.
[20] ZHU W,HU J,SUN G,et al. A key volume mining deep framework for action recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1991-1999.
[21] 左国玉, 徐兆坤, 卢佳豪, 等. 基于结构优化的DDAG-SVM上肢康复训练动作识别方法[J]. 自动化学报,2020,46(3):549-561.(ZUO G Y,XU Z K,LU J H,et al. A structure-optimized DDAG-SVM action recognition method for upper limb rehabilitation training[J]. Acta Automatica Sinica,2020,46(3):549-561.)
[22] 闫航, 陈刚, 佟瑶, 等. 基于姿态估计与GRU网络的人体康复动作识别[J]. 计算机工程,2021,47(1):12-20.(YAN H,CHEN G,TONG Y,et al. Human rehabilitation action recognition based on pose estimation and GRU network[J]. Computer Engineering, 2021,47(1):12-20.)
[23] LI H,TANG J,WU S,et al. Automatic detection and analysis of player action in moving background sports video sequences[J]. IEEE Transactions on Circuits and Systems for Video Technology, 2010,20(3):351-364.
[24] BLANK M,GORELICK L,SHECHTMAN E,et al. Actions as spacetime shapes[C]//Proceedings of the 10th IEEE International Conference on Computer Vision. Piscataway:IEEE,2005:1395-1402.
[25] BOBICK A F,DAVIS J W. The recognition of human movement using temporal templates[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2001,23(3):257-267.
[26] SCHÜLDT C, LAPTEV I, CAPUTO B. Recognizing human actions:a local SVM approach[C]//Proceedings of the 17th International Conference on Pattern Recognition. Piscataway:IEEE,2004:32-36.
[27] NIEBLES J C,WANG H,LI F F. Unsupervised learning of human action categories using spatial-temporal words[J]. International Journal of Computer Vision,2008,79(3):299-318.
[28] SALES DE SOUZA L, GATTO B B, FUKUI K. Enhancing discriminability of randomized time warping for motion recognition[C]//Proceedings of the 15th IAPR International Conference on Machine Vision Applications. Piscataway:IEEE,2017:77-80.
[29] THURAU C. Behavior histograms for action recognition and human detection[C]//Proceedings of the 2007 Workshop on Human Motion,LNCS 4814. Berlin:Springer,2007:299-312.
[30] KUMARI S,MITRA S K. Human action recognition using DFT[C]//Proceedings of the 3rd National Conference on Computer Vision,Pattern Recognition,Image Processing and Graphics. Piscataway:IEEE,2011:239-242.
[31] CHERLA S,KULKARNI K,KALE A,et al. Towards fast,viewinvariant human action recognition[C]//Proceedings of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2008:1-8.
[32] FANG H,XIE S,TAI Y W,et al. RMPE:regional multi-person pose estimation[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2353-2362.
[33] CAO Z,SIMON T,WEI S E,et al. Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:1302-1310.
[34] BUX A,ANGELOV P,HABIB Z. Vision based human activity recognition:a review[M]//ANGELOV P,GEGOV A,JAYNE C, et al. Advances in Computational Intelligence Systems,AISC 513. Cham:Springer,2017:341-371.
[35] YAO A, GALL J, FANELLI G, et al. Does human action recognition benefit from pose estimation?[C]//Proceedings of the 2011 British Machine Vision Conference. Durham:BMVA Press, 2011:No. 67.
[36] YAN S,XIONG Y,LIN D. Spatial temporal graph convolutional networks for skeleton-based action recognition[C]//Proceedings of the 32nd AAAI Conference on Artificial Intelligence. Palo Alto, CA:AAAI Press,2018:7444-7452.
[37] HSU C W,CHANG C C,LIN C J. A practical guide to support vector classification[J]. BJU International,2008,101(1):1396-1400.
[38] RAKTHANMANON T, CAMPANA B, MUEEN A, et al. Searching and mining trillions of time series subsequences under dynamic time warping[C]//Proceedings of the 18th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York:ACM,2012:262-270.