Fall detection algorithm based on joint point features

doi:10.11772/j.issn.1001-9081.2021040618

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (2): 622-630.DOI: 10.11772/j.issn.1001-9081.2021040618

• Frontier and comprehensive applications • Previous Articles Next Articles

Fall detection algorithm based on joint point features

Jianrong CAO¹^,², Yaqin ZHU¹(), Yuting ZHANG¹, Junjie LYU¹, Hongjuan YANG¹^,²

^1.School of Information and Electrical Engineering，Shandong Jianzhu University，Jinan Shandong 250101，China
^2.Shandong Provincial Key Laboratory of Intelligent Buildings Technology （Shandong Jianzhu University），Jinan Shandong 250101，China

Received:2021-04-20 Revised:2021-07-16 Accepted:2021-07-23 Online:2022-02-11 Published:2022-02-10
Contact: Yaqin ZHU
About author:CAO Jianrong， born in 1965， Ph. D.， professor. His research interests include pattern recognition intelligent information processing， video analysis， deep learning， data mining.
ZHU Yaqin， born in 1995， M. S. candidate. Her research interests include pattern recognition and intelligent information processing， video analysis， deep learning.
ZHANG Yuting， born in 1997， M. S. candidate. Her research interests include deep learning， video summary， image processing.
LYU Junjie， born in 1991， M. S. candidate. His research interests include pattern recognition and intelligent information processing， video analysis， deep learning， data mining.
YANG Hongjuan， born in 1978， Ph. D.， associate professor. Her research interests include pattern recognition and intelligent information processing， deep learning， data mining.
Supported by:
Shandong Provincial Key Research and Development Program(2019GSF111054);Shandong Province Major Scientific and Technological Innovation Project(2019JZZY010120)

基于关节点特征的跌倒检测算法

曹建荣¹^,², 朱亚琴¹(), 张玉婷¹, 吕俊杰¹, 杨红娟¹^,²

^1.山东建筑大学信息与电气工程学院，济南 250101
^2.山东省智能建筑技术重点实验室（山东建筑大学），济南 250101

通讯作者: 朱亚琴
作者简介:曹建荣（1965—），男，山东济南人，教授，博士，主要研究方向：模式识别与智能信息处理、视频分析、深度学习、数据挖掘；
朱亚琴（1995—），女，山东济南人，硕士研究生，主要研究方向：模式识别与智能信息处理、视频分析、深度学习；
张玉婷（1997—），女，山东济南人，硕士研究生，主要研究方向：深度学习、视频总结、图像处理；
吕俊杰（1991—），男，山东济南人，硕士研究生，主要研究方向：模式识别与智能信息处理、视频分析、深度学习、数据挖掘；
杨红娟（1978—），女，山东济南人，副教授，博士，主要研究方向：模式识别与智能信息处理、深度学习、数据挖掘。
基金资助:
山东省重点研发计划项目(2019GSF111054);山东省重大科技创新工程(2019JZZY010120)

Abstract

Abstract:

In order to solve the problems of large amount of network computation and difficulty in distinguishing falling-like behaviors in fall detection algorithms， a fall detection algorithm based on joint point features was proposed. Firstly， based on the current advanced CenterNet algorithm， a Depthwise Separable Convolution-CenterNet （DSC-CenterNet） joint point detection algorithm was proposed to accurately detect human joint points and obtain joint point coordinates while reducing the amount of backbone network computation. Then， based on the joint point coordinates and prior knowledge of the human body， the spatial and temporal features expressing the fall behavior were extracted as the joint point features. Finally， the joint point feature vector was input into the fully connected layer and processed by Sigmoid classifier to output two categories： fall or non-fall， thereby achieving the fall detection of human targets. Experimental results on UR Fall Detection dataset show that the proposed algorithm has the average accuracy of fall detection under different state changes reached 98.00%， the accuracy of distinguishing falling-like behaviors reached 98.22% and the fall detection speed of 18.6 frame/s. Compared with the algorithm of the original CenterNet combining with joint point features， the algorithm of DSC-CenterNet combining with joint point features has the average detection accuracy increased by 22.37%. The improved speed can effectively meet the realtime requirement of the human fall detection tasks under surveillance video. This algorithm can effectively increase fall detection speed and accurately detect the fall state of human body， which further verifies the feasibility and efficiency of fall detection algorithm based on joint point features in the video fall behavior analysis.

Key words: fall detection, deep learning, CenterNet algorithm, joint point detection, joint point feature

摘要：

针对跌倒检测算法中存在网络计算量大和类跌倒行为难以区分的问题，提出一种基于关节点特征的跌倒检测算法。首先，在目前先进的CenterNet算法基础上提出了深度可分离卷积CenterNet （DSC-CenterNet）关节点检测算法，从而在减少骨干网络计算量的同时准确检测人体关节点并获取关节点坐标；然后，基于关节点位置和人体先验知识来提取可充分表达跌倒行为的空间特征和时间特征作为关节点特征；最后，把关节点特征向量输入全连接层，并经Sigmoid分类器输出跌倒或非跌倒两种类别，从而实现人体目标的跌倒检测。实验结果表明，所提算法在UR Fall Detection数据集上对不同状态变化下跌倒检测的平均准确率达到98.00%，区分类跌倒行为的准确率达到98.22%，跌倒检测速度为18.6 frame/s。与原CenterNet结合关节点特征跌倒检测的算法相比，DSC-CenterNet结合关节点特征算法的跌倒检测速度提升了22.37%，提高后的速度可有效满足视频监控下人体跌倒检测任务的实时性。该算法能有效提高跌倒检测速度并对人体跌倒状态进行准确检测，且进一步验证了基于关节点特征的跌倒检测算法在视频跌倒行为分析中的可行性与高效性。

关键词: 跌倒检测, 深度学习, CenterNet算法, 关节点检测, 关节点特征

CLC Number:

TP391.4

Jianrong CAO, Yaqin ZHU, Yuting ZHANG, Junjie LYU, Hongjuan YANG. Fall detection algorithm based on joint point features[J]. Journal of Computer Applications, 2022, 42(2): 622-630.

曹建荣, 朱亚琴, 张玉婷, 吕俊杰, 杨红娟. 基于关节点特征的跌倒检测算法[J]. 《计算机应用》唯一官方网站, 2022, 42(2): 622-630.

Figures/Tables 13

Fig. 1 Flowchart of the proposed algorithm

Fig. 2 Network structure of DSC-CenterNet

Fig. 3 Changes of width to height ratio of human body in different fall states

Fig. 4 Changes of head to ground height in different fall states

Fig. 5 Changes of tilt angle of human body

Fig. 6 Changes of human body tilt angle in different fall states

Fig. 7 Horizontal changes of trunk in different fall states

Fig. 8 Changes of lower limb height in different fall states

Fig. 9 Fall video sequences after joint point detection by DSC-CenterNet algorithm

Tab. 1 Experimental results of fall detection algorithm under different state changes

状态变化	视频序号	检测跌倒帧数	实际跌倒帧数	精确率/%	召回率/%	准确率/%
行走— 跌倒	01	44	42	95.45	100.00	98.93
	05	30	28	93.30	100.00	98.84
	09	38	36	94.70	100.00	99.23
坐着— 跌倒	10	60	57	95.00	100.00	96.75
	20	60	57	95.00	100.00	97.25
	26	29	30	100.00	96.67	98.46
站立— 跌倒	07	53	44	83.02	100.00	95.05
	15	18	19	100.00	94.73	98.48
	17	8	9	100.00	88.89	98.98
平均						98.00

Tab. 2 Experimental results of fall detection algorithm judging falling-like behaviors

类跌倒行为	视频序号	检测非跌倒帧数	误检跌倒帧数	准确率/%
下蹲	03	182	1	99.45
下蹲	12	314	1	99.68
弯腰	06	200	9	95.69
弯腰	16	227	1	99.56
坐下	09	149	6	96.12
坐下	18	227	1	99.56
躺下	31	121	4	96.80
躺下	21	182	2	98.91
平均				98.22

Tab. 3 Comparison of joint point detection and fall detection speeds before and after CenterNet improvement

算法	检测速度
CenterNet关节点检测	15.7
DSC-CenterNet关节点检测	19.1
CenterNet关节点检测+跌倒检测	15.2
DSC-CenterNet关节点检测+跌倒检测	18.6

Tab. 4 Comparison of fall detection accuracy of different algorithms on UR Fall Detection dataset

算法	准确率	算法	准确率
文献［16］算法	90.74	文献［22］算法	97.30
文献［20］算法	95.45	本文算法	98.00
文献［21］算法	94.70

References 22

1	GUTIÉRREZ J， RODRÍGUEZ V， MARTIN S. Comprehensive review of vision-based fall detection systems［J］. Sensors， 2021， 21（3）： No.947. 10.3390/s21030947
2	郝莲，李胜，向芳，等. 2003-2018年中国跌倒死亡率的时间趋势分析［J］.现代预防医学， 2020， 47（23）： 4357-4360.
	HAO L， LI S， XIANG F， et al. Time trend of falls mortality in China， 2003-2018［J］. Modern Preventive Medicine， 2020， 47（23）： 4357-4360.
3	ZHUANG X D， HUANG J， POTAMIANOS G， et al. Acoustic fall detection using Gaussian mixture models and GMM supervectors ［C］// Proceedings of the 2009 IEEE International Conference on Acoustics， Speech and Signal Processing. Piscataway： IEEE， 2009： 69-72. 10.1109/icassp.2009.4959522
4	IRTAZA A， ADNAN S M， AZIZ S， et al. A framework for fall detection of elderly people by analyzing environmental sounds through acoustic local ternary patterns ［C］// Proceedings of the 2017 IEEE International Conference on Systems， Man， and Cybernetics. Piscataway： IEEE， 2017： 1558-1563. 10.1109/smc.2017.8122836
5	PANNURAT N， THIEMIARUS S， NANTAJEEWARAWAT E. A hybrid temporal reasoning framework for fall monitoring［J］. IEEE Sensors Journal， 2017， 17（6）： 1749-1759. 10.1109/jsen.2017.2649542
6	SABATINI A M， LIGORIO G， MANNINI A. Prior-to- and post-impact fall detection using inertial and barometric altimeter measurements［J］. IEEE Transactions on Neural Systems and Rehabilitation Engineering， 2016， 24（7）： 774-783. 10.1109/tnsre.2015.2460373
7	WANG F T， CHAN H L， HSU M H， et al. Threshold-based fall detection using a hybrid of tri-axial accelerometer and gyroscope［J］. Physiological Measurement， 2018， 39（10）： No.105002. 10.1088/1361-6579/aae0eb
8	HUA M J， NAN Y B， LIAN S G. Falls prediction based on body keypoints and Seq2Seq architecture ［C］// Proceedings of the 2019 IEEE/CVF International Conference on Computer Vision Workshop. Piscataway： IEEE， 2019： 1251-1259. 10.1109/iccvw.2019.00158
9	LIU J X， XIA Y Y， TANG Z. Privacy-preserving video fall detection using visual shielding information［J］. The Visual Computer， 2021， 37（2）： 359-370. 10.1007/s00371-020-01804-w
10	FAN Y X， LEVINE M D， WEN G J， et al. A deep neural network for real-time detection of falling humans naturally occurring scenes［J］. Neurocomputing， 2017， 260： 43-58. 10.1016/j.neucom.2017.02.082
11	ZULJ S， SEKETA G， LACKOVIC I， et al. Accuracy comparison of ML-based fall detection algorithms using two different acceleration derived feature vectors［M］// LHOTSKA L， SUKUPOVA L， LACKOVIĆ I， et al. Proceeding of the 2018 World Congress on Medical Physics and Biomedical Engineering， IFMBE 68/2.Singapore： Springer， 2019： 481-485. 10.1007/978-981-10-9038-7_89
12	申代友，库洪安，皮红英，等.基于深度相机的老年跌倒监护系统［J］.中国医学物理学杂志， 2019， 36（2）： 223-228. 10.3969/j.issn.1005-202X.2019.02.019
	SHEN D Y， KU H A， PI H Y， et al. Depth camera-based fall detection system for the elderly［J］. Chinese Journal of Medical Physics， 2019， 36（2）： 223-228. 10.3969/j.issn.1005-202X.2019.02.019
13	朱靖.基于卷积神经网络的二维人体姿态估计研究［D］.杭州：杭州电子科技大学， 2019： 36-42. 10.20937/rica.2019.35.esp01.06
	ZHU J. Human pose estimation based on convolutional neural network［D］. Hangzhou： Hangzhou Dianzi University， 2019： 36-42. 10.20937/rica.2019.35.esp01.06
14	李文阳.基于Kinect v2的跌倒行为检测与分析［D］.银川：：北方民族大学， 2019： 13-44. （LI W Y. Detection and analysis on fall behavior based on Kinect sensor v2［D］. Yinchuan：： North Minzu University， 2019： 13-44.）. 10.16652/j.issn.1004-373x.2019.06.034
15	LIN C B， DONG Z Q， KUAN W K， et al. A framework for fall detection based on OpenPose skeleton and LSTM/GRU models［J］. Applied Sciences， 2021， 11（1）： No.329. 10.3390/app11010329
16	蔡文郁，郑雪晨，郭嘉豪，等.基于SVM-MultiCNN模型的视觉感知跌倒检测算法［J］.杭州电子科技大学学报（自然科学版）， 2020， 40（5）： 60-66.
	CAI W Y， ZHENG X C， GUO J H， et al. Visual perception fall detection algorithm based on SVM-MultiCNN model［J］. Journal of Hangzhou Dianzi University （Natural Sciences）， 2020， 40（5）： 60-66.
17	CAO Z， HIDALGO G， SIMON T， et al. OpenPose： realtime multi- person 2D pose estimation using part affinity fields［J］. IEEE Transaction on Pattern Analysis and Machine Intelligence， 2021， 43（1）： 172-186. 10.1109/tpami.2019.2929257
18	袁鹏泰，刘宁钟.基于关节点提取的老年人跌倒检测算法［J］.计算机技术与发展， 2019， 29（9）： 200-204， 151. 10.3969/j.issn.1673-629X.2019.09.038
	YUAN P T， LIU N Z. Elderly fall detection algorithm based on joint point extraction［J］. Computer Technology and Development， 2019， 29（9）： 200-204， 151. 10.3969/j.issn.1673-629X.2019.09.038
19	ZHOU X Y， WANG D Q， KRÄHENBÜHL P. Objects as points［EB/OL］. （2019-04-25）［2021-02-05］. . 10.1109/cvpr.2019.00094
20	李冬冬.基于视频监控的室内跌倒行为检测研究［D］.南昌：南昌大学， 2020： 33-51. 10.18402/resci.2020.05.04
	LI D D. Research on indoor fall behavior detection based on video monitoring［D］. Nanchang： Nanchang University， 2020： 33-51. 10.18402/resci.2020.05.04
21	CAI X， LIU X Y， AN M Y， et al. Vision-based fall detection using dense block with multi-channel convolutional fusion strategy［J］. IEEE Access， 2021， 9： 18318-18325. 10.1109/access.2021.3054469
22	MERROUCHE F， BAHA N. Fall detection based on shape deformation［J］. Multimedia Tools and Applications， 2020， 79（41/42）： 30489-30508. 10.1007/s11042-019-08428-w

[1]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[2]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[3]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[4]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[5]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[6]	Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557.
[7]	Yanjie GU, Yingjun ZHANG, Xiaoqian LIU, Wei ZHOU, Wei SUN. Traffic flow forecasting via spatial-temporal multi-graph fusion [J]. Journal of Computer Applications, 2024, 44(8): 2618-2625.
[8]	Qianhong SHI, Yan YANG, Yongquan JIANG, Xiaocao OUYANG, Wubo FAN, Qiang CHEN, Tao JIANG, Yuan LI. Multi-granularity abrupt change fitting network for air quality prediction [J]. Journal of Computer Applications, 2024, 44(8): 2643-2650.
[9]	Yiqun ZHAO, Zhiyu ZHANG, Xue DONG. Anisotropic travel time computation method based on dense residual connection physical information neural networks [J]. Journal of Computer Applications, 2024, 44(7): 2310-2318.
[10]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[11]	Xun SUN, Ruifeng FENG, Yanru CHEN. Monocular 3D object detection method integrating depth and instance segmentation [J]. Journal of Computer Applications, 2024, 44(7): 2208-2215.
[12]	Zheng WU, Zhiyou CHENG, Zhentian WANG, Chuanjian WANG, Sheng WANG, Hui XU. Deep learning-based classification of head movement amplitude during patient anaesthesia resuscitation [J]. Journal of Computer Applications, 2024, 44(7): 2258-2263.
[13]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[14]	Zhi ZHANG, Xin LI, Naifu YE, Kaixi HU. DKP： defending against model stealing attacks based on dark knowledge protection [J]. Journal of Computer Applications, 2024, 44(7): 2080-2086.
[15]	Yajuan ZHAO, Fanjun MENG, Xingjian XU. Review of online education learner knowledge tracing [J]. Journal of Computer Applications, 2024, 44(6): 1683-1698.

Fall detection algorithm based on joint point features

基于关节点特征的跌倒检测算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 22

Related Articles 15

Recommended Articles

Metrics