Smoking behavior detection algorithm based on human skeleton key points

doi:10.11772/j.issn.1001-9081.2021061063

Journal of Computer Applications ›› 2021, Vol. 41 ›› Issue (12): 3602-3607.DOI: 10.11772/j.issn.1001-9081.2021061063

Special Issue: 第十八届中国机器学习会议(CCML 2021)

• The 18th China Conference on Machine Learning • Previous Articles Next Articles

Smoking behavior detection algorithm based on human skeleton key points

Wanqing XU¹^,²^,³, Baodong WANG¹^,²^,³, Yimei HUANG¹^,²^,³, Jinping LI¹^,²^,³()

^1.School of Information Science and Engineering，University of Jinan，Jinan Shandong 250022，China
^2.Shandong Provincial Key Laboratory of Network Based Intelligent Computing （University of Jinan），Jinan Shandong 250022，China
^3.Shandong College and University Key Laboratory of Information Processing and Cognitive Computing in the 13th Five-Year Plan （University of Jinan），Jinan Shandong 250022，China

Received:2021-05-12 Revised:2021-07-27 Accepted:2021-08-05 Online:2021-12-28 Published:2021-12-10
Contact: Jinping LI
About author:XU Wanqing， born in 1998， M. S. candidate. Her research interests include image processing， machine learning.
WANG Baodong， born in 1995， M. S. candidate. His research interests include image processing， machine learning.
HUANG Yimei， born in 1980， M. S.， lecturer. Her research interests include computer vision， pattern recognition， image processing， machine learning.
Supported by:
the Shandong Provincial Key Research and Development Program(2017CXGC0810)

基于人体骨骼关键点的吸烟行为检测算法

徐婉晴¹^,²^,³, 王保栋¹^,²^,³, 黄艺美¹^,²^,³, 李金屏¹^,²^,³()

^1.济南大学信息科学与工程学院，济南 250022
^2.山东省网络环境智能计算技术重点实验室（济南大学），济南 250022
^3.山东省“十三五”高校信息处理与认知计算重点实验室（济南大学），济南 250022

通讯作者: 李金屏
作者简介:徐婉晴（1998—），女，河北石家庄人，硕士研究生，CCF会员，主要研究方向：图像处理、机器学习
王保栋（1995—），男，山东青岛人，硕士研究生，CCF会员，主要研究方向：图像处理、机器学习
黄艺美（1980—），女，福建永泰人，讲师，硕士，CCF会员，主要研究方向：计算机视觉、模式识别、图像处理、机器学习；
基金资助:
山东省重点研发计划项目(2017CXGC0810)

Abstract

Abstract:

In view of the small target of cigarette butts in surveillance videos of public places and the easy divergence of smoke generated by smoking， it is difficult to determine the smoking behavior only by target detection algorithm. Considering that the algorithm of posture estimation using skeleton key points is becoming more and more mature， a smoking behavior detection algorithm was proposed by using the relationship between human skeleton key points and smoking behavior. Firstly， AlphaPose and RetinaFace were used to detect the key points of human skeleton and face respectively. According to the ratio of distance between wrist and middle point of two corners of mouth and between wrist and the eye on the same side， a method for calculating whether the Smoking Action Ratio （SAR） in humans falls within the Golden Ratio of Smoking Actions （GRSA） to distinguish smoking from non-smoking behaviors was proposed. Then， YOLOv4 was used to detect whether cigarette butts existed in the video. The results of GRSA determination and YOLOv4 were combined to determine the possibility of smoking behavior in the video and make a determination of whether smoking behavior was present. The self-recorded dataset test shows that the proposed algorithm can accurately detect smoking behavior with the accuracy reached 92%.

Key words: smoking behavior detection, key points of human skeleton, AlphaPose, RetinaFace, YOLOv4

摘要：

针对公共场所的监控视频中烟头目标较小并且吸烟产生的烟雾易发散，仅依靠目标检测算法检测烟头或者烟雾来判定吸烟行为存在较大难度的问题，考虑到利用骨骼关键点来进行姿态估计的算法越来越成熟，提出一种利用人体骨骼关键点和吸烟行为之间的关系来进行吸烟行为检测的算法。该算法首先利用AlphaPose和RetinaFace分别检测出人体骨骼关键点和脸部关键点信息，根据手腕到两嘴角中点和手腕到同侧眼睛的距离之比，提出一种计算人体的吸烟动作比例（SAR）是否属于吸烟动作黄金比例（GRSA）的方法以区分吸烟与非吸烟行为；再利用YOLOv4检测视频中是否存在烟头；最后结合GRSA判定和YOLOv4的结果来确定视频中存在吸烟行为的可能性高低，作出是否有吸烟行为的判定。经过笔者录制的数据集测试，结果表明所提算法可以准确检测到吸烟行为，准确率达到92%。

关键词: 吸烟行为检测, 人体骨骼关键点, AlphaPose, RetinaFace, YOLOv4

CLC Number:

TP391.4

Wanqing XU, Baodong WANG, Yimei HUANG, Jinping LI. Smoking behavior detection algorithm based on human skeleton key points[J]. Journal of Computer Applications, 2021, 41(12): 3602-3607.

徐婉晴, 王保栋, 黄艺美, 李金屏. 基于人体骨骼关键点的吸烟行为检测算法[J]. 《计算机应用》唯一官方网站, 2021, 41(12): 3602-3607.

Figures/Tables 14

References 16

1	陈睿龙，罗磊，蔡志平，等. 基于深度学习的实时吸烟检测算法［J］. 计算机科学与探索， 2021， 15（2）：327-337.
	CHEN R L， LUO L， CAI Z P， et al. Algorithm for real-time smoking detection based on deep learning［J］. Journal of Frontiers of Computer Science and Technology， 2021， 15（2）： 327-337.
2	程淑红，马晓菲，张仕军，等. 基于多任务分类的吸烟行为检测［J］. 计量学报， 2020， 41（5）：538-543. 10.3969/j.issn.1000-1158.2020.05.05
	CHENG S H， MA X F， ZHANG S J， et al. Smoking detection algorithm based on multitask classification［J］. Acta Metrologica Sinica， 2020， 41（5）： 538-543. 10.3969/j.issn.1000-1158.2020.05.05
3	黄训平，贾克斌，刘鹏宇. 基于交通监控的出租车司机吸烟行为自动检测［J］. 计算机仿真， 2020， 37（12）：337-344. 10.3969/j.issn.1006-9348.2020.12.070
	HUANG X P， JIA K B， LIU P Y. Automatic detection of taxi driver smoking behavior based on traffic monitoring［J］. Computer Simulation， 2020， 37（12）： 337-344. 10.3969/j.issn.1006-9348.2020.12.070
4	林镇涛. 中距离户外施工人员吸烟行为检测方法研究［D］. 成都：电子科技大学， 2020. 10.18240/ijo.2021.06.16
	LIN Z T. Research on smoking behavior detection method for outdoor constructors in mid-distance［D］. Chengdu： University of Electronic Science and Technology of China， 2020. 10.18240/ijo.2021.06.16
5	尹建芹，刘小丽，田国会，等. 基于关键点序列的人体动作识别［J］. 机器人， 2016， 38（2）：200-207， 216. 10.13973/j.cnki.robot.2016.0200
	YIN J Q， LIU X L， TIAN G H， et al. Human action recognition based on the sequence of key points［J］. Robot， 2016， 38（2）： 200-207， 216. 10.13973/j.cnki.robot.2016.0200
6	丁重阳，刘凯，李光，等. 基于时空权重姿态运动特征的人体骨架行为识别研究［J］. 计算机学报， 2020， 43（1）：29-40. 10.11897/SP.J.1016.2020.00029
	DING C Y， LIU K， LI G， et al. Spatio-temporal weighted posture motion features for human skeleton action recognition research［J］. Chinese Journal of Computers， 2020， 43（1）： 29-40. 10.11897/SP.J.1016.2020.00029
7	谭嘉崴，丁其川，白忠玉. 基于视频帧连贯信息的3维人体姿势优化估计方法［J］. 机器人， 2021， 43（1）：9-16.
	TAN J W， DING Q C， BAI Z Y. Optimal estimation method of 3-dimensional human pose based on video frame coherent information［J］. Robot， 2021， 43（1）： 9-16.
8	刘婧，杨旭，刘董经典，等.基于人体关节点的多人吸烟动作识别算法［J］. 计算机工程与应用， 2021，57（1）：234-241.
	LIU J， YANG X， LIU D J D， et al. Multi-person smoking action recognition algorithm based on human joint points［J］. Computer Engineering and Applications， 2021， 57（1）： 234-241.
9	FANG H S， XIE S Q， TAI Y W， et al. RMPE： regional multi-person pose estimation［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2353-2362. 10.1109/iccv.2017.256
10	牛作东，覃涛，李捍东，等. 改进RetinaFace的自然场景口罩佩戴检测算法［J］. 计算机工程与应用， 2020， 56（12）：1-7. 10.3778/j.issn.1002-8331.2002-0402
	NIU Z D， QIN T， LI H D， et al. Improved algorithm of RetinaFace for natural scene mask wear detection［J］. Computer Engineering and Applications， 2020， 56（12）： 1-7. 10.3778/j.issn.1002-8331.2002-0402
11	YANG S， LUO P， LOY C C， et al. WIDER FACE： a face detection benchmark［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 5525-5533. 10.1109/cvpr.2016.596
12	ZHU C C， TAO R， LUU K， et al. Seeing small faces from robust anchor’s perspective［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 5127-5136. 10.1109/cvpr.2018.00538
13	LI J， WANG Y B， WANG C A， et al. DSFD： dual shot face detector［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 5055-5064. 10.1109/cvpr.2019.00520
14	ZHANG S F， WEN L Y， SHI H L， et al. Single-shot scale-aware network for real-time face detection［J］. International Journal of Computer Vision， 2019， 127（6/7）： 537-559. 10.1007/s11263-019-01159-3
15	DENG J K， GUO J， ZHOU Y X， et al. RetinaFace： single-stage dense face localisation in the wild［EB/OL］. （2019-05-04）［2021-02-02］.. 10.1109/cvpr42600.2020.00525
16	BOCHKOVSKIY A， WANG C Y， LIAO H Y M. YOLOv4： optimal speed and accuracy of object detection［EB/OL］. （2020-04-23）［2021-02-02］.. 10.1109/cvpr46437.2021.01283

只含吸烟动作	不含吸烟动作		混合各种动作
只含吸烟动作	含喝水动作	含挠头动作	混合各种动作
200	30	30	40

只含吸烟动作	不含吸烟动作		混合各种动作
只含吸烟动作	含喝水动作	含挠头动作	混合各种动作
200	30	30	40

人脸和摄像头大致处于同一水平高度			摄像头处于人脸斜上方大约45°
GRSA	查准率P/%	查全率R/%	GRSA	查准率P/%	查全率R/%
标准①［0.55， 0.85］	85.3	95.2	标准④［0.50， 0.80］	87.6	96.0
标准②［0.6， 0.8］	92.8	92.4	标准⑤［0.55， 0.75］	92.1	93.2
标准③［0.65， 0.75］	93.0	79.2	标准⑥［0.60， 0.70］	90.2	62.8

人脸和摄像头大致处于同一水平高度			摄像头处于人脸斜上方大约45°
GRSA	查准率P/%	查全率R/%	GRSA	查准率P/%	查全率R/%
标准①［0.55， 0.85］	85.3	95.2	标准④［0.50， 0.80］	87.6	96.0
标准②［0.6， 0.8］	92.8	92.4	标准⑤［0.55， 0.75］	92.1	93.2
标准③［0.65， 0.75］	93.0	79.2	标准⑥［0.60， 0.70］	90.2	62.8

算法	人脸和摄像头大致处于同一水平高度		摄像头处于人脸斜上方约45°
算法	P/%	R/%	P/%	R/%
YOLOv4	80.0	76.8	85.1	82.4
GRSA	90.3	82.0	91.9	86.4
YOLOv4+GRSA	92.8	92.4	92.1	93.2

Smoking behavior detection algorithm based on human skeleton key points

基于人体骨骼关键点的吸烟行为检测算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 14

References 16

Related Articles 9

Recommended Articles

Metrics

[1]	Juming HAO, Jingyu YANG, Shumei HAN, Yangping WANG. YOLOv4 highway pavement crack detection method using Ghost module and ECA [J]. Journal of Computer Applications, 2023, 43(4): 1284-1290.
[2]	GUO Keyou, LI Xue, YANG Min. Real‑time detection method of traffic information based on lightweight YOLOv4 [J]. Journal of Computer Applications, 2023, 43(1): 74-80.
[3]	Zhihong XI, Jiaxu WEN. Indoor dynamic scene localization and mapping based on target detection [J]. Journal of Computer Applications, 2022, 42(9): 2853-2857.
[4]	Xinyu ZHANG, Sheng DING, Zhipei YANG. Traffic sign detection algorithm based on improved attention mechanism [J]. Journal of Computer Applications, 2022, 42(8): 2378-2385.
[5]	Zhifeng ZHONG, Yifan XIA, Dongping ZHOU, Yangtian YAN. Lightweight object detection algorithm based on improved YOLOv4 [J]. Journal of Computer Applications, 2022, 42(7): 2201-2209.
[6]	Qianzhou CAI, Bochuan ZHENG, Xiangyin ZENG, Jin HOU. Wildlife object detection combined with solving method of long-tail data [J]. Journal of Computer Applications, 2022, 42(4): 1284-1291.
[7]	Tianmin DENG, Guotao MAO, Zhenhao ZHOU, Zhijian DUAN. Road vehicle detection and recognition algorithm based on densely connected convolutional neural network [J]. Journal of Computer Applications, 2022, 42(3): 883-889.
[8]	TAN Zhaoyi, CHEN Baifan. Enhanced vehicle 3D surround view based on coordinate inverse mapping [J]. Journal of Computer Applications, 2021, 41(4): 1165-1171.
[9]	Haojie LOU, Yuanlin ZHENG, Kaiyang LIAO, Hao LEI, Jia LI. Defect target detection for printed matter based on Siamese-YOLOv4 [J]. Journal of Computer Applications, 2021, 41(11): 3206-3212.

SAR属于GRSA的持续时间t是否大于T	是否存在烟头	吸烟行为可能性/%	是否吸烟行为
是	是	100	是
是	否	60	是
否	是	40	是
否	否	0	否

SAR属于GRSA的持续时间t是否大于T	是否存在烟头	吸烟行为可能性/%	是否吸烟行为
是	是	100	是
是	否	60	是
否	是	40	是
否	否	0	否