基于人眼状态的瞌睡识别算法

doi:10.11772/j.issn.1001-9081.2020122058

《计算机应用》唯一官方网站 ›› 2021, Vol. 41 ›› Issue (11): 3213-3218.DOI: 10.11772/j.issn.1001-9081.2020122058

基于人眼状态的瞌睡识别算法

孙琳¹(), 袁玉波¹^,²

^1.华东理工大学信息科学与工程学院，上海 200237
^2.上海大数据与互联网受众工程技术研究中心，上海 200072

收稿日期:2020-12-29 修回日期:2021-04-28 接受日期:2021-05-19 发布日期:2021-04-28 出版日期:2021-11-10
通讯作者: 孙琳
作者简介:孙琳（1998—），女，江西抚州人，硕士研究生，主要研究方向：数据分析、机器视觉
袁玉波（1976—），男，云南宣威人，副教授，博士，主要研究方向：人工智能、数据科学、大数据分析、数据质量评估、数据挖掘。
基金资助:
上海市工程技术中心项目(18DZ2252300)

Drowsiness recognition algorithm based on human eye state

Lin SUN¹(), Yubo YUAN¹^,²

^1.School of Information Science and Engineering，East China University of Science and Technology，Shanghai 200237，China
^2.Shanghai Engineering Research Center of Big Data and Internet Audience，Shanghai 200072，China

Received:2020-12-29 Revised:2021-04-28 Accepted:2021-05-19 Online:2021-04-28 Published:2021-11-10
Contact: Lin SUN
About author:SUN Lin，born in 1998，M. S. candidate. Her research interests include data analysis，computer vision
YUAN Yubo，born in 1976，Ph. D.，associate professor. His research interests include artificial intelligence，data science，big data analysis，data quality assessment，data mining.
Supported by:
the Shanghai Engineering Technology Center Project(18DZ2252300)

摘要/Abstract

摘要：

已有瞌睡识别算法多数基于机器学习或深度学习，没有考虑到人眼闭合状态序列与瞌睡之间的关系。针对上述问题，提出了一种基于人眼状态的瞌睡识别算法。首先，提出了人眼分割和面积计算模型，基于人脸68个特征点，根据人眼特征点构成的极大多边形分割出眼睛区域，并利用眼睛像素点的总数代表眼睛面积大小；其次，计算极大状态下的人眼面积，并利用关键帧挑选算法挑选出最能代表睁眼程度的4帧，根据这4帧的人眼面积与极大状态下的人眼面积计算睁眼阈值，从而构建眼睛闭合度得分模型来确定人眼闭合状态；最后，根据输入视频的人眼闭合得分序列，构建了基于连续多帧序列分析的瞌睡识别模型。在两个国际常用的打哈欠检测数据集（YawDD）和NTHU-DDD数据集上进行瞌睡状态识别，实验结果表明，所提算法在两个数据集上的识别准确率均在80%以上，尤其是在YawDD数据集上，识别准确率达到94%以上。该算法可应用于驾驶员驾驶状态检测、学习者课中状态分析等。

关键词: 面部特征, 人眼定位, 人眼状态识别, 多帧序列分析, 瞌睡识别

Abstract:

Most of the existing drowsiness recognition algorithms are based on machine learning or deep learning， without considering the relationship between the sequence of human eye closed state and drowsiness. In order to solve the problem， a drowsiness recognition algorithm based on human eye state was proposed. Firstly， a human eye segmentation and area calculation model was proposed. Based on 68 feature points of the face， the eye area was segmented according to the extremely large polygon formed by the feature points of human eye， and the total number of eye pixels was used to represent the size of the eye area. Secondly， the area of the human eye in the maximum state was calculated， and the key frame selection algorithm was used to select 4 frames representing the eye opening state the most， and the eye opening threshold was calculated based on the areas of human eye in these 4 frames and in the maximum state. Therefore， the eye closure degree score model was constructed to determine the closed state of the human eye. Finally， according the eye closure degree score sequence of the input video， a drowsiness recognition model was constructed based on continuous multi-frame sequence analysis. The drowsiness state recognition was conducted on the two commonly used international datasets such as Yawning Detection Dataset （YawDD） and NTHU-DDD dataset.Experimental results show that， the recognition accuracy of the proposed algorithm is more than 80% on the two datasets， especially on the YawDD， the proposed algorithm has the recognition accuracy above 94%. The proposed algorithm can be applied to driver status detection during driving， learner status analysis in class and so on.

Key words: facial feature, human eye location, human eye state recognition, multi-frame sequence analysis, drowsiness recognition

中图分类号:

TP391.41

孙琳, 袁玉波. 基于人眼状态的瞌睡识别算法[J]. 计算机应用, 2021, 41(11): 3213-3218.

Lin SUN, Yubo YUAN. Drowsiness recognition algorithm based on human eye state[J]. Journal of Computer Applications, 2021, 41(11): 3213-3218.

图/表 15

图1 瞌睡状态识别流程

Fig. 1 Flow chart of drowsiness state recognition

图2 不同角度人脸特征点示意图

Fig. 2 Schematic diagram of face feature points from different angles

图3 YawDD视频数据集的样本帧示例

Fig. 3 Sample frame examples of YawDD video dataset

图4 清醒状态视频帧示例

Fig. 4 Non-yawning video frame examples

图5 瞌睡状态视频帧示例

Fig. 5 Yawning video frame examples

图6 NTHU-DDD视频数据集的样本帧示例

Fig. 6 Sample frame examples of NTHU-DDD video dataset

图7 SER、EAR随时间的变化情况

Fig. 7 SER， EAR varying with time

图8 YawDD数据集上瞌睡识别可视化示例

Fig. 8 Visualization example of drowsiness recognition on YawDD dataset

图9 YawDD数据集瞌睡检测部分结果

Fig. 9 Some results of drowsiness detection on YawDD dataset

图10 NTHU-DDD数据集上的瞌睡识别可视化示例

Fig. 10 Visualization example of drowsiness recognition on NTHU-DDD dataset

表1 本文算法在YawDD上的实验结果

Tab. 1 Experimental results of proposed algorithm on YawDD

视频分类	总数	正确预测数
女性不戴眼镜	8	7
女性戴眼镜	5	5
男性不戴眼镜	6	6
男性戴眼镜	10	9

图11 不同算法的特征值对比

Fig. 11 Eigenvalue comparison of different algorithms

图12 不同算法的准确率对比

Fig. 12 Accuracy comparison of different algorithms

表2 NTHU-DDD数据集上的实验结果 (%)

Tab. 2 Experimental results on NTHU-DDD dataset

场景	F1	准确率	平均值
平均	85.25	81.5	81.87
不戴眼镜	95.00	93.0	94.00
戴眼镜	81.00	79.0	80.00
晚上不戴眼镜	87.00	82.0	84.50
晚上戴眼镜	78.00	72.0	75.00

图13 不同算法的耗时箱型图对比

Fig. 13 Comparison of time consumption box plot of different algorithms

参考文献 17

1	ANDERSSON C， KROISANDT G. Work-in-progress： application of cooperative learning in e-learning-based laboratory classes ［C］// Proceeding of the 2019 IEEE World Conference on Engineering Education. Piscataway： IEEE， 2019： 1-4. 10.1109/edunine.2019.8875772
2	YANG H， LIU L， MIN W D， et al. Driver yawning detection based on subtle facial action recognition ［J］. IEEE Transactions on Multimedia， 2020， 23： 572-583. 10.1109/tmm.2020.2985536
3	HAMID M， SAMUEL S， BOROWSKY A， et al. Evaluation of training interventions to mitigate effects of fatigue and sleepiness on driving performance ［J］. Transportation Research Record， 2016， 2584（1）：30-38. 10.3141/2584-05
4	何春.一种基于积分投影的人眼定位算法［J］.智能计算机与应用，2016，6（6）：90-91，93. 10.3969/j.issn.2095-2163.2016.06.026
	HE C. An algorithm of eye location based on integral projection ［J］. Intelligent Computers and Applications， 2016， 6（6）： 90-91， 93. 10.3969/j.issn.2095-2163.2016.06.026
5	周立宇，常侃.基于边缘检测和灰度投影的人眼定位［J］.广西科学，2017，24（3）：242-246. 10.13656/j.cnki.gxkx.20170525.001
	ZHOU L Y， CHANG K. Eye localization method based on edge detection and gray projection ［J］. Guangxi Sciences， 2017， 24（3）： 242-246. 10.13656/j.cnki.gxkx.20170525.001
6	侯向丹，赵丹，刘洪普，等.基于积分投影和差分投影的人眼定位［J］.计算机工程与科学，2017，39（3）：534-539. 10.3969/j.issn.1007-130X.2017.03.019
	HOU X D， ZHAO D， LIU H P， et al. Eyes localization based on integral projection and difference projection ［J］. Computer Engineering and Science， 2017， 39（3）： 534-539. 10.3969/j.issn.1007-130X.2017.03.019
7	SONOG F Y， TAN X Y， CHEN S C， et al. A literature survey on robust and efficient eye localization in real-life scenarios ［J］. Pattern Recognition， 2013， 46（12）： 3157-3173. 10.1016/j.patcog.2013.05.009
8	BARSOUM E， ZHANG C， FERER C C. Training deep networks for facial expression recognition with crowd-sourced label distribution ［C］// Proceeding of the 2016 18th ACM International Conference on Multimodal Interaction. New York： ACM， 2016：279-283. 10.1145/2993148.2993165
9	JACOB J E， NAIR G K， CHERIAN A， et al. Application of fractal dimension for EEG based diagnosis of encephalopathy ［J］. Analog Integrated Circuits and Signal Processing， 2019， 100（2）： 429-436. 10.1007/s10470-019-01388-z
10	贾项南，于凤芹，杨慧中，等.改进级联回归模型的人脸特征点定位［J］.传感器与微系统，2018，37（4）：58-61.
	JIA X N， YU F Q， YANG H Z， et al. Improved cascade regression model for face feature point alignment ［J］. Transducer and Microsystem Technologies， 2018， 37（4）： 58-61.
11	ABTAHI S， OMIDYEGANEH M， SHIRMOHAMMADI S， et al. YawDD： a yawning detection dataset ［C］// Proceedings of the 2014 5th ACM Multimedia Systems Conference. New York： ACM， 2014：24-28. 10.1145/2557642.2563678
12	WENG C H， LAI Y H， LAI S H. Driver drowsiness detection via a hierarchical temporal deep belief network ［C］// Proceeding of the 2016 Asian Conference on Computer Vision， LNCS10118. Cham： Springer， 2016： 117-133.
13	YU J， PARK S， LEE S， et al. Driver drowsiness detection using condition-adaptive representation learning framework ［J］. IEEE Transactions on Intelligent Transportation Systems， 2019， 20（11）： 4206-4218. 10.1109/tits.2018.2883823
14	SOUKUPOVÁ T， ČECH J. Real-time eye blink detection using facial landmarks ［EB/OL］. ［2020-11-10］. .
15	RAMOS A L A， ERANDIO J C， MANGILAYA D H T， et al. Driver drowsiness detection based on eye movement and yawning using facial landmark analysis ［J］. International Journal of Simulation： Systems， Science and Technology， 2019， 20（S2）： Article No.37. 10.5013/IJSSST.a.20.S2.37
16	PANICKER A D， NAIR M S. Open-eye detection using iris —sclera pattern analysis for driver drowsiness detection ［J］. Sādhanā， 2017， 42（11）： 1835-1849. 10.1007/s12046-017-0728-3
17	BONTEANU P， BOZOMITU R G， CRACAN A， et al. A high detection rate pupil detection algorithm based on contour circularity evaluation ［C］// Proceeding of the 2019 IEEE 25th International Symposium for Design and Technology in Electronic Packaging. Piscataway： IEEE， 2019： 264-267. 10.1109/siitme47687.2019.8990688

[1]	罗元, 云明静, 王艺, 赵立明. 基于人眼信息特征的人体疲劳检测[J]. 计算机应用, 2019, 39(7): 2098-2102.
[2]	谭阳唐德权唐钊轶. 基于人像图像的随机序列发生器[J]. 计算机应用, 2012, 32(06): 1623-1626.

基于人眼状态的瞌睡识别算法

Drowsiness recognition algorithm based on human eye state

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 15

参考文献 17

相关文章 2

编辑推荐

Metrics