Real-time facial expression recognition based on convolutional neural network with multi-scale kernel feature

doi:10.11772/j.issn.1001-9081.2019030540

Journal of Computer Applications ›› 2019, Vol. 39 ›› Issue (9): 2568-2574.DOI: 10.11772/j.issn.1001-9081.2019030540

• Artificial intelligence • Previous Articles Next Articles

Real-time facial expression recognition based on convolutional neural network with multi-scale kernel feature

LI Minze1, LI Xiaoxia1,2, WANG Xueyuan1,2, SUN Wei1

1. School of Information Engineering, Southwest University of Science and Technology, Mianyang Sichuan 621010, China;
2. Key Laboratory of Special Environmental Robotics in Sichuan Province(Southwest University of Science and Technology), Mianyang Sichuan 621010, China

Received:2019-04-03 Revised:2019-06-07 Online:2019-06-10 Published:2019-09-10
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61771411), the Sichuan Science and Technology Project (2019YJ0449), the Graduate Innovation Fund of Southwest University of Science and Technology (18ycx123).

基于多尺度核特征卷积神经网络的实时人脸表情识别

李旻择¹, 李小霞^1,2, 王学渊^1,2, 孙维¹

1. 西南科技大学信息工程学院, 四川绵阳 621010;
2. 特殊环境机器人技术四川省重点实验室(西南科技大学), 四川绵阳 621010

通讯作者: 李小霞
作者简介:李旻择(1992-),男,四川南充人,硕士研究生,CCF会员,主要研究方向:深度学习、计算机视觉;李小霞(1976-),女,四川安岳人,教授,博士,主要研究方向:模式识别、计算机视觉;王学渊(1974-),男,四川绵阳人,副教授,博士,主要研究方向:图像处理;孙维(1995-),男,四川达州人,硕士研究生,主要研究方向:图像处理。
基金资助:
国家自然科学基金资助项目（61771411）；四川省科技计划项目（2019YJ0449）；西南科技大学研究生创新基金资助项目（18ycx123）。

Abstract

Abstract:

Aiming at the problems of insufficient generalization ability, poor stability and difficulty in meeting the real-time requirement of facial expression recognition, a real-time facial expression recognition method based on multi-scale kernel feature convolutional neural network was proposed. Firstly, an improved MSSD (MobileNet+Single Shot multiBox Detector) lightweight face detection network was proposed, and the detected face coordinates information was tracked by Kernel Correlation Filter (KCF) model to improve the detection speed and stability. Then, three linear bottlenecks of three different scale convolution kernels were used to form three branches. The multi-scale kernel convolution unit was formed by the feature fusion of channel combination, and the diversity feature was used to improve the accuracy of expression recognition. Finally, in order to improve the generalization ability of the model and prevent over-fitting, different linear transformation methods were used for data enhancement to augment the dataset, and the model trained on the FER-2013 facial expression dataset was migrated to the small sample CK+ dataset for retraining. The experimental results show that the recognition rate of the proposed method on the FER-2013 dataset reaches 73.0%, which is 1.8% higher than that of the Kaggle Expression Recognition Challenge champion, and the recognition rate of the proposed method on the CK+ dataset reaches 99.5%. For 640×480 video, the face detection speed of the proposed method reaches 158 frames per second, which is 6.3 times of that of the mainstream face detection network MTCNN (MultiTask Cascaded Convolutional Neural Network). At the same time, the overall speed of face detection and expression recognition of the proposed method reaches 78 frames per second. It can be seen that the proposed method can achieve fast and accurate facial expression recognition.

Key words: Facial Expression Recognition (FER), Convolutional Neural Network (CNN), face detection, Kernel Correlation Filter (KCF), transfer learning

摘要：

针对人脸表情识别的泛化能力不足、稳定性差以及速度慢难以满足实时性要求的问题，提出了一种基于多尺度核特征卷积神经网络的实时人脸表情识别方法。首先，提出改进的MobileNet结合单发多盒检测器（MSSD）轻量化人脸检测网络，并利用核相关滤波（KCF）模型对检测到的人脸坐标信息进行跟踪来提高检测速度和稳定性；然后，使用三种不同尺度卷积核的线性瓶颈层构成三条支路，用通道合并的特征融合方式形成多尺度核卷积单元，利用其多样性特征来提高表情识别的精度；最后，为了提升模型泛化能力和防止过拟合，采用不同的线性变换方式进行数据增强来扩充数据集，并将FER-2013人脸表情数据集上训练得到的模型迁移到小样本CK+数据集上进行再训练。实验结果表明，所提方法在FER-2013数据集上的识别率达到73.0%，较Kaggle表情识别挑战赛冠军提高了1.8%，在CK+数据集上的识别率高达99.5%。对于640×480的视频，人脸检测速度达到每秒158帧，是主流人脸检测网络多任务级联卷积神经网络（MTCNN）的6.3倍，同时人脸检测和表情识别整体速度达到每秒78帧。因此所提方法能够实现快速精确的人脸表情识别。

关键词: 人脸表情识别, 卷积神经网络, 人脸检测, 核相关滤波, 迁移学习

CLC Number:

TP391.4

LI Minze, LI Xiaoxia, WANG Xueyuan, SUN Wei. Real-time facial expression recognition based on convolutional neural network with multi-scale kernel feature[J]. Journal of Computer Applications, 2019, 39(9): 2568-2574.

李旻择, 李小霞, 王学渊, 孙维. 基于多尺度核特征卷积神经网络的实时人脸表情识别[J]. 计算机应用, 2019, 39(9): 2568-2574.

References

[1] EKMAN P. Contacts across cultures in the face and emotion[J]. Journal of Personality and Social Psychology, 1971, 17(2):124-129.
[2] ZHAO X, ZHANG S. Facial expression recognition based on local binary patterns and kernel discriminant isomap[J]. Sensors, 2011, 11(10):9573-9588.
[3] KUMAR P, HAPPY S L, ROUTRAY A. A real-time robust facial expression recognition system using HOG features[C]//CAST 2016:Proceedings of the 2016 International Conference on Computing, Analytics and Security Trends. Piscataway, NJ:IEEE, 2016:289-293.
[4] 刘帅师,田彦涛,万川.基于Gabor多方向特征融合与分块直方图的人脸表情识别方法[J]. 自动化学报,2011,37(12):1455-1463.(LIU S S, TIAN Y T, WAN C. Facial expression recognition method based on gabor multi-orientation features fusion and block histogram[J]. Acta Automatica Sinica, 2011, 37(12):1455-1463.)
[5] BERRETTI S, del BIMBO A, PALA P, et al. A set of selected SIFT features for 3D facial expression recognition[C]//ICPR 2010:Proceedings of the 2010 20th International Conference on Pattern Recognition. Piscataway, NJ:IEEE, 2010:4125-4128.
[6] CHEON Y, KIM D. Natural facial expression recognition using differential-AAM and manifold learning[J]. Pattern Recognition, 2009, 42(7):1340-1350.
[7] 尹星云,王洵,董兰芳,等.用隐马尔可夫模型设计人脸表情识别系统[J].电子科技大学学报,2003, 32(6):725-728.(YIN X Y, WANG X, DONG L F, et al. Design of recognition for facial expression by hidden markov model[J]. Journal of University of Electronic Science and Technology of China, 2003, 32(6):725-728.)
[8] VAPNIK V N, LERNER A Y. Recognition of patterns with help of generalized portraits[J]. Avtomatika I Telemekhanika, 1963, 24(6):774-780.
[9] ROWEIS S T. Nonlinear dimensionality reduction by locally linear embedding[J]. Science, 2000, 290(5500):2323-2326.
[10] HART P E. The condensed nearest neighbor rule[J]. IEEE Transactions on Information Theory, 1968, 14(3):515-516.
[11] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//NIPS'12:Proceedings of the 25th International Conference on Neural Information Processing Systems. North Miami Beach, FL, USA:Curran Associates, 2012:1097-1105.
[12] LYONS M J, AKAMATSU S, KAMACHI M G, et al. Coding facial expressions with Gabor wavelets[C]//AFGR 1998:Proceedings of the 3rd IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 1998:200-205.
[13] LUCEY P, COHN J F, KANADE T, et al. The extended Cohn-Kanade dataset (CK+):a complete dataset for action unit and emotion-specified expression[C]//CVPRW 2010:Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2010:94-101.
[14] GOODFELLOW I J, ERHAN D, CARRIER P L, et al. Challenges in representation learning:a report on three machine learning contests[J]. Neural Networks, 2013, 64:59-63.
[15] DHALL A, GOECKE R, LUCEY S, et al. Static facial expression analysis in tough conditions:data, evaluation protocol and benchmark[C]//ICCVW 2011:Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops. Piscataway, NJ:IEEE, 2011:2106-2112.
[16] TANG Y. Deep learning using linear support vector machines[EB/OL].[2018-12-21]. https://arxiv.org/pdf/1306.0239.pdf.
[17] AL-SHABI M, CHEAH W P, CONNIE T. Facial expression recognition using a hybrid CNN-SIFT aggregator[EB/OL].[2018-08-17]. https://arxiv.org/ftp/arxiv/papers/1608/1608.02833.pdf.
[18] FANG H, PARTHALÁIN N M, AUBREY A J, et al. Facial expression recognition in dynamic sequences:an integrated approach[J]. Pattern Recognition, 2014, 47(3):1271-1281.
[19] JEON J, PARK J-C, JO Y J, et al. A real-time facial expression recognizer using deep neural network[C]//IMCOM'16:Proceedings of the 10th International Conference on Ubiquitous Information Management and Communication. New York:ACM, 2016:Article No. 94.
[20] NEHAL O, NOHA A, FAYEZ W. Intelligent real-time facial expression recognition from video sequences based on hybrid feature tracking algorithms[J]. International Journal of Advanced Computer Science and Applications, 2017, 8(1):245-260.
[21] LIU W, ANGUELOV D, ERHAN D, et al. SSD:single shot multibox detector[C]//Proceedings of the 2016 European Conference on Computer Vision, LNCS 9905. Berlin:Springer, 2016:21-37.
[22] HENRIQUES J F, CASEIRO R, MARTINS, et al. High-speed tracking with kernelized correlation filters[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(3):583-596.
[23] SIMONYAN K, ZISSERMAN A. Very deep convolutional networks for large-scale image recognition[EB/OL].[2019-01-10]. https://arxiv.org/pdf/1409.1556.pdf.
[24] HOWARD A G, ZHU M, CHEN B. et al. MobileNets:efficient convolutional neural networks for mobile vision applications[EB/OL].[2018-12-17]. https://arxiv.org/pdf/1704.04861.pdf.
[25] SANDLER M, HOWARD A, ZHU M, et al. Inverted residuals and linear bottlenecks:mobile networks for classification, detection and segmentation[EB/OL].[2018-12-16]. https://arxiv.org/pdf/1801.04381v2.pdf.
[26] HE K, ZHANG X, REN S, et al. Delving deep into rectifiers:surpassing human-level performance on ImageNet classification[EB/OL].[2018-12-06]. https://arxiv.org/pdf/1502.01852.pdf.
[27] JARRETT K, KAVUKCUOGLU K, RANZATO M, et al. What is the best multi-stage architecture for object recognition?[C]//ICCV 2009:Proceedings of the IEEE 12th International Conference on Computer Vision. Piscataway, NJ:IEEE, 2009:2146-2153.
[28] LIEW S S, KHALIL-HANI M, BAKHTERI R. Bounded activation functions for enhanced training stability of deep neural networks on visual pattern recognition problems[J]. Neurocomputing, 2016, 216(C):718-734.
[29] DJORK-ARNÉ C, UNTERTHINER T, HOCHREITER S. Fast and accurate deep network learning by Exponential Linear Units (ELUs)[EB/OL].[2019-01-22]. https://arxiv.org/pdf/1511.07289.pdf.
[30] YANG S, LUO P, LOY C C, et al. WIDER FACE:a face detection benchmark[C]//CVPR 2016:Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2016:5525-5533.
[31] DENG J, DONG W, SOCHER R, et al. ImageNet:a large-scale hierarchical image database[C]//CVPR 2009:Proceedings of the 2009 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2009:248-255.
[32] YANG S, LUO P, LOY C C, et al. From facial parts responses to face detection:a deep learning approach[C]//ICCV 2015:Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 2015:3676-3684.
[33] ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10):1499-1503.
[34] SZEGEDY C, IOFFE S, VANHOUCKE V, et al. Inception-v4, Inception-ResNet and the impact of residual connections on learning[C]//AAAI 2017:Proceedings of the 31st AAAI Conference on Artificial Intelligence. Menlo Park, CA:AAAI Press, 2017:23-38.
[35] GUO Y, TAO D, YU J, et al. Deep neural networks with relativity learning for facial expression recognition[C]//ICMEW 2016:Proceedings of the 2016 IEEE International Conference on Multimedia and Expo Workshops. Piscataway, NJ:IEEE, 2016:1-6.
[36] YAN J, ZHENG W, CUI Z, et al. A joint convolutional bidirectional LSTM framework for facial expression recognition[J]. IEICE Transactions on Information and Systems, 2018, 101(4):1217-1220.
[37] FERNANDEZ P D M, PEÑA F A G, REN T I, et al. FERAtt:facial expression recognition with attention net[EB/OL].[2019-02-08]. https://arxiv.org/pdf/1902.03284.pdf.
[38] SONG X, BAO H. Facial expression recognition based on video[C]//AIPR 2017:Proceedings of the 2016 IEEE Applied Imagery Pattern Recognition Workshop. Washington, DC:IEEE Computer Society, 2016, 1:1-5.
[39] ZHANG K, HUANG Y, DU Y, et al. Facial expression recognition based on deep evolutional spatial-temporal networks[J]. IEEE Transactions on Image Processing, 2017, 26(9):4193-4203.

Real-time facial expression recognition based on convolutional neural network with multi-scale kernel feature

基于多尺度核特征卷积神经网络的实时人脸表情识别

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	Yan RONG, Jiawen LIU, Xinlei LI. Adaptive hybrid network for affective computing in student classroom [J]. Journal of Computer Applications, 2024, 44(9): 2919-2930.
[2]	Yun LI, Fuyou WANG, Peiguang JING, Su WANG, Ao XIAO. Uncertainty-based frame associated short video event detection method [J]. Journal of Computer Applications, 2024, 44(9): 2903-2910.
[3]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[4]	Hong CHEN, Bing QI, Haibo JIN, Cong WU, Li’ang ZHANG. Class-imbalanced traffic abnormal detection based on 1D-CNN and BiGRU [J]. Journal of Computer Applications, 2024, 44(8): 2493-2499.
[5]	Dongwei WANG, Baichen LIU, Zhi HAN, Yanmei WANG, Yandong TANG. Deep network compression method based on low-rank decomposition and vector quantization [J]. Journal of Computer Applications, 2024, 44(7): 1987-1994.
[6]	Yangyi GAO, Tao LEI, Xiaogang DU, Suiyong LI, Yingbo WANG, Chongdan MIN. Crowd counting and locating method based on pixel distance map and four-dimensional dynamic convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2233-2242.
[7]	Mengyuan HUANG, Kan CHANG, Mingyang LING, Xinjie WEI, Tuanfa QIN. Progressive enhancement algorithm for low-light images based on layer guidance [J]. Journal of Computer Applications, 2024, 44(6): 1911-1919.
[8]	Jianjing LI, Guanfeng LI, Feizhou QIN, Weijun LI. Multi-relation approximate reasoning model based on uncertain knowledge graph embedding [J]. Journal of Computer Applications, 2024, 44(6): 1751-1759.
[9]	Feiyu ZHAI, Handa MA. Hybrid classical-quantum classification model based on DenseNet [J]. Journal of Computer Applications, 2024, 44(6): 1905-1910.
[10]	Wenshuo GAO, Xiaoyun CHEN. Point cloud classification network based on node structure [J]. Journal of Computer Applications, 2024, 44(5): 1471-1478.
[11]	Hongtian LI, Xinhao SHI, Weiguo PAN, Cheng XU, Bingxin XU, Jiazheng YUAN. Few-shot object detection via fusing multi-scale and attention mechanism [J]. Journal of Computer Applications, 2024, 44(5): 1437-1444.
[12]	Min SUN, Qian CHENG, Xining DING. CBAM-CGRU-SVM based malware detection method for Android [J]. Journal of Computer Applications, 2024, 44(5): 1539-1545.
[13]	Wangjun SHI, Jing WANG, Xiaojun NING, Youfang LIN. Sleep stage classification model by meta transfer learning in few-shot scenarios [J]. Journal of Computer Applications, 2024, 44(5): 1445-1451.
[14]	Tianhua CHEN, Jiaxuan ZHU, Jie YIN. Bird recognition algorithm based on attention mechanism [J]. Journal of Computer Applications, 2024, 44(4): 1114-1120.
[15]	Lijun XU, Hui LI, Zuyang LIU, Kansong CHEN, Weixuan MA. 3D-GA-Unet： MRI image segmentation algorithm for glioma based on 3D-Ghost CNN [J]. Journal of Computer Applications, 2024, 44(4): 1294-1302.