Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine

doi:10.11772/j.issn.1001-9081.2021071270

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (4): 1253-1259.DOI: 10.11772/j.issn.1001-9081.2021071270

• The 36 CCF National Conference of Computer Applications (CCF NCCA 2020) • Previous Articles

Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine

Guifang QIAO¹, Shouming HOU¹(), Yanyan LIU²

^1.College of Computer Science and Technology，Henan Polytechnic University，Jiaozuo Henan 454003，China
^2.Alibaba Business School，Hangzhou Normal University，Hangzhou Zhejiang 311121，China

Received:2021-07-06 Revised:2021-08-22 Accepted:2021-08-31 Online:2022-04-15 Published:2022-04-10
Contact: Shouming HOU
About author:QIAO Guifang， born in 1995， M. S. candidate. Her research interests include graphics and image processing.
LIU Yanyan， born in 1990， M. S. .Her research interests include computer vision， image processing.
Supported by:
National Key Research and Development Project of China(2018YFB1004900);Science and Technology Project of Henan Province(172102210273)

基于改进卷积神经网络与支持向量机结合的面部表情识别算法

乔桂芳¹, 侯守明¹(), 刘彦彦²

^1.河南理工大学计算机科学与技术学院，河南焦作 454003
^2.杭州师范大学阿里巴巴商学院，杭州 311121

通讯作者: 侯守明
作者简介:乔桂芳（1995—），女，河南开封人，硕士研究生，主要研究方向：图形图像处理
刘彦彦（1990—），女，河南鹤壁人，硕士，主要研究方向：计算机视觉、图像处理。
基金资助:
国家重点研发计划项目(2018YFB1004900);河南省科技攻关计划项目(172102210273)

Abstract

Abstract:

In view of the problems of the current Convolutional Neural Network （CNN） using end layer features to recognize facial expression， such as complex model structure， too many parameters and unsatisfactory recognition， an optimization algorithm based on the combination of improved CNN and Support Vector Machine （SVM） was proposed. First， the network model was designed by the idea of continuous convolution to obtain more nonlinear activations. Then， the adaptive Global Average Pooling （GAP） layer was used to replace the fully connected layer in traditional CNN to reduce the network parameters. Finally， in order to improve generalization ability of the model， SVM classifier instead of the traditional Softmax function was used to realize expression recognition. Experimental results show that the proposed algorithm achieves 73.4% and 98.06% recognition accuracy on Fer2013 and CK+ datasets， which is 2.2 percentage points higher than the traditional LeNet-5 algorithm on Fer2013 dataset. Moreover， this network model has simple structure， less parameters and good robustness.

Key words: Convolutional Neural Network (CNN), small size convolution kernel, expression recognition, Global Average Pooling (GAP), nonlinear Support Vector Machine (SVM)

摘要：

针对当前卷积神经网络（CNN）利用端层特征进行面部表情识别存在模型结构繁琐、训练参数过多、识别不够理想的问题，提出一种基于改进CNN与支持向量机（SVM）相结合的优化算法。首先，利用连续卷积的思想设计网络模型，以获取更多非线性激活；然后，采用自适应全局平均池化（GAP）层取代传统CNN中的全连接层，以减少网络参数量；最后，用SVM分类器代替传统Softmax函数实现表情识别，以提高模型泛化能力。实验结果表明，所提算法在Fer2013和CK+数据集上分别取得了73.4%和98.06%的识别准确率，与传统LeNet-5算法相比，在Fer2013数据集上提升了2.2个百分点，且该网络模型结构简单、参数量较少，具有良好的鲁棒性。

关键词: 卷积神经网络, 小尺寸卷积核, 表情识别, 全局平均池化, 非线性支持向量机

CLC Number:

TP391.4

Guifang QIAO, Shouming HOU, Yanyan LIU. Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine[J]. Journal of Computer Applications, 2022, 42(4): 1253-1259.

乔桂芳, 侯守明, 刘彦彦. 基于改进卷积神经网络与支持向量机结合的面部表情识别算法[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1253-1259.

Figures/Tables 15

Fig. 1 Basic structure of CNN

Fig. 2 SVM multi-classification algorithm process

Fig. 3 Flow of facial expression recognition algorithm based on improved CNN+SVM

Fig. 4 Overall improvement strategy of proposed algorithm

Fig. 5 Structure of facial expression recognition model based on improved CNN+SVM algorithm

Tab. 1 Parameter description of each layer of facial expression recognition model based on improved CNN+SVM algorithm

模型结构	层类型	输入尺寸	卷积核	步长	填充	输出尺寸
	Input	48×48×1				48×48×1
ConvBlock_1	Convolution1_1	48×48×1	3×3	1	Same	48×48×32
	Convolution1_2	48×48×32	3×3	1	Same	48×48×32
	Max pooling	48×48×32	2×2	2		24×24×32
ConvBlock_2	Convolution2_1	24×24×32	3×3	1	Same	24×24×64
	Convolution2_2	24×24×64	3×3	1	Same	24×24×64
	Maxpooling	24×24×64	2×2	2		12×12×64
ConvBlock_3	Convolution3_1	12×12×64	3×3	1	Same	12×12×128
	Convolution3_2	12×12×128	3×3	1	Same	12×12×128
	Maxpooling	12×12×128	2×2	2		6×6×128
全局平均池化	GAP	6×6×128				1×1×128
分类判别	SVM	1×1×128				1×7

Tab. 2 Chinese and English labels and numbers of different categories in Fer2013 dataset

标签	英文表示	中文表示	样本数
标签	英文表示	中文表示	训练集	测试集
0	angry	生气	3 196	799
1	disgust	厌恶	348	88
2	fear	恐惧	3 277	820
3	happy	高兴	5 772	1 443
4	sad	悲伤	3 864	966
5	surprised	惊讶	2 536	635
6	normal	中性	3 972	993

Tab. 3 Number of each expression category in CK+ dataset

表情类别	样本数
表情类别	训练集	测试集	总数量
愤怒	108	27	135
蔑视	42	12	54
厌恶	142	35	177
恐惧	60	15	75
高兴	166	41	207
悲伤	67	17	84
惊讶	199	50	249

Fig. 6 Sample diagrams of 7 categories of facial expression in Fer2013 and CK+ datasets

Fig. 7 Comparison before and after facial image data augmentation in CK+ dataset

Tab. 4 Model training parameter description

参数	值
批量（Batch_size）	24
迭代次数（Epochs）	200
随机失活（Dropout）	0.2
学习率（Lr）	0.001
表情类别（Nc）	7

Fig. 8 Training process curves on Fer2013 and CK+ datasets

Fig. 9 Confusion matrix of expression category generated by each dataset

Fig. 10 Comparison of recognition accuracy between traditional model and the improved model on Fer2013 and CK+ datasets

Fig. 11 Comparison of recognition effects of different methods on Fer2013 and CK+ datasets

References 26

1	MEHRABIAN A， FERRIS S R. Inference of attitudes from nonverbal communication in two channels［J］. Journal of Consulting Psychology， 1967， 31（3）：248-252. 10.1037/h0024648
2	李飞. 基于深度学习的面部表情识别技术研究［D］. 大连：大连海事大学， 2020：1-2.
	LI F. Facial expression recognition technology based on deep learning［D］. Dalian： Dalian Maritime University， 2020：1-2.
3	ZHANG B R， LIU G Y， XIE G Q. Facial expression recognition using LBP and LPQ based on Gabor wavelet transform［C］// Proceedings of the 2nd IEEE International Conference on Computer and Communications. Piscataway： IEEE， 2016：365-369. 10.1109/compcomm.2016.7924724
4	XU F， WANG Z. A facial expression recognition method based on cubic spline interpolation and HOG features［C］// Proceedings of the 2017 IEEE International Conference on Robotics and Biomimetics. Piscataway： IEEE， 2017：1300-1305. 10.1109/robio.2017.8324597
5	SHIN M， KIM M， KWON D S. Baseline CNN structure analysis for facial expression recognition［C］// Proceedings of the 25th IEEE International Symposium on Robot and Human Interactive Communication. Piscataway： IEEE， 2016：724-729. 10.1109/roman.2016.7745199
6	LeCUN Y， BOTTOU L， BENGIO Y， et al. Gradient-based learning applied to document recognition［J］. Proceedings of the IEEE， 1998， 86（11）： 2278-2324. 10.1109/5.726791
7	KRIZHEVSKY A， SUTSKEVER I， HINTON G E. ImageNet classification with deep convolutional neural networks［C］// Proceedings of the 25th International Conference on Neural Information Processing Systems. Red Hook， NY： Curran Associates Inc.， 2012：1097-1105.
8	SIMONYAN K， ZISSERMAN. A very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2021-05-20］.. 10.1109/cvpr.2014.219
9	LI Y， ZENG J B， SHAN S G， et al. Occlusion aware facial expression recognition using CNN with attention mechanism［J］. IEEE Transactions on Image Processing， 2019， 28（5）：2439-2450. 10.1109/tip.2018.2886767
10	XIE S Y， HU H F， WU Y B. Deep multi-path convolutional neural network joint with salient region attention for facial expression recognition［J］. Pattern Recognition， 2019， 92：177-191. 10.1016/j.patcog.2019.03.019
11	XIA B， WANG S F. Occluded facial expression recognition with step-wise assistance from unpaired non-occluded images［C］// Proceedings of the 28th ACM International Conference on Multimedia. New York： ACM， 2020：2927-2935. 10.1145/3394171.3413773
12	王忠民，李和娜，张荣，等. 融合卷积神经网络与支持向量机的表情识别［J］. 计算机工程与设计， 2019， 40（12）：3594-3600.
	WANG Z M， LI H N， ZHANG R， et al. Fusing convolutional neural network and support vector machine for expression recognition ［J］. Computer Engineering and Design， 2019， 40（12）：3594-3600.
13	LIN M， CHEN Q， YAN S C. Network in network［EB/OL］. （2014-03-04）［2021-05-20］.. 10.1109/icicta.2014.118
14	CORTES C， VAPNIK V. Support-vector networks［J］. Machine Learning， 1995， 20（3）：273-297. 10.1007/bf00994018
15	SUN W J， ZHAO R， YAN R Q， et al. Convolutional discriminative feature learning for induction motor fault diagnosis［J］. IEEE Transactions on Industrial Informatics， 2017， 13（3）： 1350-1359. 10.1109/tii.2017.2672988
16	GOODFELLOW I J， ERHAN D， CARRIER P L， et al. Challenges in representation learning： a report on three machine learning contests［J］. Neural Networks， 2015， 64：59-63. 10.1016/j.neunet.2014.09.005
17	LUCEY P， COHN J F， KANADE T， et al. The extended Cohn-Kanade dataset （CK+）： a complete dataset for action unit and emotion specified expression［C］// Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition - Workshops. Piscataway： IEEE， 2010：94-101. 10.1109/cvprw.2010.5543262
18	尹鹏博，潘伟民，张海军. 基于卷积注意力的轻量级人脸表情识别方法［J］. 激光与光电子学进展， 2021， 58（12）：261-267.
	YIN P B， PAN W M， ZHANG H J. Lightweight facial expression recognition method based on convolutional attention［J］. Laser & Optoelectronics Progress， 2021， 58（12）：261-267.
19	魏赟，李栋. 结合改进卷积神经网络与自编码器的表情识别［J/OL］. 小型微型计算机系统. （2021-03-22）［2021-04-25］.，. 10.1142/s0219265921410036
	LI D. Expression Recognition based on improved convolutional neural networks and autoencoder［J/OL］. Journal of Chinese Computer Systems. （2021-03-22）［2021-04-25］.. 10.1142/s0219265921410036
20	李旻择，李小霞，王学渊，等. 基于多尺度核特征卷积神经网络的实时人脸表情识别［J］. 计算机应用， 2019， 39（9）：2568-2574. 10.11772/j.issn.1001-9081.2019030540
	LI M Z， LI X X， WANG X Y， et al. Real-time facial expression recognition based on convolutional neural network with multi-scale kernel feature［J］. Journal of Computer Applications， 2019， 39（9）：2568-2574. 10.11772/j.issn.1001-9081.2019030540
21	ZHOU J C， JIA X， SHEN L L， et al. Improved softmax loss for deep learning-based face and expression recognition［J］. Cognitive Computation and Systems， 2019， 1（4）：97-102. 10.1049/ccs.2019.0010
22	MIAO S， XU H Y， HAN Z Q， et al. Recognizing facial expressions using a shallow convolutional neural network［J］. IEEE Access， 2019， 7：78000-78011. 10.1109/access.2019.2921220
23	李宽. 基于浅层卷积网络的人脸表情识别方法研究［D］. 合肥：中国科学技术大学， 2019：16-18.
	LI K. Facial expression recognition based on shallow convolutional network［D］. Hefei： University of Science and Technology of China， 2019：16-18.
24	石翠萍，谭聪，左江，等. 基于改进AlexNet卷积神经网络的人脸表情识别［J］. 电讯技术， 2020， 60（9）：1005-1012. 10.3969/j.issn.1001-893x.2020.09.002
	SHI C P， TAN C， ZUO J， et al. Expression recognition based on improved AlexNet convolutional neural network［J］. Telecommunications Engineering， 2020， 60（9）：1005-1012. 10.3969/j.issn.1001-893x.2020.09.002
25	杨旭，尚振宏. 基于改进AlexNet的人脸表情识别［J］. 激光与光电子学进展， 2020， 57（14）：243-250. 10.3788/lop57.141026
	YANG X， SHANG Z H. Face expression recognition based on improved AlexNet［J］. Laser & Optoelectronics Progress， 2020， 57（14）：243-250. 10.3788/lop57.141026
26	RAVI R， YADHUKRISHNA S V， PRITHVIRAJ R. A face expression recognition using CNN & LBP［C］// Proceedings of the 4th International Conference on Computing Methodologies and Communication. Piscataway： IEEE， 2020：684-689. 10.1109/iccmc48092.2020.iccmc-000127

[1]	Zumin WANG, Zhihao ZHANG, Jing QIN, Changqing JI. Review of mechanical fault diagnosis technology based on convolutional neural network [J]. Journal of Computer Applications, 2022, 42(4): 1036-1043.
[2]	Changqing JI, Zhiyong GAO, Jing QIN, Zumin WANG. Review of image classification algorithms based on convolutional neural network [J]. Journal of Computer Applications, 2022, 42(4): 1044-1049.
[3]	Leping LIN, Hongmin ZHOU, Ning OUYANG. Compressed sensing image reconstruction method fusing spatial location and structure information [J]. Journal of Computer Applications, 2022, 42(3): 930-937.
[4]	Dejian WEI, Wenming WANG, Quanyu WANG, Haopan REN, Yanyan GAO, Zhi WANG. Improved 3D hand pose estimation network based on anchor [J]. Journal of Computer Applications, 2022, 42(3): 953-959.
[5]	Lu ZHANG, Chun FANG, Ming ZHU. Indoor fall detection algorithm based on Res2Net-YOLACT and fusion feature [J]. Journal of Computer Applications, 2022, 42(3): 757-763.
[6]	Dingkang YANG, Shuai HUANG, Shunli WANG, Peng ZHAI, Yidan LI, Lihua ZHANG. EE-GAN：facial expression recognition method based on generative adversarial network and network integration [J]. Journal of Computer Applications, 2022, 42(3): 750-756.
[7]	Wanying YU, Meiyu LIANG, Xiaoxiao WANG, Zheng CHEN, Xiaowen CAO. Student expression recognition and intelligent teaching evaluation in classroom teaching videos based on deep attention network [J]. Journal of Computer Applications, 2022, 42(3): 743-749.
[8]	Renzhi PAN, Fulan QIAN, Shu ZHAO, Yanping ZHANG. Recommendation model for user attribute preference modeling based on convolutional neural network interaction [J]. Journal of Computer Applications, 2022, 42(2): 404-411.
[9]	Wei LI, Yaochi FAN, Qiaoyong JIANG, Lei WANG, Qingzheng XU. Variable convolutional autoencoder method based on teaching-learning-based optimization for medical image classification [J]. Journal of Computer Applications, 2022, 42(2): 592-598.
[10]	Yinxin BAO, Yang CAO, Quan SHI. Improved spatio-temporal residual convolutional neural network for urban road network short-term traffic flow prediction [J]. Journal of Computer Applications, 2022, 42(1): 258-264.
[11]	Hengxin LI, Kan CHANG, Yufei TAN, Mingyang LING, Tuanfa QIN. Color image demosaicking network based on inter-channel correlation and enhanced information distillation [J]. Journal of Computer Applications, 2022, 42(1): 245-251.
[12]	Huiqing XU, Bin CHEN, Jingfei WANG, Zhiyi CHEN, Jian QIN. Elongated pavement distress detection method based on convolutional neural network [J]. Journal of Computer Applications, 2022, 42(1): 265-272.
[13]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[14]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[15]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.

Facial expression recognition algorithm based on combination of improved convolutional neural network and support vector machine

基于改进卷积神经网络与支持向量机结合的面部表情识别算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 15

References 26

Related Articles 15

Recommended Articles

Metrics