基于图像的面部表情识别方法综述

doi:10.11772/j.issn.1001-9081.2017.12.3509

计算机应用 ›› 2017, Vol. 37 ›› Issue (12): 3509-3516.DOI: 10.11772/j.issn.1001-9081.2017.12.3509

• 计算机视觉与虚拟现实 • 上一篇下一篇

基于图像的面部表情识别方法综述

徐琳琳, 张树美, 赵俊莉

青岛大学数据科学与软件工程学院, 山东青岛 266071

收稿日期:2017-06-26 修回日期:2017-09-18 出版日期:2017-12-10 发布日期:2017-12-18
通讯作者: 张树美
作者简介:徐琳琳(1992-),女,山东莱芜人,硕士研究生,主要研究方向:图像识别与处理、深度学习;张树美(1964-),女,山东莱西人,教授,博士,主要研究方向:时滞非线性系统的分析与控制、图像识别与处理;赵俊莉(1977-),山西新绛人,助理教授,博士,CCF会员,主要研究方向:计算机视觉、计算机图形学、虚拟现实。
基金资助:
国家自然科学基金资助项目（61702293，41506198）；虚拟现实应用教育部工程研究中心开放基金课题（MEOBNUEVRA201601）。

Summary of facial expression recognition methods based on image

XU Linlin, ZHANG Shumei, ZHAO Junli

School of Data Science and Software Engineering, Qingdao University, Qingdao Shandong 266071, China

Received:2017-06-26 Revised:2017-09-18 Online:2017-12-10 Published:2017-12-18
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61702293, 41506198), the Open Research Fund of the Ministry of Education for Engineering Research Center of Virtual Reality Application (MEOBNUEVRA201601).

摘要/Abstract

摘要： 近年来，面部表情识别在教育、医学、心理分析以及商业领域得到了广泛关注。针对目前表情识别方法不够系统、概念模糊的问题，对面部表情识别的步骤及其方法进行了综述探讨。首先，介绍了目前常用的人脸表情数据集，并回顾了面部表情识别的发展历程；然后，介绍了人脸表情识别的面部表情编码和面部表情识别过程这两个方面，归纳了人脸面部表情识别的四个过程，重点总结了特征提取和表情分类两个过程中的经典算法以及这些算法的基本原理和优劣比较；最后，指出了目前面部表情识别存在的问题和未来可能的发展趋势。

关键词: 表情识别, 表情数据集, 表情编码, 特征提取, 表情分类

Abstract: In recent years, facial expression recognition has received extensive attention in education, medicine, psychoanalysis and business. Aiming at the problems of not systematic enough and fuzzy concept of facial expression recognition method, the steps and methods of facial expression recognition were reviewed and discussed. Firstly, the commonly used facial expression databases were introduced and the development of facial expression recognition was reviewed. Then, two aspects of facial expression recognition were introduced, such as facial expression coding and facial expression recognition. The four processes of face facial expression recognition were summarized. The classical algorithms, the basic principles of these algorithms and the comparisons of their advantages and disadvantages were summarized emphatically in the two processes of feature extraction and facial expression classification. Finally, the existing problems and possible development trends in the future of the current facial expression recognition were pointed out.

Key words: expression recognition, expression database, expression coding, feature extraction, expression classification

中图分类号:

TP391.41

徐琳琳, 张树美, 赵俊莉. 基于图像的面部表情识别方法综述[J]. 计算机应用, 2017, 37(12): 3509-3516.

XU Linlin, ZHANG Shumei, ZHAO Junli. Summary of facial expression recognition methods based on image[J]. Journal of Computer Applications, 2017, 37(12): 3509-3516.

参考文献

[1] GIRARD J M, COHN J F, MAHOOR M H, et al. Nonverbal social withdrawal in depression:evidence from manual and automatic analyses[J]. Image and Vision Computing, 2014, 32(10):641-647.
[2] LUCEY P, COHN J F, MATTHEWS I, et al. Automatically detecting pain in video through facial action units[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B:Cybernetics, 2011, 41(3):664-674.
[3] IRANI R, NASROLLAHI K, SIMON M O, et al. Spatiotemporal analysis of RGB-D-T facial images for multimodal pain level recognition[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2015:88-95.
[4] EKMAN P. Universal and cultural differences in facial expression of emotion[J]. Nebraska Symposium on Motivation, 1972, 19:207-284.
[5] LYONS M, AKAMATSU S, KAMACHI M, et al. Coding facial expressions with Gabor wavelets[C]//Proceedings of the 1998 IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 1998:200-205.
[6] LUCEY P, COHN J F, KANADE T, et al. The extended Cohn-Kanade dataset (CK+):a complete dataset for action unit and emotion-specified expression[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2010:94-101.
[7] VALSTAR M F, PANTIC M. Induced disgust, happiness and surprise:an addition to the mmi facial expression database[EB/OL].[2017-04-16]. https://ibug.doc.ic.ac.uk/media/uploads/documents/EMOTION-2010-ValstarPantic-CAMERA.pdf.
[8] 薛雨丽,毛峡,张帆.BHU人脸表情数据库的设计与实现[J].北京航空航天大学学报,2007,33(2):224-228.(XUE Y L, MAO X, ZHANG F. Design and realization of BHU facial expression database[J]. Journal of Beijing University of Aeronautics and Astronautics, 2007, 33(2):224-228.)
[9] WANG S F, LIU Z L, LYU S L, et al. A natural visible and infrared facial expression database for expression recognition and emotion Inference[J]. IEEE Transactions on Multimedia, 2010, 12(7):682-691.
[10] DHALL A, GOECKE R, LUCEY S, et al. Collecting large, richly annotated facial-expression databases from movies[J]. IEEE Multimedia, 2012, 19(3):34-41.
[11] DHALL A, GOECKE R, LUCEY S, et al. Static facial expression analysis in tough conditions:data, evaluation protocol and benchmark[C]//Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops. Piscataway, NJ:IEEE, 2011:2106-2112.
[12] MASE K, PENTLAND A. Automatic lipreading by optical-flow analysis[J]. Systems & Computers in Japan, 2015, 22(6):67-76.
[13] TIAN Y L, KANADE T, COHN J F. Recognizing action units for facial expression analysis[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2001, 23(2):97-115.
[14] COHEN I, SEBE N, GARG A, et al. Facial expression recognition from video sequences:temporal and static modeling[J]. Computer Vision & Image Understanding, 2003, 91(1/2):160-187.
[15] ALEKSIC P S, KATSAGGELOS A K. Automatic facial expression recognition using facial animation parameters and multi-stream hmms[J]. IEEE Transactions on Information Forensics & Security, 2006, 1(1):3-11.
[16] KOTSIA I, PITAS I. Facial expression recognition in image sequences using geometric deformation features and support vector machines[J]. IEEE Transactions on Image Processing, 2007, 16(1):172-187.
[17] LITTLEWORT G, BARTLETT M S, FASEL I, et al. Dynamics of facial expression extracted automatically from video[J]. Image & Vision Computing, 2006, 24(6):615-625.
[18] SHAN C F, GONG S G, MCOWAN P W. Facial expression recognition based on local binary patterns:a comprehensive study[J]. Image and Vision Computing, 2009, 27(6):803-816.
[19] YIN L J, WEI X Z, SUN Y, et al. A 3D facial expression database for facial behavior research[C]//Proceedings of the 20067th International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 2006:211-216.
[20] TANG H, HUANG T S. 3D facial expression recognition based on automatically selected features[C]//CVPRW 2008:Proceedings of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2008:1-8.
[21] VRETOS N, NIKOLAIDIS N, PITAS I. 3D facial expression recognition using Zernike moments on depth images[C]//Proceedings of the 201118th IEEE International Conference on Image Processing. Piscataway, NJ:IEEE, 2011:773-776.
[22] KONG S G, HEO J, ABIDI B R, et al. Recent advances in visual and infrared face recognition-a review[J]. Computer Vision & Image Understanding, 2005, 97(1):103-135.
[23] 王国胤,龚勋.人脸感知:从二维到三维[M].北京:科学出版社,2011:10-30.(WANG G Y, GONG X. Human Face Perception:from 2D to 3D[M]. Beijing:Science Press, 2011:10-30.)
[24] FREUND Y, SCHAPIRE R E. A decision-theoretic generalization of on-line learning and an application to boosting[C]//Proceedings of the 1995 European Conference on Computational Learning Theory, LNCS 904. Berlin:Springer, 1995:23-37.
[25] 王建.基于Gentle AdaBoost算法的人脸检测研究[D].成都:电子科技大学,2011:24-38.(WANG J. Research on face detection based on Gentle AdaBoost algorithm[D]. Chengdu:University of Electronic Science and Technology of China, 2011:24-38.)
[26] NAKAMURA M, NOMUYA H, UEHARA K. Improvement of boosting algorithm by modifying weighting rule[J]. Annals of Mathematics & Artificial Intelligence, 2004, 41(1):95-109.
[27] ZHANG Z Q, LI M J, LI S Z, et al. Multi-view face detection with FloatBoost[C]//Proceedings of the 2002 IEEE Workshop on Applications of Computer Vision. Washington, DC:IEEE Computer Society, 2002:184.
[28] LECUN Y, BOSER B, DENKER J S, et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(4):541-551.
[29] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.
[30] 汪济民.基于卷积神经网络的人脸检测和性别识别研究[D].南京:南京理工大学,2015:42-47.(WANG J M. Research on face detection and gender recognition based on convolution neural network[D]. Nanjing:Nanjing University of Science and Technology, 2015:42-47.)
[31] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks[C]//NIPS 2015:Proceedings of the 201528th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2015:2017-2025.
[32] ZHAO J B, MATHIEU M, GOROSHIN R, et al. Stacked what-where auto-encoders[EB/OL].[2017-04-10]. http://xueshu.baidu.com/s?wd=paperuri%3A%28c2e4cc91c19dc2d85fe0c72fcf535763%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Farxiv.org%2Fpdf%2F1506.02351.pdf&ie=utf-8&sc_us=5921033091152195876.
[33] 李辉,石波.基于卷积神经网络的人脸识别算法[J].软件导刊,2017,16(3):26-29.(LI H, SHI B. Face recognition algorithm based on convolution neural network[J]. Software Guide, 2017, 16(3):26-29.)
[34] COOTES T F, TAYLOR C J, COOPER D H, et al. Active shape models-their training and application[J]. Computer Vision & Image Understanding, 1995, 61(1):38-59.
[35] COOTES T F, EDWARDS G J, TAYLOR C J. Active appearance models[C]//Proceedings of the 1998 European Conference on Computer Vision, LNCS 1407. Berlin:Springer, 1998:484-498.
[36] 金鑫,谭晓阳.受限局部模型在人脸特征点定位中应用综述[J].小型微型计算机系统,2017,38(2):371-375.(JIN X, TAN X Y. Facial feature point detection with constrained local models:a survey[J]. Journal of Chinese Computer Systems, 2017, 38(2):371-375)
[37] SUN Y, WANG X G, TANG X O. Deep convolutional network cascade for facial point detection[C]//CVPR 2013:Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2013:3476-3483.
[38] CAO X D, WEI Y C, WEN F, et al. Face alignment by explicit shape regression[J]. International Journal of Computer Vision, 2014, 107(2):177-190.
[39] REN S Q, CAO X D, WEI Y C, et al. Face alignment at 3000 FPS via regressing local binary features[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2014:1685-1692.
[40] CALDER A J, BURTON A M, MILLER P, et al. A principal component analysis of facial expressions[J]. Vision Research, 2001, 41(9):1179-1208.
[41] AHONEN T, HADID A, PIETIKAINEN M. Face recognition with local binary patterns[C]//Proceedings of the 2004 European Conference on Computer Vision, LNCS 3021. Berlin:Springer, 2004:469-481.
[42] LE H T, HA D T T. Facial expression representation and classification using 2DPCA[C]//Proceedings of the 2012 International Conference on Control, Automation and Information Sciences. Piscataway, NJ:IEEE, 2012:42-47.
[43] 叶学义,王大安,宦天枢,等.基于张量的2D-PCA人脸识别算法[J].计算机工程与应用,2017,53(6):1-6.(YE X Y, WANG D A, HUAN T S, et al. Novel 2D-PCA face recognition based on tensor[J]. Computer Engineering and Applications, 2017, 53(6):1-6.)
[44] FANG Y C, LUO J, LOU C S. Fusion of multi-directional rotation invariant uniform LBP features for face recognition[C]//ⅡTA 2009:Proceedings of the 2009 Third International Symposium on Intelligent Information Technology Application. Piscataway, NJ:IEEE, 2009:332-335.
[45] 李春利,沈鲁娟.基于改进LBP算子的纹理图像分类方法[J].计算机工程与设计,2016,37(1):232-236.(LI C L,SHEN L J. Texture image classification method based on improved LBP operator[J]. Computer Engineering and Design, 2016, 37(1):232-236.)
[46] LIU C, WECHSLER H. Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition[J]. IEEE Transactions on Image Processing, 2002, 11(4):467-476.
[47] 孔健.基于MLBP-TOP与光流多特征的人脸表情融合识别研究[D].镇江:江苏大学,2009:40-44.(KONG J. Research on face expression fusion recognition based on MLBP-TOP and optical flow multi-feature[D]. Zhenjiang:Jiangsu University, 2009:40-44.)
[48] BOUREL F, CHIBELUSHI C C, LOW A A. Robust facial expression recognition using a state-based model of spatially-localised facial dynamics[C]//Proceedings of the 2002 Fifth IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 2002:106-111.
[49] TIE Y, GUAN L. A deformable 3D facial expression model for dynamic human emotional state recognition[J]. IEEE Transactions on Circuits & Systems for Video Technology, 2013, 23(1):142-157.
[50] WISKOTT L, FELLOUS J M, KRVGER N, et al. Face recognition by elastic bunch graph matching[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1997, 19(7):775-779.
[51] VOGL T P, MANGIS J K, RIGLER A K, et al. Accelerating the convergence of the back-propagation method[J].Biological Cybernetics,1988, 59(4/5):257-263.
[52] HASTIE T, TIBSHIRANI R. Discriminant adaptive nearest neighbor classification[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1996, 18(6):607-616.
[53] SEBE N, LEW M S, SUN Y, et al. Authentic facial expression analysis[J]. Image & Vision Computing, 2007, 25(12):1856-1863.
[54] YOSHITOMI Y, MIYAWAKI N, TOMITA S, et al. Facial expression recognition using thermal image processing and neural network[C]//Proceedings of the 19976th IEEE International Workshop on Robot and Human Communication. Piscataway, NJ:IEEE, 1997:380-385.
[55] HERNÁNDEZ B, OLAGUE G, HAMMOUD R, et al. Visual learning of texture descriptors for facial expression recognition in thermal imagery[J]. Computer Vision & Image Understanding, 2007, 106(2/3):258-269.
[56] RANZATO M, SUSSKIND J, MNIH V, et al. On deep generative models with applications to recognition[C]//Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2011:2857-2864.
[57] WALECKI R, RUDOVIC O, PAVLOVIC V, et al. Variable-state latent conditional random fields for facial expression recognition and action unit detection[C]//Proceedings of the 201511th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 2015:1-8.
[58] AMBADAR Z, SCHOOLER J W, COHN J F. Deciphering the enigmatic face:the importance of facial dynamics in interpreting subtle facial expressions[J]. Psychological Science, 2005, 16(5):403-410.

基于图像的面部表情识别方法综述

Summary of facial expression recognition methods based on image

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	牛瑞华, 杨俊, 邢斓馨, 吴仁彪. 基于卷积注意力模块和双通道网络的微表情识别算法[J]. 计算机应用, 2021, 41(9): 2552-2559.
[2]	郑志强, 胡鑫, 翁智, 王雨禾, 程曦. 基于改进DenseNet的牛眼图像特征提取方法[J]. 计算机应用, 2021, 41(9): 2780-2784.
[3]	佘玉龙, 张晓龙, 程若勤, 邓春华. 基于边缘关注模型的语义分割方法[J]. 计算机应用, 2021, 41(2): 343-349.
[4]	赵津, 宋文爱, 邰隽, 杨吉江, 王青, 李晓丹, 雷毅, 邱悦. 儿童阻塞性睡眠呼吸暂停计算机人脸辅助诊断综述[J]. 《计算机应用》唯一官方网站, 2021, 41(11): 3394-3401.
[5]	朱新成, 何坤金, 倪娜, 郝博. 基于改进迭代最近点算法的接骨板贴合性快捷计算方法[J]. 计算机应用, 2021, 41(10): 3033-3039.
[6]	尹春勇, 何苗. 基于改进胶囊网络的文本分类[J]. 计算机应用, 2020, 40(9): 2525-2530.
[7]	周云, 陈淑荣. 基于双流非局部残差网络的行为识别方法[J]. 计算机应用, 2020, 40(8): 2236-2240.
[8]	黄俊, 张娜娜, 章惠. 融合头部姿态和面部表情的互动式活体检测[J]. 计算机应用, 2020, 40(7): 2089-2095.
[9]	张家岗, 李达平, 杨晓东, 邹茂扬, 吴锡, 胡金蓉. 基于深度卷积特征光流的形变医学图像配准算法[J]. 计算机应用, 2020, 40(6): 1799-1805.
[10]	徐代, 岳璋, 杨文霞, 任潇. 基于改进的三向流Faster R-CNN的篡改图像识别[J]. 计算机应用, 2020, 40(5): 1315-1321.
[11]	郭志强, 胡永武, 刘鹏, 杨杰. 基于特征融合的室外天气图像分类[J]. 计算机应用, 2020, 40(4): 1023-1029.
[12]	沈亮, 王鑫, 陈曙晖. 面向移动应用识别的结构化特征提取方法[J]. 计算机应用, 2020, 40(4): 1109-1114.
[13]	刘尚旺, 刘承伟, 张爱丽. 基于深度可分卷积神经网络的实时人脸表情和性别分类[J]. 计算机应用, 2020, 40(4): 990-995.
[14]	张俊升, 徐晶晶, 余伟. 面部美化图像质量无参考评价方法[J]. 计算机应用, 2020, 40(4): 1184-1190.
[15]	朱喆, 许少华. 降噪自编码器深度卷积过程神经网络及在时变信号分类中的应用[J]. 计算机应用, 2020, 40(3): 698-703.