基于图像的面部表情识别方法综述

doi:10.11772/j.issn.1001-9081.2017.12.3509

计算机应用 ›› 2017, Vol. 37 ›› Issue (12): 3509-3516.DOI: 10.11772/j.issn.1001-9081.2017.12.3509

• 计算机视觉与虚拟现实 • 上一篇下一篇

基于图像的面部表情识别方法综述

徐琳琳, 张树美, 赵俊莉

青岛大学数据科学与软件工程学院, 山东青岛 266071

收稿日期:2017-06-26 修回日期:2017-09-18 发布日期:2017-12-18 出版日期:2017-12-10
通讯作者: 张树美
作者简介:徐琳琳(1992-),女,山东莱芜人,硕士研究生,主要研究方向:图像识别与处理、深度学习;张树美(1964-),女,山东莱西人,教授,博士,主要研究方向:时滞非线性系统的分析与控制、图像识别与处理;赵俊莉(1977-),山西新绛人,助理教授,博士,CCF会员,主要研究方向:计算机视觉、计算机图形学、虚拟现实。
基金资助:
国家自然科学基金资助项目（61702293，41506198）；虚拟现实应用教育部工程研究中心开放基金课题（MEOBNUEVRA201601）。

Summary of facial expression recognition methods based on image

XU Linlin, ZHANG Shumei, ZHAO Junli

School of Data Science and Software Engineering, Qingdao University, Qingdao Shandong 266071, China

Received:2017-06-26 Revised:2017-09-18 Online:2017-12-18 Published:2017-12-10
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61702293, 41506198), the Open Research Fund of the Ministry of Education for Engineering Research Center of Virtual Reality Application (MEOBNUEVRA201601).

摘要/Abstract

摘要： 近年来，面部表情识别在教育、医学、心理分析以及商业领域得到了广泛关注。针对目前表情识别方法不够系统、概念模糊的问题，对面部表情识别的步骤及其方法进行了综述探讨。首先，介绍了目前常用的人脸表情数据集，并回顾了面部表情识别的发展历程；然后，介绍了人脸表情识别的面部表情编码和面部表情识别过程这两个方面，归纳了人脸面部表情识别的四个过程，重点总结了特征提取和表情分类两个过程中的经典算法以及这些算法的基本原理和优劣比较；最后，指出了目前面部表情识别存在的问题和未来可能的发展趋势。

关键词: 表情识别, 表情数据集, 表情编码, 特征提取, 表情分类

Abstract: In recent years, facial expression recognition has received extensive attention in education, medicine, psychoanalysis and business. Aiming at the problems of not systematic enough and fuzzy concept of facial expression recognition method, the steps and methods of facial expression recognition were reviewed and discussed. Firstly, the commonly used facial expression databases were introduced and the development of facial expression recognition was reviewed. Then, two aspects of facial expression recognition were introduced, such as facial expression coding and facial expression recognition. The four processes of face facial expression recognition were summarized. The classical algorithms, the basic principles of these algorithms and the comparisons of their advantages and disadvantages were summarized emphatically in the two processes of feature extraction and facial expression classification. Finally, the existing problems and possible development trends in the future of the current facial expression recognition were pointed out.

Key words: expression recognition, expression database, expression coding, feature extraction, expression classification

中图分类号:

TP391.41

徐琳琳, 张树美, 赵俊莉. 基于图像的面部表情识别方法综述[J]. 计算机应用, 2017, 37(12): 3509-3516.

XU Linlin, ZHANG Shumei, ZHAO Junli. Summary of facial expression recognition methods based on image[J]. Journal of Computer Applications, 2017, 37(12): 3509-3516.

参考文献

[1] GIRARD J M, COHN J F, MAHOOR M H, et al. Nonverbal social withdrawal in depression:evidence from manual and automatic analyses[J]. Image and Vision Computing, 2014, 32(10):641-647.
[2] LUCEY P, COHN J F, MATTHEWS I, et al. Automatically detecting pain in video through facial action units[J]. IEEE Transactions on Systems, Man, and Cybernetics, Part B:Cybernetics, 2011, 41(3):664-674.
[3] IRANI R, NASROLLAHI K, SIMON M O, et al. Spatiotemporal analysis of RGB-D-T facial images for multimodal pain level recognition[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2015:88-95.
[4] EKMAN P. Universal and cultural differences in facial expression of emotion[J]. Nebraska Symposium on Motivation, 1972, 19:207-284.
[5] LYONS M, AKAMATSU S, KAMACHI M, et al. Coding facial expressions with Gabor wavelets[C]//Proceedings of the 1998 IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 1998:200-205.
[6] LUCEY P, COHN J F, KANADE T, et al. The extended Cohn-Kanade dataset (CK+):a complete dataset for action unit and emotion-specified expression[C]//Proceedings of the 2010 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2010:94-101.
[7] VALSTAR M F, PANTIC M. Induced disgust, happiness and surprise:an addition to the mmi facial expression database[EB/OL].[2017-04-16]. https://ibug.doc.ic.ac.uk/media/uploads/documents/EMOTION-2010-ValstarPantic-CAMERA.pdf.
[8] 薛雨丽,毛峡,张帆.BHU人脸表情数据库的设计与实现[J].北京航空航天大学学报,2007,33(2):224-228.(XUE Y L, MAO X, ZHANG F. Design and realization of BHU facial expression database[J]. Journal of Beijing University of Aeronautics and Astronautics, 2007, 33(2):224-228.)
[9] WANG S F, LIU Z L, LYU S L, et al. A natural visible and infrared facial expression database for expression recognition and emotion Inference[J]. IEEE Transactions on Multimedia, 2010, 12(7):682-691.
[10] DHALL A, GOECKE R, LUCEY S, et al. Collecting large, richly annotated facial-expression databases from movies[J]. IEEE Multimedia, 2012, 19(3):34-41.
[11] DHALL A, GOECKE R, LUCEY S, et al. Static facial expression analysis in tough conditions:data, evaluation protocol and benchmark[C]//Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops. Piscataway, NJ:IEEE, 2011:2106-2112.
[12] MASE K, PENTLAND A. Automatic lipreading by optical-flow analysis[J]. Systems & Computers in Japan, 2015, 22(6):67-76.
[13] TIAN Y L, KANADE T, COHN J F. Recognizing action units for facial expression analysis[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 2001, 23(2):97-115.
[14] COHEN I, SEBE N, GARG A, et al. Facial expression recognition from video sequences:temporal and static modeling[J]. Computer Vision & Image Understanding, 2003, 91(1/2):160-187.
[15] ALEKSIC P S, KATSAGGELOS A K. Automatic facial expression recognition using facial animation parameters and multi-stream hmms[J]. IEEE Transactions on Information Forensics & Security, 2006, 1(1):3-11.
[16] KOTSIA I, PITAS I. Facial expression recognition in image sequences using geometric deformation features and support vector machines[J]. IEEE Transactions on Image Processing, 2007, 16(1):172-187.
[17] LITTLEWORT G, BARTLETT M S, FASEL I, et al. Dynamics of facial expression extracted automatically from video[J]. Image & Vision Computing, 2006, 24(6):615-625.
[18] SHAN C F, GONG S G, MCOWAN P W. Facial expression recognition based on local binary patterns:a comprehensive study[J]. Image and Vision Computing, 2009, 27(6):803-816.
[19] YIN L J, WEI X Z, SUN Y, et al. A 3D facial expression database for facial behavior research[C]//Proceedings of the 20067th International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 2006:211-216.
[20] TANG H, HUANG T S. 3D facial expression recognition based on automatically selected features[C]//CVPRW 2008:Proceedings of the 2008 IEEE Computer Society Conference on Computer Vision and Pattern Recognition Workshops. Piscataway, NJ:IEEE, 2008:1-8.
[21] VRETOS N, NIKOLAIDIS N, PITAS I. 3D facial expression recognition using Zernike moments on depth images[C]//Proceedings of the 201118th IEEE International Conference on Image Processing. Piscataway, NJ:IEEE, 2011:773-776.
[22] KONG S G, HEO J, ABIDI B R, et al. Recent advances in visual and infrared face recognition-a review[J]. Computer Vision & Image Understanding, 2005, 97(1):103-135.
[23] 王国胤,龚勋.人脸感知:从二维到三维[M].北京:科学出版社,2011:10-30.(WANG G Y, GONG X. Human Face Perception:from 2D to 3D[M]. Beijing:Science Press, 2011:10-30.)
[24] FREUND Y, SCHAPIRE R E. A decision-theoretic generalization of on-line learning and an application to boosting[C]//Proceedings of the 1995 European Conference on Computational Learning Theory, LNCS 904. Berlin:Springer, 1995:23-37.
[25] 王建.基于Gentle AdaBoost算法的人脸检测研究[D].成都:电子科技大学,2011:24-38.(WANG J. Research on face detection based on Gentle AdaBoost algorithm[D]. Chengdu:University of Electronic Science and Technology of China, 2011:24-38.)
[26] NAKAMURA M, NOMUYA H, UEHARA K. Improvement of boosting algorithm by modifying weighting rule[J]. Annals of Mathematics & Artificial Intelligence, 2004, 41(1):95-109.
[27] ZHANG Z Q, LI M J, LI S Z, et al. Multi-view face detection with FloatBoost[C]//Proceedings of the 2002 IEEE Workshop on Applications of Computer Vision. Washington, DC:IEEE Computer Society, 2002:184.
[28] LECUN Y, BOSER B, DENKER J S, et al. Backpropagation applied to handwritten zip code recognition[J]. Neural Computation, 1989, 1(4):541-551.
[29] LECUN Y, BOTTOU L, BENGIO Y, et al. Gradient-based learning applied to document recognition[J]. Proceedings of the IEEE, 1998, 86(11):2278-2324.
[30] 汪济民.基于卷积神经网络的人脸检测和性别识别研究[D].南京:南京理工大学,2015:42-47.(WANG J M. Research on face detection and gender recognition based on convolution neural network[D]. Nanjing:Nanjing University of Science and Technology, 2015:42-47.)
[31] JADERBERG M, SIMONYAN K, ZISSERMAN A, et al. Spatial transformer networks[C]//NIPS 2015:Proceedings of the 201528th International Conference on Neural Information Processing Systems. Cambridge, MA:MIT Press, 2015:2017-2025.
[32] ZHAO J B, MATHIEU M, GOROSHIN R, et al. Stacked what-where auto-encoders[EB/OL].[2017-04-10]. http://xueshu.baidu.com/s?wd=paperuri%3A%28c2e4cc91c19dc2d85fe0c72fcf535763%29&filter=sc_long_sign&tn=SE_xueshusource_2kduw22v&sc_vurl=http%3A%2F%2Farxiv.org%2Fpdf%2F1506.02351.pdf&ie=utf-8&sc_us=5921033091152195876.
[33] 李辉,石波.基于卷积神经网络的人脸识别算法[J].软件导刊,2017,16(3):26-29.(LI H, SHI B. Face recognition algorithm based on convolution neural network[J]. Software Guide, 2017, 16(3):26-29.)
[34] COOTES T F, TAYLOR C J, COOPER D H, et al. Active shape models-their training and application[J]. Computer Vision & Image Understanding, 1995, 61(1):38-59.
[35] COOTES T F, EDWARDS G J, TAYLOR C J. Active appearance models[C]//Proceedings of the 1998 European Conference on Computer Vision, LNCS 1407. Berlin:Springer, 1998:484-498.
[36] 金鑫,谭晓阳.受限局部模型在人脸特征点定位中应用综述[J].小型微型计算机系统,2017,38(2):371-375.(JIN X, TAN X Y. Facial feature point detection with constrained local models:a survey[J]. Journal of Chinese Computer Systems, 2017, 38(2):371-375)
[37] SUN Y, WANG X G, TANG X O. Deep convolutional network cascade for facial point detection[C]//CVPR 2013:Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2013:3476-3483.
[38] CAO X D, WEI Y C, WEN F, et al. Face alignment by explicit shape regression[J]. International Journal of Computer Vision, 2014, 107(2):177-190.
[39] REN S Q, CAO X D, WEI Y C, et al. Face alignment at 3000 FPS via regressing local binary features[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2014:1685-1692.
[40] CALDER A J, BURTON A M, MILLER P, et al. A principal component analysis of facial expressions[J]. Vision Research, 2001, 41(9):1179-1208.
[41] AHONEN T, HADID A, PIETIKAINEN M. Face recognition with local binary patterns[C]//Proceedings of the 2004 European Conference on Computer Vision, LNCS 3021. Berlin:Springer, 2004:469-481.
[42] LE H T, HA D T T. Facial expression representation and classification using 2DPCA[C]//Proceedings of the 2012 International Conference on Control, Automation and Information Sciences. Piscataway, NJ:IEEE, 2012:42-47.
[43] 叶学义,王大安,宦天枢,等.基于张量的2D-PCA人脸识别算法[J].计算机工程与应用,2017,53(6):1-6.(YE X Y, WANG D A, HUAN T S, et al. Novel 2D-PCA face recognition based on tensor[J]. Computer Engineering and Applications, 2017, 53(6):1-6.)
[44] FANG Y C, LUO J, LOU C S. Fusion of multi-directional rotation invariant uniform LBP features for face recognition[C]//ⅡTA 2009:Proceedings of the 2009 Third International Symposium on Intelligent Information Technology Application. Piscataway, NJ:IEEE, 2009:332-335.
[45] 李春利,沈鲁娟.基于改进LBP算子的纹理图像分类方法[J].计算机工程与设计,2016,37(1):232-236.(LI C L,SHEN L J. Texture image classification method based on improved LBP operator[J]. Computer Engineering and Design, 2016, 37(1):232-236.)
[46] LIU C, WECHSLER H. Gabor feature based classification using the enhanced Fisher linear discriminant model for face recognition[J]. IEEE Transactions on Image Processing, 2002, 11(4):467-476.
[47] 孔健.基于MLBP-TOP与光流多特征的人脸表情融合识别研究[D].镇江:江苏大学,2009:40-44.(KONG J. Research on face expression fusion recognition based on MLBP-TOP and optical flow multi-feature[D]. Zhenjiang:Jiangsu University, 2009:40-44.)
[48] BOUREL F, CHIBELUSHI C C, LOW A A. Robust facial expression recognition using a state-based model of spatially-localised facial dynamics[C]//Proceedings of the 2002 Fifth IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 2002:106-111.
[49] TIE Y, GUAN L. A deformable 3D facial expression model for dynamic human emotional state recognition[J]. IEEE Transactions on Circuits & Systems for Video Technology, 2013, 23(1):142-157.
[50] WISKOTT L, FELLOUS J M, KRVGER N, et al. Face recognition by elastic bunch graph matching[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1997, 19(7):775-779.
[51] VOGL T P, MANGIS J K, RIGLER A K, et al. Accelerating the convergence of the back-propagation method[J].Biological Cybernetics,1988, 59(4/5):257-263.
[52] HASTIE T, TIBSHIRANI R. Discriminant adaptive nearest neighbor classification[J]. IEEE Transactions on Pattern Analysis & Machine Intelligence, 1996, 18(6):607-616.
[53] SEBE N, LEW M S, SUN Y, et al. Authentic facial expression analysis[J]. Image & Vision Computing, 2007, 25(12):1856-1863.
[54] YOSHITOMI Y, MIYAWAKI N, TOMITA S, et al. Facial expression recognition using thermal image processing and neural network[C]//Proceedings of the 19976th IEEE International Workshop on Robot and Human Communication. Piscataway, NJ:IEEE, 1997:380-385.
[55] HERNÁNDEZ B, OLAGUE G, HAMMOUD R, et al. Visual learning of texture descriptors for facial expression recognition in thermal imagery[J]. Computer Vision & Image Understanding, 2007, 106(2/3):258-269.
[56] RANZATO M, SUSSKIND J, MNIH V, et al. On deep generative models with applications to recognition[C]//Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2011:2857-2864.
[57] WALECKI R, RUDOVIC O, PAVLOVIC V, et al. Variable-state latent conditional random fields for facial expression recognition and action unit detection[C]//Proceedings of the 201511th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Piscataway, NJ:IEEE, 2015:1-8.
[58] AMBADAR Z, SCHOOLER J W, COHN J F. Deciphering the enigmatic face:the importance of facial dynamics in interpreting subtle facial expressions[J]. Psychological Science, 2005, 16(5):403-410.

基于图像的面部表情识别方法综述

Summary of facial expression recognition methods based on image

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	杨鑫, 陈雪妮, 吴春江, 周世杰. 结合变种残差模型和Transformer的城市公路短时交通流预测[J]. 《计算机应用》唯一官方网站, 2024, 44(9): 2947-2951.
[2]	付帅, 郭小英, 白茹意, 闫涛, 陈斌. 改进的CloFormer模型与有序回归相结合的年龄评估方法[J]. 《计算机应用》唯一官方网站, 2024, 44(8): 2372-2380.
[3]	陈彤, 杨丰玉, 熊宇, 严荭, 邱福星. 基于多尺度频率通道注意力融合的声纹库构建方法[J]. 《计算机应用》唯一官方网站, 2024, 44(8): 2407-2413.
[4]	龙伍丹, 彭博, 胡节, 申颖, 丁丹妮. 基于加强特征提取的道路病害检测算法[J]. 《计算机应用》唯一官方网站, 2024, 44(7): 2264-2270.
[5]	刘瑞华, 郝子赫, 邹洋杨. 基于多层级精细特征融合的步态识别算法[J]. 《计算机应用》唯一官方网站, 2024, 44(7): 2250-2257.
[6]	吴郅昊, 迟子秋, 肖婷, 王喆. 基于元学习自适应的小样本语音合成[J]. 《计算机应用》唯一官方网站, 2024, 44(5): 1629-1635.
[7]	崔晨辉, 蔺素珍, 李大威, 禄晓飞, 武杰. 基于孪生网络和Transformer的红外弱小目标跟踪方法[J]. 《计算机应用》唯一官方网站, 2024, 44(2): 563-571.
[8]	范艺扬, 张洋, 曾尚, 曾渝, 付茂栗. 基于分解和频域特征提取的多变量长时间序列预测模型[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3442-3448.
[9]	赵培, 乔焰, 胡荣耀, 袁新宇, 李敏悦, 张本初. 基于多域特征提取的多变量时间序列异常检测[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3419-3426.
[10]	刘涛, 鞠事宏, 高一萌. 基于改进YOLOv8n的无人机视角下小目标检测算法[J]. 《计算机应用》唯一官方网站, 2024, 44(11): 3603-3609.
[11]	花晓雨, 李冬芬, 付优, 毕可骏, 应时, 王瑞锦. 结合层次图神经网络与长短期记忆的产业链风险评估预警模型[J]. 《计算机应用》唯一官方网站, 2024, 44(10): 3223-3231.
[12]	张雨宁, 阿布都克力木·阿布力孜, 梅悌胜, 徐春, 麦尔达娜·买买提热依木, 哈里旦木·阿布都克里木, 侯钰涛. 基于自监督特征提取的骨骼X线影像异常检测方法[J]. 《计算机应用》唯一官方网站, 2024, 44(1): 175-181.
[13]	李牧, 杨宇恒, 柯熙政. 基于混合特征提取与跨模态特征预测融合的情感识别模型[J]. 《计算机应用》唯一官方网站, 2024, 44(1): 86-93.
[14]	田悦霖, 黄瑞章, 任丽娜. 融合局部语义特征的学者细粒度信息提取方法[J]. 《计算机应用》唯一官方网站, 2023, 43(9): 2707-2714.
[15]	王先兰, 周金坤, 穆楠, 王晨. 基于多任务联合学习的跨视角地理定位方法[J]. 《计算机应用》唯一官方网站, 2023, 43(5): 1625-1635.