基于卷积注意力模块和双通道网络的微表情识别算法

doi:10.11772/j.issn.1001-9081.2020111743

计算机应用 ›› 2021, Vol. 41 ›› Issue (9): 2552-2559.DOI: 10.11772/j.issn.1001-9081.2020111743

所属专题：人工智能

基于卷积注意力模块和双通道网络的微表情识别算法

牛瑞华, 杨俊, 邢斓馨, 吴仁彪

天津市智能信号与图像处理重点实验室(中国民航大学), 天津 300300

收稿日期:2020-11-09 修回日期:2021-01-05 出版日期:2021-09-10 发布日期:2021-09-15
通讯作者: 吴仁彪
作者简介:牛瑞华(1995-),女,河北衡水人,硕士研究生,主要研究方向:微表情识别、图像处理、深度学习;杨俊(1989-),男,浙江温州人,助理研究员,博士研究生,主要研究方向:图像处理、深度学习;邢斓馨(1998-),女,海南乐东人,硕士研究生,主要研究方向:微表情视频处理、神经网络;吴仁彪(1966-),男,湖北武汉人,教授,博士生导师,博士,主要研究方向:自适应信号处理,现代谱分析及其在雷达、卫星导航和空管中的应用。
基金资助:
中央高校基本科研业务费专项基金资助项目（3122019185）。

Micro-expression recognition algorithm based on convolutional block attention module and dual path networks

NIU Ruihua, YANG Jun, XING Lanxin, WU Renbiao

Tianjin Key Laboratory for Advanced Signal Processing(Civil Aviation University of China), Tianjin 300300, China

Received:2020-11-09 Revised:2021-01-05 Online:2021-09-10 Published:2021-09-15
Supported by:
This work is partially supported by the Fundamental Research Funds for the Central Universities (3122019185).

摘要/Abstract

摘要： 微表情是一种人类在试图隐藏自己真实情感时作出的面部动作，具有持续时间短、幅度小的典型特点。针对微表情识别难度大、识别效果不理想的问题，提出一种基于卷积注意力模块（CBAM）和双通道网络（DPN）的微表情识别算法——CBAM-DPN。首先，进行典型微表情数据集的数据融合；然后，分析序列帧中像素的变化值以确定顶点帧位置，再对顶点帧进行图像增强处理；最后，基于CBAM-DPN对图像增强后的微表情顶点帧进行特征的有效提取，并构建分类器对微表情进行识别。优化后模型的未加权F1值（UF1）和未加权平均召回率（UAR）分别可以达到0.720 3和0.729 3，相较于DPN模型分别提高了0.048 9和0.037 9，相较于CapsuleNet模型分别提高了0.068 3和0.078 7。实验结果表明，CBAM-DPN算法融合了CBAM和DPN的共同优势，可增强微小特征的信息提取能力，有效改善微表情识别性能。

关键词: 微表情识别, 双通道网络, 卷积注意力模块, 顶点帧, 结构优化

Abstract: Micro-expression is a facial movement that humans make when they are trying to hide their true emotions. It has the typical characteristics of short duration and small amplitude. Concerning the problems of the difficulty in recognition and the unsatisfactory recognition effect of micro-expression, a micro-expression recognition algorithm based on Convolutional Block Attention Module (CBAM) and Dual Path Networks (DPN), namely CBAM-DPN, was proposed. Firstly, data fusion of typical micro-expression datasets was performed. Then, the change values of pixels in the sequence frames were analyzed to determine the position of the apex frame, after that, image enhancement was performed to the apex frame. Finally, based on the CBAM-DPN network, the features of the enhanced micro-expression apex frame was effectively extracted, and a classifier was constructed to recognize the micro-expression. The Unweighted F1-score (UF1) and Unweighted Average Recall (UAR) of the model after optimization can reach 0.720 3 and 0.729 3 respectively, which are improved by 0.048 9 and 0.037 9 respectively compared with those of the DPN model, and are improved by 0.068 3 and 0.078 7 respectively compared with those of the CapsuleNet model. Experimental results show that the CBAM-DPN algorithm combined with the advantages of CBAM and DPN can enhance the information extraction ability of small features, and effectively improve the performance of micro-expression recognition.

Key words: micro-expression recognition, Dual Path Networks (DPN), Convolutional Block Attention Module (CBAM), apex frame, structure optimization

中图分类号:

TP391

牛瑞华, 杨俊, 邢斓馨, 吴仁彪. 基于卷积注意力模块和双通道网络的微表情识别算法[J]. 计算机应用, 2021, 41(9): 2552-2559.

NIU Ruihua, YANG Jun, XING Lanxin, WU Renbiao. Micro-expression recognition algorithm based on convolutional block attention module and dual path networks[J]. Journal of Computer Applications, 2021, 41(9): 2552-2559.

参考文献

[1] ZHANG Y L,JIANG H X,LI X W,et al. A new framework combining local-region division and feature selection for microexpressions recognition[J]. IEEE Access,2020,8:94499-94509.
[2] XIA Z Q,PENG W,KHOR H Q,et al. Revealing the invisible with model and data shrinking for composite-database microexpression recognition[J]. IEEE Transactions on Image Processing,2020,29:8590-8605.
[3] TAKALKAR M,XU M,WU Q,et al. A survey:facial microexpression recognition[J]. Multimedia Tools and Applications, 2018,77(15):19301-19325.
[4] LI X B,HONG X P,MOILANEN A,et al. Towards reading hidden emotions:a comparative study of spontaneous microexpression spotting and recognition methods[J]. IEEE Transactions on Affective Computing,2018,9(4):563-577.
[5] 张延良, 卢冰, 洪晓鹏, 等. 基于局部区域方法的微表情识别[J]. 计算机应用,2019,39(5):1282-1287.(ZHANG Y L,LU B, HONG X P,et al. Micro-expression recognition based on local region method[J]. Journal of Computer Applications,2019,39(5):1282-1287.)
[6] LI X B,PFISTER T,HUANG X H,et al. A spontaneous microexpression database:inducement,collection and baseline[C]//Proceedings of the 10th IEEE International Conference and Workshops on Automatic Face and Gesture Recognition. Piscataway:IEEE,2013:1-6.
[7] XU F,ZHANG J P,WANG J Z. Microexpression identification and categorization using a facial dynamics map[J]. IEEE Transactions on Affective Computing,2017,8(2):254-267.
[8] LIU Y J,ZHANG J K,YAN W J,et al. A main directional mean optical flow feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing,2016,7(4):299-310.
[9] LIU Y J, LI B J, LAI Y K. Sparse MDMO:learning a discriminative feature for spontaneous micro-expression recognition[J]. IEEE Transactions on Affective Computing,2021,12(1):254-261.
[10] PENG M,WU Z,ZHANG Z H,et al. From macro to micro expression recognition:deep learning on small datasets using transfer learning[C]//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway:IEEE,2018:657-661.
[11] KHOR H Q,SEE J,PHAN R C W,et al. Enriched long-term recurrent convolutional network for facial micro-expression recognition[C]//Proceedings of the 13th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway:IEEE,2018:667-674.
[12] PENG M,WANG C Y,BI T,et al. A novel Apex-Time Network for cross-dataset micro-expression recognition[C]//Proceedings of the 8th International Conference on Affective Computing and Intelligent Interaction. Piscataway:IEEE,2019:1-6.
[13] HUANG J,ZHAO X R,ZHENG L M. SHCFNet on microexpression recognition system[C]//Proceedings of the 13th International Congress on Image and Signal Processing, BioMedical Engineering and Informatics. Piscataway:IEEE, 2020:163-168.
[14] ZHOU L,MAO Q R,XUE L Y. Cross-database micro-expression recognition:a style aggregated and attention transfer approach[C]//Proceedings of the 2019 IEEE International Conference on Multimedia and Expo Workshops. Piscataway:IEEE, 2019:102-107.
[15] PAN H,XIE L,LV Z P,et al. Hierarchical support vector machine for facial micro-expression recognition[J]. Multimedia Tools and Applications,2020,79(41/42):31451-31465.
[16] QUANG N V,CHUN J,TOKUYAMA T. CapsuleNet for microexpression recognition[C]//Proceedings of the 14th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway:IEEE,2019:1-7.
[17] CHEN Y P,LI J N,XIAO H X,et al. Dual path networks[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook, NY:Curran Associates Inc.,2017:4470-4478.
[18] WOO S,PARK J,LEE J Y,et al. CBAM:convolutional block attention module[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11211. Cham:Springer, 2018:3-19.
[19] HE K M,ZHANG X Y,REN S Q,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:770-778.
[20] 张顺, 龚怡宏, 王进军. 深度卷积神经网络的发展及其在计算机视觉领域的应用[J]. 计算机学报,2019,42(3):453-482. (ZHANG S,GONG Y H,WANG J J. The development of deep convolution neural network and its applications computer vision[J]. Chinese Journal of Computers,2019,42(3):453-482.)
[21] ZHU Y,NEWSAM S. DenseNet for dense flow[C]//Proceedings of the 2017 IEEE International Conference on Image Processing. Piscataway:IEEE,2017:790-794.
[22] XIE S N,GIRSHICK R,DOLLÁR P,et al. Aggregated residual transformations for deep neural networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:5987-5995.
[23] 吴仁彪, 赵娅倩, 屈景怡, 等. 基于CBAM-CondenseNet的航班延误波及预测模型[J]. 电子与信息学报,2021,43(1):187-195. (WU R B,ZHAO Y Q,QU J Y,et al. Flight delay propagation prediction model based on CBAM-CondenseNet[J]. Journal of Electronics and Information Technology,2021,43(1):187-195.)
[24] KANG X D,ZHUO B B,DUAN P H. Dual-path network-based hyperspectral image classification[J]. IEEE Geoscience and Remote Sensing Letters,2019,16(3):447-451.
[25] YAN W J,LI X B,WANG S J,et al. CASME Ⅱ:an improved spontaneous micro-expression database and the baseline evaluation[J]. PLoS ONE,2014,9(1):No. e86041.
[26] DAVISION A K,LANSLEY C,COSTEN N,et al. SAMM:a spontaneous micro-facial movement dataset[J]. IEEE Transactions on Affective Computing,2018,9(1):116-129.
[27] ZHANG K P,ZHANG Z P,LI Z F,et al. Joint face detection and alignment using multitask cascaded convolutional networks[J]. IEEE Signal Processing Letters,2016,23(10):1499-1503.
[28] GEITGEY A. Face_recognition[EB/OL].[2020-10-30]. https://github.com/ageitgey/face_recognition.
[29] SELVARAJU R R,COGSWELL M,DAS A,et al. Grad-CAM:visual explanations from deep networks via gradient-based localization[J]. International Journal of Computer Vision,2020, 128(2):336-359.

基于卷积注意力模块和双通道网络的微表情识别算法

Micro-expression recognition algorithm based on convolutional block attention module and dual path networks

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 3

编辑推荐

Metrics

[1]	林筠超, 万源. 基于图结构优化的自适应多度量非监督特征选择方法[J]. 计算机应用, 2021, 41(5): 1282-1289.
[2]	张延良, 卢冰, 洪晓鹏, 赵国英, 张伟涛. 基于局部区域方法的微表情识别[J]. 计算机应用, 2019, 39(5): 1282-1287.
[3]	李祥飞邹莉华彭可. 一类模糊神经网络结构的混沌优化设计[J]. 计算机应用, 2007, 27(6): 1411-1414.