Lightweight face liveness detection method based on multi-modal feature fusion

doi:10.11772/j.issn.1001-9081.2020050660

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (12): 3658-3665.DOI: 10.11772/j.issn.1001-9081.2020050660

• Virtual reality and multimedia computing • Previous Articles Next Articles

Lightweight face liveness detection method based on multi-modal feature fusion

PI Jiatian^1,2,3, YANG Jiezhi^1,2,3, YANG Linxi^2,3,4, PENG Mingjie^1,2,3, DENG Xiong^3,4, ZHAO Lijun³, TANG Wanmei¹, WU Zhiyou^3,4

1. School of Computer and Information Science, Chongqing Normal University, Chongqing 401331, China;
2. Chongqing Digital Agriculture Service Engineering Technology Research Center(Chongqing Normal University), Chongqing 401331, China;
3. Chongqing Key Laboratory of Intelligent Finance and Big Data Analysis(Chongqing Normal University), Chongqing 401331, China;
4. School of Mathematical Sciences, Chongqing Normal University, Chongqing 401331, China

Received:2020-05-18 Revised:2020-07-13 Online:2020-12-10 Published:2020-08-14
Supported by:
This work is partially supported by the National Natural Science Foundation of China （11971083）， the Youth Project of Chongqing Municipal Education Commission Science and Technology Project （KJQN201800521）， the Chongqing Basic Research and Frontier Exploration Project （cstc2018jcyjAX0470）， the Chongqing Normal University 2019 Graduate Research and Innovation Project （YKC19014）.

基于多模态特征融合的轻量级人脸活体检测方法

皮家甜^1,2,3, 杨杰之^1,2,3, 杨琳希^2,3,4, 彭明杰^1,2,3, 邓雄^3,4, 赵立军³, 唐万梅¹, 吴至友^3,4

1. 重庆师范大学计算机与信息科学学院, 重庆 401331;
2. 重庆市数字农业服务工程技术研究中心(重庆师范大学), 重庆 401331;
3. 智慧金融与大数据分析重庆市重点实验室(重庆师范大学), 重庆 401331;
4. 重庆师范大学数学科学学院, 重庆 401331

通讯作者: 唐万梅(1965-),女,重庆人,教授,博士,主要研究方向:神经网络、预测理论与方法。1093895431@qq.com
作者简介:皮家甜(1990-),男,湖北潜江人,讲师,博士,主要研究方向:计算机视觉、仿生双眼、同步定位与地图构建;杨杰之(1994-),男,重庆人,硕士研究生,主要研究方向:计算机视觉、深度学习、模式识别;杨琳希(1996-),女,重庆人,硕士研究生,主要研究方向:机器学习、最优化问题算法、非线性规划;彭明杰(1993-),男,四川南充人,硕士研究生,主要研究方向:计算机视觉、深度学习、目标检测;邓雄(1991-),男,河南邓州人,硕士研究生,主要研究方向:统计机器学习、模式识别、不确定性推理;赵立军(1970-),男,辽宁朝阳人,博士,主要研究方向:人工智能;吴至友(1967-),女,重庆人,教授,博士,主要研究方向:非线性规划、最优化理论与算法、全局优化理论与算法、组合最优化
基金资助:
国家自然科学基金资助项目（11971083）；重庆市教委科技项目青年项目（KJQN201800521）；重庆市基础研究与前沿探索项目（cstc2018jcyjAX0470）；重庆师范大学2019年研究生科研创新项目（YKC19014）。

Abstract

Abstract: Face liveness detection is an important part of the face recognition process, and is particularly important for the security of identity verification. In view of the cheating methods such as photo, video, mask, hood and head model in the face recognition process, the RGB map and depth map information of the face was collected by the Intel Realsense camera, and a lightweight liveness detection of feature fusion was proposed based on MobileNetV3 to fuse the features of the depth map and the RGB map together and perform the end-to-end training. To solve the problem of large parameter quantity in deep learning and the distinction of the weight areas by the network tail, the method of using Streaming Module at the network tail was proposed to reduce the quantity of network parameters and distinguish weight regions. Simulation experiments were performed on CASIA-SURF dataset and the constructed CQNU-LN dataset. The results show that, on both datasets, the proposed method achieves an accuracy of 95% with TPR@FPR=10E-4, which is increased by 0.1% and 0.05% respectively compared to ShuffleNet with the highest accuracy in the comparison methods. The accuracy of the proposed method reaches an accuracy of 95.2% with TPR@FPR=10E-4 on the constructed CQNU-3Dmask dataset, which is improved by 0.9% and 6.5% respectively compared to those of the method training RGB maps only and the method training depth maps only. In addition, the proposed model has the parameter quantity of only 1.8 MB and FLoating-point Operations Per second (FLOPs) of only 1.5×10⁶. The proposed method can perform accurate and real-time liveness detection on the extracted face target in practical applications.

Key words: computer vision, convolutional neural network, face liveness detection, multi-modal feature fusion, lightweight network

摘要： 人脸活体检测是人脸识别过程中的一个重要环节，对于身份验证的安全性尤为重要。针对人脸识别过程存在照片、视频、面具、头套、头模等欺骗手段，通过Intel Realsense相机采集人脸RGB图和深度图信息，并在MobileNetV3的基础上提出了特征融合的轻量级活体检测网络，将深度图与RGB图的特征融合起来并且进行端到端的训练。而为了解决深度学习中参数量较大以及网络尾部对于权重区域的区分的问题，提出在网络尾部采用Streaming Module以减少网络参数量并且对权重区域进行区分。在CASIA-SURF数据集以及所制作的CQNU-LN数据集上进行仿真实验，结果表明所提方法在两个数据集上均于TPR@FPR=10E-4的级别上达到了95%的精度，相较对比方法中精度最高的ShuffleNet分别提高了0.1%和0.05%；在所制作的CQNU-3Dmask数据集上，所提方法于TPR@FPR=10E-4的级别达到了95.2%的精度，比仅训练RGB图或仅训练深度图的方法分别提升了0.9%和6.5%，并且，模型的参数文件的大小仅为1.8 MB，每秒浮点数运算量（FLOPs）仅为1.5×10⁶。该方法能够在实际应用中对提取到的人脸进行准确的实时检测。

关键词: 计算机视觉, 卷积神经网络, 人脸活体检测, 多模态特征融合, 轻量级网络

CLC Number:

TP183

PI Jiatian, YANG Jiezhi, YANG Linxi, PENG Mingjie, DENG Xiong, ZHAO Lijun, TANG Wanmei, WU Zhiyou. Lightweight face liveness detection method based on multi-modal feature fusion[J]. Journal of Computer Applications, 2020, 40(12): 3658-3665.

皮家甜, 杨杰之, 杨琳希, 彭明杰, 邓雄, 赵立军, 唐万梅, 吴至友. 基于多模态特征融合的轻量级人脸活体检测方法[J]. 计算机应用, 2020, 40(12): 3658-3665.

References

[1] CHINGOVSKA I,ANJOS A,MARCEL S. On the effectiveness of local binary patterns in face anti-spoofing[C]//Proceedings of the 2012 International Conference of Biometrics Special Interest Group. Piscataway:IEEE,2012:1-7.
[2] 邓雄, 王洪春. 基于深度学习和特征融合的人脸活体检测算法[J]. 计算机应用, 2020, 40(4):1009-1015.(DENG X,WANG H C. Face liveness detection algorithm based on deep learning and feature fusion[J]. Journal of Computer Applications,2020,40(4):1009-1015.)
[3] YANG J,LEI Z,LI S Z. Learn convolutional neuralnetwork for face anti-spoofing[EB/OL].[2019-08-26]. https://arxiv.org/pdf/1408.5601.pdf.
[4] SHEN T,HUANG Y,TONG Z. FaceBagNet:bag-of-local-features model for multi-modal face anti-spoofing[C]//Proceedings of the 2019 IEEE/CVF Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2019:1611-1616.
[5] PARKIN A, GRINCHUK O. Recognizing multi-modal face spoofing with face recognitionnetworks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2019:1617-1623.
[6] HOWARD A G,ZHU M,CHEN B,et al. MobileNets:efficient convolutional neuralnetworks for mobile vision applications[EB/OL].[2019-04-17]. https://arxiv.org/pdf/1704.04861.pdf.
[7] SANDLER M,HOWRD A,ZHU M,et al. MobileNetV2:inverted residuals and linear bottlenecks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:4510-4520.
[8] HOWARD A, SANDLER M, CHU G, et al. Searching for MobileNetV3[EB/OL].[2019-11-20]. https://arxiv.org/pdf/1905.02244.pdf.
[9] IANDOLA F N,HAN S,MOSKEWICZ M W,et al. SqueezeNet:AlexNet-level accuracy with 50x fewer parameters and <0.5 MB model size[EB/OL].[2019-11-04]. https://arxiv.org/pdf/1602.07360.pdf.
[10] ZHANG P,ZOU F,WU Z,et al. FeatherNets:convolutional neuralnetworks as light as feather for face anti-spoofing[C]//Proceedings of the 2019 IEEE/CVF Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:1574-1583.
[11] 邓雄, 王洪春, 赵立军, 等. 人脸识别活体检测研究方法综述[J]. 计算机应用研究, 2020, 37(9):2579-2585.(DENG X, WANG H C,ZHAO L J,et al. Survey on face anti-spoofing in face recognition[J]. Application Research of Computers,2020, 37(9):2579-2585.)
[12] HU J,SHEN L,SUN G. Squeeze-and-excitationnetworks[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:7132-7141.
[13] ZOPH B,LE Q V. Neural architecture search with reinforcement learning[EB/OL].[2019-02-15]. https://arxiv.org/pdf/1611.01578.pdf.
[14] YANG T J,HOWARD A,CHEN B,et al. NetAdapt:platformaware neuralnetwork adaptation for mobile applications[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11214. Cham:Springer,2018:289-304.
[15] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[16] IANDOLA F,MOSKEWICZ M,KARAYEV S,et al. DenseNet:implementing efficient ConvNet descriptor pyramids[EB/OL].[2019-04-07]. https://arxiv.org/pdf/1404.1869.pdf.
[17] MA N,ZHANG X,ZHENG H,et al. ShuffleNet V2:practical guidelines for efficient CNN architecture design[C]//Proceedings of 2018 European Conference on Computer Vision,LNCS 11218. Cham:Springer,2018:122-138.
[18] ZHANG X,ZHOU X,LIN M,et al. ShuffleNet:an extremely efficient convolutional neuralnetwork for mobile devices[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:6848-6856.
[19] CHEN S,LIU Y,GAO X,et al. MobilefaceNets:efficient CNNs for accurate real-time face verification on mobile devices[C]//Proceeding of the 2018 Chinese Conference on Biometric Recognition,LNCS 10996. Cham:Springer,2018:428-438.
[20] ZHANG S,WANG X,LIU A,et al. A dataset and benchmark for large-scale multi-modal face anti-spoofing[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:919-928.
[21] LIU W,ANGUELOV D,ERHAN D,et al. SSD:Single Shot MultiBox Detector[C]//Proceedings of the 2016 European Conference on Computer Vision,LNCS 9905. Cham:Springer, 2016:21-37.
[22] 蒋方玲, 刘鹏程, 周祥东. 人脸活体检测综述[J/OL]. 自动化学报[2019-12-02]. https://kns.cnki.net/kcms/detail/11.2109.TP.20190428.1046.003.html.(JIANG F L, LIU P C,ZHOU X D. A review on face anti-spoofing[J/OL]. Acta Automatica Sinica[2019-12-02]. https://kns.cnki.net/kcms/detail/11.2109.TP.20190428.1046.003.html.)

Lightweight face liveness detection method based on multi-modal feature fusion

基于多模态特征融合的轻量级人脸活体检测方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	MA Jialiang, CHEN Bin, SUN Xiaofei. General object detection framework based on improved Faster R-CNN [J]. Journal of Computer Applications, 2021, 41(9): 2712-2719.
[2]	WANG Hebing, ZHANG Chunmei. Facial landmark detection based on ResNeXt with asymmetric convolution and squeeze excitation [J]. Journal of Computer Applications, 2021, 41(9): 2741-2747.
[3]	SONG Zhongshan, LIANG Jiarui, ZHENG Lu, LIU Zhenyu, TIE Jun. Remote sensing scene classification based on bidirectional gated scale feature fusion [J]. Journal of Computer Applications, 2021, 41(9): 2726-2735.
[4]	LI Kangkang, ZHANG Jing. Multi-layer encoding and decoding model for image captioning based on attention mechanism [J]. Journal of Computer Applications, 2021, 41(9): 2504-2509.
[5]	ZHANG Yongbin, CHANG Wenxin, SUN Lianshan, ZHANG Hang. Detection method of domains generated by dictionary-based domain generation algorithm [J]. Journal of Computer Applications, 2021, 41(9): 2609-2614.
[6]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[7]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[8]	MOU Changning, WANG Haipeng, ZHOU Piyu, HOU Xinhang. De novo peptide sequencing by tandem mass spectrometry based on graph convolutional neural network [J]. Journal of Computer Applications, 2021, 41(9): 2773-2779.
[9]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[10]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[11]	HUANG Chengcheng, DONG Xiaoxiao, LI Zhao. Deep pipeline 5×5 convolution method based on two-dimensional Winograd algorithm [J]. Journal of Computer Applications, 2021, 41(8): 2258-2264.
[12]	ZENG Xiangyin, ZHENG Bochuan, LIU Dan. Detection of left and right railway tracks based on deep convolutional neural network and clustering [J]. Journal of Computer Applications, 2021, 41(8): 2324-2329.
[13]	GAO Qinquan, HUANG Bingcheng, LIU Wenzhe, TONG Tong. Bamboo strip surface defect detection method based on improved CenterNet [J]. Journal of Computer Applications, 2021, 41(7): 1933-1938.
[14]	TAN Daoqiang, ZENG Cheng, QIAO Jinxia, ZHANG Jun. Shadow detection method based on hybrid attention model [J]. Journal of Computer Applications, 2021, 41(7): 2076-2081.
[15]	WU Guangli, LI Leiting, GUO Zhenzhou, WANG Chengxiang. Video summarization generation model based on improved bi-directional long short-term memory network [J]. Journal of Computer Applications, 2021, 41(7): 1908-1914.