Real-time multi-face landmark localization algorithm based on deep residual and feature pyramid neural network

doi:10.11772/j.issn.1001-9081.2019040600

Abstract

Abstract: Most face landmark detection algorithms include two steps:face detection and face landmark localization, increasing the processing time. Aiming at the problem, a one-step and real-time algorithm for multi-face landmark localization was proposed. The corresponding heatmaps were generated as data labels by the face landmark coordinates. Deep residual network was used to realize the early feature extraction of image and feature pyramid network was used to fuse the information features representing receptive fields with different scales in different network depths. And then based on intermediate supervision, multiple landmark prediction networks were cascaded to realize the one-step coarse-to-fine facial landmark regression without face detection. With high accuracy localization, a forward propagation of the proposed algorithm only takes about 0.0075 s (133 frames per second), satisfying the requirement of real-time facial landmark localization. And the proposed algorithm has achieved the mean error of 6.06% and failure rate of 11.70% on Wider Facial Landmarks in-the-Wild (WFLW) dataset.

Key words: residual network, feature pyramid network, real-time face landmark localization, intermediate supervision

摘要： 针对人脸关键点定位算法需要分为人脸区域检测与单人脸关键点定位两个步骤，导致处理时间成倍增加的情况，提出一步到位的实时且准确的多人脸关键点定位算法。该算法将人脸关键点坐标生成对应的热度图作为数据标签，利用深度残差网络完成前期的图像特征提取，使用特征金字塔网络融合在不同网络深度中表征不同尺度感受野的信息特征，应用中间监督思想，级联多个预测网络由粗到精地一次性回归图中所有人脸的关键点，而无需人脸检测步骤。在保持高定位精度的同时，该算法完成一次前向传播只需要约0.0075 s （约每秒133帧），满足了实时人脸关键点定位的要求，且在WFLW测试集中取得了6.06%的平均误差与11.70%的错误率。

关键词: 残差网络, 特征金字塔网络, 实时人脸关键点定位, 中间监督

CLC Number:

TP391.4

XIE Jinheng, ZHANG Yansheng. Real-time multi-face landmark localization algorithm based on deep residual and feature pyramid neural network[J]. Journal of Computer Applications, 2019, 39(12): 3659-3664.

谢金衡, 张炎生. 基于深度残差和特征金字塔网络的实时多人脸关键点定位算法[J]. 计算机应用, 2019, 39(12): 3659-3664.

References

[1] SCHROFF F, KALENICHENKO D, PHILBIN J. FaceNet:a unified embedding for face recognition and clustering[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2015:815-823.
[2] ZHU Z, LUO P, WANG X, et al. Deep learning identity-preserving face space[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2013:113-120.
[3] SUN Y, WANG X, TANG X. Deep learning face representation from predicting 10000 classes[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2014:1891-1898.
[4] SUN Y, WANG X, TANG X. Hybrid deep learning for face verification[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2013:1489-1496.
[5] CAO Z, SIMON T, WEI S E, et al. Realtime multi-person 2D pose estimation using part affinity fields[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:1302-1310.
[6] OSOKIN D. Real-time 2D multi-person pose estimation on CPU:lightweight OpenPose[EB/OL].[2019-01-02]. https://arxiv.org/pdf/1811.12004.pdf.
[7] FANG H, XIE S, TAI Y W, et al. RMPE:regional multi-person pose estimation[C]//Proceedings of the 2017 IEEE Conference on Computer Vision. Piscataway:IEEE, 2017:2353-2362.
[8] LI W, WANG Z, YIN B, et al. Rethinking on multi-stage networks for human pose estimation[EB/OL].[2019-01-02]. https://arxiv.org/pdf/1901.00148.pdf.
[9] CHEN Y, WANG Z, PENG Y, et al. Cascaded pyramid network for multi-person pose estimation[C]//Proceedings of the 2018 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:7103-7112.
[10] WEI S E, RAMAKRISHNA V, KANADE T, et al. Convolutional pose machines[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:4724-4732.
[11] ZHANG K, ZHANG Z, LI Z, et al. Joint face detection and alignment using multi-task cascaded convolutional networks[J]. IEEE Signal Processing Letters, 2016, 23(10):1499-1503.
[12] LV J, SHAO X, XING J, et al. A deep regression architecture with two-stage re-initialization for high performance facial landmark detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:3691-3700.
[13] KOWALSKI M, NARUNIEC J, TRZCINSKI T. Deep alignment network:a convolutional neural network for robust face alignment[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE, 2017:2034-2043.
[14] WU W, QIAN C, YANG S, et al. Look at boundary:a boundary-aware face alignment algorithm[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:2129-2138.
[15] SAGONAS C, TZIMIROPOULOS G, ZAFEIRIOU S, et al. 300 faces in-the-wild challenge:the first facial landmark localization challenge[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision Workshops. Piscataway:IEEE, 2013:397-403.
[16] KOSTINGER M, WOHLHART P, ROTH P M, et al. Annotated facial landmarks in the wild:a large-scale, real world database for facial landmark localization[C]//Proceedings of the 2011 IEEE International Conference on Computer Vision Workshops. Piscataway:IEEE, 2011:2144-2151.
[17] BURGOS-ARTIZZU X P, PERONA P, DOLLÁR P. Robust face landmark estimation under occlusion[C]//Proceedings of the 2013 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2013:1513-1520.
[18] HE K, ZHANG X, REN S, et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[19] LIN T Y, DOLLÁR P, GIRSHICK R, et al. Feature pyramid networks for object detection[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:936-944.
[20] CAO X, WEI Y, WEN F, et al. Face alignment by explicit shape regression[J]. International Journal of Computer Vision, 2014, 107(2):177-190.
[21] XIONG X, DE LA TORRE F. Supervised descent method and its applications to face alignment[C]//Proceedings of the 2013 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2013:532-539.
[22] ZHU S, LI C, LOY C C, et al. Face alignment by coarse-to-fine shape searching[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2015:4998-5006.
[23] WU W, YANG S. Leveraging intra and inter-dataset variations for robust face alignment[C]//Proceedings of the 2017 IEEE Conference on computer vision and Pattern Recognition Workshops. Piscataway:IEEE, 2017:2096-2105.
[24] DONG X, YAN Y, OUYANG W, et al. Style aggregated network for facial landmark detection[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:379-388.