Super-resolution and frontalization in unconstrained face images

doi:10.11772/j.issn.1001-9081.2017.11.3226

Abstract

Abstract: Concerning the problem that face recognition is affected by the factors such as attitude, occlusion, resolution and so on, a method for image super-resolution and face frontalization in unconstrained image was proposed, which could generate high-quality and standard front images. The projection matrix between the input image and 3D model was estimated to generate the standard front image. Also, through the characteristics of face symmetry, the missing pixels by occlusion and attitude could be filled. In order to avoid the loss of pixel information during the process of generating standard front image and improve the image quality, a deeply-recursive convolutional network which had 16 layers was introduced for image super-resolution. To ease the difficulty of training, two extensions were proposed:recursive-supervision and skip-connection. The experimental results on the processed LFW datasets show that it is surprisingly effective when used for face recognition and gender estimation.

Key words: face recognition, face frontalization, 3D reconstruction, image super-resolution, deeply-recursive convolutional network

摘要： 针对人脸识别算法准确率受面部姿态、遮挡、图像分辨率等因素影响的问题，提出一种超分辨率摆正的方法，作用于低质量无约束输入图像上，生成高清晰度标准正面视图。主要通过估计输入图像与3D模型间的投影矩阵，产生标准正面视图，通过人脸对称性的特点，补全由于姿态、遮挡等原因所产生的面部缺失像素。在摆正过程中，为了提高图像分辨率以及避免面部像素信息丢失，引入一个16层的深度递归卷积神经网络进行超分辨率重构；并提出两个扩展：递归监督和跳跃链接，来降低网络训练难度以及缩小模型体量。在经过处理的LFW数据集上实验表明，该方法对人脸识别和性别检测算法的性能具有显著提升作用。

关键词: 人脸识别, 人脸摆正, 3D重建, 超分辨率重构, 深度递归卷积神经网络

CLC Number:

TP389.1

SUN Qiang, TAN Xiaoyang. Super-resolution and frontalization in unconstrained face images[J]. Journal of Computer Applications, 2017, 37(11): 3226-3230.

孙强, 谭晓阳. 低质量无约束人脸图像下的超分辨率摆正[J]. 计算机应用, 2017, 37(11): 3226-3230.

References

[1] 苏楠,吴冰, 徐伟,等. 人脸识别综合技术的发展[J]. 信息安全研究,2016, 2(1):33-39.(SU N,WU B,XU W, et al. The comprehensive technology development of face recognition[J].Journal of Information Security Research, 2016,2(1):33-39.)
[2] KRIZHEVSKY A, SUTSKEVER I, HINTON G E. ImageNet classification with deep convolutional neural networks[C]//Proceedings of the 25th International Conference on Neural Information Processing Systems. Lake Tahoe:Curran Associates Inc., 2012:1097-1105.
[3] HUANG G B, MATTAR M, BERG T, et al. Labeled faces in the wild:a database for studying face recognition in unconstrained environments[EB/OL].[2016-11-20]. http://cs.umass.edu/~elm/papers/lfw.pdf.
[4] DING C, XU C, TAO D. Multi-task pose-invariant face recognition[J]. IEEE Transactions on Image Processing, 2015, 24(3):980-993.
[5] BLANZ V, VETTER T. Face recognition based on fitting a 3D morphable model[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2003, 25(9):1063-1074.
[6] BLANZ V, VETTER T. A morphable model for the synthesis of 3D faces[C]//Proceedings of the 26th Annual Conference on Computer Graphics and Interactive Techniques. New York:ACM, 1999:187-194.
[7] TAIGMAN Y, YANG M, RANZATO M, et al. DeepFace:closing the gap to human-level performance in face verification[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2014:1701-1708.
[8] ZHU Z, LUO P, WANG X, et al. Recover canonical-view faces in the wild with deep neural networks[EB/OL].[2016-11-20]. http://arxiv-web.arxiv.org/pdf/1404.3543v1.
[9] 贺飞翔, 赵启军. 基于深度学习的头部姿态估计[J]. 计算机技术与发展, 2016, 26(11):32-37.(HE F X,ZHAO Q J. Head pose estimation based on deep learning[J]. Computer Technology and Development, 2016,26(11):32-37.)
[10] 龙超. 图像超分辨率重建算法综述[J]. 科技视界,2015(13):88-89.(LONG C. Survey of super-resolution algorithms[J]. Science and Technology Vision, 2015(13):88-89.)
[11] BISWAS S, BOWYER K W, FLYNN P J. Multidimensional scaling for matching low-resolution face images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2012, 34(10):2019-2030.
[12] LI B, CHANG H, SHAN S, et al. Low-resolution face recognition via coupled locality preserving mappings[J]. IEEE Signal Processing Letters, 2010, 17(1):20-23.
[13] REN C X, DAI D Q, YAN H. Coupled kernel embedding for low resolution face image recognition[J]. IEEE Transactions on Image Processing, 2012, 21(8):3770-3783.
[14] SUN J, SUN J, XU Z, et al. Image super-resolution using gradient profile prior[C]//Proceedings of the 2008 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2008:1-8.
[15] KIM K I, KWON Y. Single-image super-resolution using sparse regression and natural image prior[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2010, 32(6):1127-1133.
[16] WANG Z, LIU D, YANG J, et al. Deep networks for image super-resolution with sparse prior[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2015:370-378.
[17] DONG C, LOY C C, HE K, et al. Image super-resolution using deep convolutional networks[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(2):295-307.
[18] SCHULTER S, LEISTNER C, BISCHOF H. Fast and accurate image upscaling with super-resolution forests[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2015:3791-3799.
[19] VIOLA P, JONES M J. Robust real-time face detection[J]. International Journal of Computer Vision, 2004, 57(2):137-154.
[20] XIONG X, TORRE F D L. Supervised descent method and its applications to face alignment[J]. Computer Vision & Pattern Recognition, 2013, 9(4):532-539.
[21] RAMANAN D, ZHU X. Face detection, pose estimation, and landmark localization in the wild[C]//Proceedings of the 2012 IEEE Conference on Computer Vision and Pattern Recognition. Washington, DC:IEEE Computer Society, 2012:2879-2886.
[22] KAZEMI V, SULLIVAN J. One millisecond face alignment with an ensemble of regression trees[C]//Proceedings of the 2014 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2014:1867-1874.
[23] HASSNER T, HAREL S, PAZ E, et al. Effective face frontalization in unconstrained images[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway, NJ:IEEE, 2014:4295-4304.
[24] KUTULAKOS K N, SEITZ S M. A theory of shape by space carving[C]//Proceedings of the 7th IEEE International Conference on Computer Vision. Piscataway, NJ:IEEE, 1999:307-314.
[25] ZENG G, PARIS S, QUAN L, et al. Progressive surface reconstruction from images using a local prior[C]//Proceedings of the 10th IEEE International Conference on Computer Vision. Washington, DC:IEEE Computer Society, 2005:1230-1237.
[26] OJALA T, PIETIKÄINEN M, MÄENPÄÄ T. A generalized local binary pattern operator for multiresolution gray scale and rotation invariant texture classification[C]//Proceedings of the Second International Conference on Advances in Pattern Recognition. London:Springer-Verlag, 2001:397-406.
[27] OJALA T, PIETIKÄINEN M, MÄENPÄÄ T. Multiresolution gray-scale and rotation invariant texture classification with local binary patterns[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2002, 24(7):971-987.
[28] WOLF L, HASSNER T, TAIGMAN Y. Descriptor based methods in the wild[EB/OL].[2016-11-20]. http://www.openu.ac.il/home/hassner/projects/Patchlbp/WolfHassnerTaigman_ECCVW08.pdf.