Multi-pose feature fusion generative adversarial network based face reconstruction method

doi:10.11772/j.issn.1001-9081.2020020205

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (10): 2856-2862.DOI: 10.11772/j.issn.1001-9081.2020020205

• Artificial intelligence • Previous Articles Next Articles

Multi-pose feature fusion generative adversarial network based face reconstruction method

LIN Leping^1,2, LI Sanfeng², OUYANG Ning^1,2

1. Key Laboratory of Cognitive Radio and Information Processing, Ministry of Education;(Guilin University of Electronic Technology), Guilin Guangxi 541004, China;
2. School of Information and Communication, Guilin University of Electronic Technology, Guilin Guangxi 541004, China

Received:2020-02-28 Revised:2020-05-14 Online:2020-10-10 Published:2020-10-17
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61661017), the Surface Program of China Postdoctoral Science Foundation (2016M602923XB), the Natural Science Foundation of Guangxi (2017GXNSFBA198212), the Science and Technology Base and Talent Project of Guangxi (AD19110060), the Project of the Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education (CRKL190107, CRKL160104), the Graduate Education Innovation Program of Guilin University of Electronic Technology (2019YCXS022).

基于多姿态特征融合生成对抗网络的人脸校正方法

林乐平^1,2, 李三凤², 欧阳宁^1,2

1. 认知无线电与信息处理省部共建教育部重点实验室(桂林电子科技大学), 广西桂林 541004;
2. 桂林电子科技大学信息与通信学院, 广西桂林 541004

通讯作者: 欧阳宁
作者简介:林乐平(1980-),女,广西桂平人,副教授,博士,主要研究方向:机器学习、智能信息处理、图像信号处理;李三凤(1993-),女,安徽亳州人,硕士研究生,主要研究方向:模式识别、深度学习;欧阳宁(1972-),男,湖南宁远人,教授,硕士,主要研究方向:数字图像处理、智能信息处理。
基金资助:
国家自然科学基金资助项目（61661017）；中国博士后科学基金面上项目（2016M602923XB）；广西自然科学基金资助项目（2017GXNSFBA198212）；广西科技基地和人才专项（AD19110060）；认知无线电与信息处理教育部重点实验室资助项目（CRKL190107，CRKL160104）；桂林电子科技大学研究生教育创新计划项目（2019YCXS022）。

Abstract

Abstract: Concerning the problem that single face image is difficult to solve the large-pose profile face in face reconstruction, a face reconstruction method based on Multi-pose Feature Fusion Generative Adversarial Network (MFFGAN) was proposed. In this method, the relevant information between multiple profile faces with different poses was used for face reconstruction, and the adversarial mechanism was used to adjust network parameters. A new network was designed in the method, which consisted of a generator including multi-pose feature extraction, multi-pose feature fusion and frontal face synthesis, and a discriminator for adversarial training. In the multi-pose feature extraction module, multiple convolution layers were used to extract the multi-pose features of profile face images. In the multi-pose feature fusion module, the multi-pose features were fused into a fusion feature containing multi-pose face information. And, the fusion feature was added during the face reconstruction process in the frontal face synthesis module. Obtaining the relevant information and global structure by exploring the feature dependency between multi-pose profile face images can effectively improve the reconstruction results. Experimental results show that, compared with those of the state-of-the-art deep learning based face reconstruction methods, the contours of the frontal face recovered by the proposed method are clear, and the recognition rate of the frontal face recovered from two profile faces is increased by 1.9 percentage points on average; and the more profile faces are input, the higher the recognition rate of the recovered frontal face is, which indicates that the proposed method can effectively fuse multi-pose features to recover a clear frontal face.

Key words: multi-face reconstruction, multi-pose feature fusion, feature dependency, deep learning, generative adversarial network

摘要： 针对人脸校正中单幅图像难以解决大姿态侧脸的问题，提出一种基于多姿态特征融合生成对抗网络（MFFGAN）的人脸校正方法，利用多幅不同姿态侧脸之间的相关信息来进行人脸校正，并采用对抗机制对网络参数进行调整。该方法设计了一种新的网络，包括由多姿态特征提取、多姿态特征融合、正脸合成三个模块组成的生成器，以及用于对抗训练的判别器。多姿态特征提取模块利用多个卷积层提取侧脸图像的多姿态特征；多姿态特征融合模块将多姿态特征融合成包含多姿态侧脸信息的融合特征；而正脸合成模块在进行姿态校正的过程中加入融合特征，通过探索多姿态侧脸图像之间的特征依赖关系来获取相关信息与全局结构，可以有效提高校正结果。实验结果表明，与现有基于深度学习的人脸校正方法相比，所提方法恢复出的正脸图像不仅轮廓清晰，而且从两幅侧脸中恢复出的正脸图像的识别率平均提高了1.9个百分点，并且输入侧脸图像越多，恢复出的正脸图像的识别率越高，表明所提方法可以有效融合多姿态特征来恢复出轮廓清晰的正脸图像。

关键词: 多幅人脸校正, 多姿态特征融合, 特征依赖关系, 深度学习, 生成对抗网络

CLC Number:

TP183

LIN Leping, LI Sanfeng, OUYANG Ning. Multi-pose feature fusion generative adversarial network based face reconstruction method[J]. Journal of Computer Applications, 2020, 40(10): 2856-2862.

林乐平, 李三凤, 欧阳宁. 基于多姿态特征融合生成对抗网络的人脸校正方法[J]. 计算机应用, 2020, 40(10): 2856-2862.

References

[1] ZHOU E,CAO Z,SUN J. GridFace:face rectification via learning local homography transformations[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11220. Cham:Springer,2018:3-20.
[2] CAO J,HU Y,ZHANG H,et al. Learning a high fidelity pose invariant model for high-resolution face frontalization[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY:Curran Associates Inc.,2018:2872-2882.
[3] DING C,TAO D. Pose-invariant face recognition with homographybased normalization[J]. Pattern Recognition,2017,66:144-152.
[4] YU Y,MORA K A F,ODOBEZ J M. Robust and accurate 3D head pose estimation through 3DMM and online head model reconstruction[C]//Proceedings of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway:IEEE,2017:711-718.
[5] TRAN L,YIN X,LIU X. Representation learning by rotating your faces[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(12):3007-3021.
[6] ZHAO J,CHENG Y,XU Y,et al. Towards pose invariant face recognition in the wild[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:2207-2216.
[7] YIN X, YU X, SOHN K, et al. Towards large-pose face frontalization in the wild[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:4010-4019.
[8] KITTLER J,HUBER P,FENG Z,et al. 3D morphable face models and their applications[C]//Proceedings of the 2016 International Conference on Articulated Motion and Deformable Objects,LNCS 9756. Cham:Springer,2016:185-206.
[9] ZHANG Z,CHEN X,WANG B,et al. Face frontalization using an appearance-flow-based convolutional neural network[J]. IEEE Transactions on Image Processing,2019,28(5):2187-2199.
[10] HUANG R,ZHANG S,LI T,et al. Beyond face rotation:global and local perception GAN for photorealistic and identity preserving frontal view synthesis[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:2458-2467.
[11] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:2672-2680.
[12] SHI X,CHEN Z,WANG H,et al. Convolutional LSTM network:a machine learning approach for precipitation nowcasting[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2015:802-810.
[13] HUANG G,HU H. c-RNN:a fine-grained language model for image captioning[J]. Neural Processing Letters,2019,49(2):683-691.
[14] LIU Y,HOU D,BAO J,et al. Multi-step ahead time series forecasting for different data patterns based on LSTM recurrent neural network[C]//Proceedings of the 14th Web Information Systems and Applications Conference. Piscataway:IEEE,2017:305-310.
[15] SHRIVASTAVA A,PFISTER T,TUZEL O,et al. Learning from simulated and unsupervised images through adversarial training[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:2242-2251.
[16] CAO J,HU Y,YU B,et al. Load balanced GANs for multi-view face image synthesis[EB/OL].[2020-02-27]. http://arxiv.org/pdf/1802.07447.pdf.
[17] 徐海月, 姚乃明, 彭晓兰, 等. 基于编解码网络的多姿态人脸图像正面化方法[J]. 中国科学:信息科学,2019,49(4):450-463.(XU H Y,YAO N M,PENG X L,et al. A multi-pose face frontalization method based on encoder-decoder network[J]. SCIENTIA SINICA Informationis,2019,49(4):450-463.)

Multi-pose feature fusion generative adversarial network based face reconstruction method

基于多姿态特征融合生成对抗网络的人脸校正方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	ZHENG Zhiqiang, HU Xin, WENG Zhi, WANG Yuhe, CHENG Xi. Cattle eye image feature extraction method based on improved DenseNet [J]. Journal of Computer Applications, 2021, 41(9): 2780-2784.
[2]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[3]	DAI Yurou, YANG Qing, ZHANG Fengli, ZHOU Fan. Trajectory prediction model of social network users based on self-supervised learning [J]. Journal of Computer Applications, 2021, 41(9): 2545-2551.
[4]	CHEN Chengrui, SUN Ning, HE Shibiao, LIAO Yong. Deep learning-based joint channel estimation and equalization algorithm for C-V2X communications [J]. Journal of Computer Applications, 2021, 41(9): 2687-2693.
[5]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[6]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[7]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[8]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[9]	GUAN Qijie, ZHANG Ting, LI Deya, ZHOU Shaojing, DU Yi. Indefinite reconstruction method of spatial data based on multi-resolution generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2306-2311.
[10]	SUN Xiao, XU Jindong. Remote sensing image dehazing method based on cascaded generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2440-2444.
[11]	HE Zhenghai, XIAN Yantuan, WANG Meng, YU Zhengtao. Case reading comprehension method combining syntactic guidance and character attention mechanism [J]. Journal of Computer Applications, 2021, 41(8): 2427-2431.
[12]	LI Yafang, LIANG Ye, FENG Weiwei, ZU Baokai, KANG Yujian. Deep network embedding method based on community optimization [J]. Journal of Computer Applications, 2021, 41(7): 1956-1963.
[13]	TANG Guihua, SUN Lei, MAO Xiuqing, DAI Leyu, HU Yongjin. Generative adversarial network synthesized face detection based on deep alignment network [J]. Journal of Computer Applications, 2021, 41(7): 1922-1927.
[14]	DU Yan, LYU Liangfu, JIAO Yichen. Fuzzy prototype network based on fuzzy reasoning [J]. Journal of Computer Applications, 2021, 41(7): 1885-1890.
[15]	GAO Qinquan, HUANG Bingcheng, LIU Wenzhe, TONG Tong. Bamboo strip surface defect detection method based on improved CenterNet [J]. Journal of Computer Applications, 2021, 41(7): 1933-1938.