基于多姿态特征融合生成对抗网络的人脸校正方法

doi:10.11772/j.issn.1001-9081.2020020205

计算机应用 ›› 2020, Vol. 40 ›› Issue (10): 2856-2862.DOI: 10.11772/j.issn.1001-9081.2020020205

基于多姿态特征融合生成对抗网络的人脸校正方法

林乐平^1,2, 李三凤², 欧阳宁^1,2

1. 认知无线电与信息处理省部共建教育部重点实验室(桂林电子科技大学), 广西桂林 541004;
2. 桂林电子科技大学信息与通信学院, 广西桂林 541004

收稿日期:2020-02-28 修回日期:2020-05-14 出版日期:2020-10-10 发布日期:2020-10-17
通讯作者: 欧阳宁
作者简介:林乐平(1980-),女,广西桂平人,副教授,博士,主要研究方向:机器学习、智能信息处理、图像信号处理;李三凤(1993-),女,安徽亳州人,硕士研究生,主要研究方向:模式识别、深度学习;欧阳宁(1972-),男,湖南宁远人,教授,硕士,主要研究方向:数字图像处理、智能信息处理。
基金资助:
国家自然科学基金资助项目（61661017）；中国博士后科学基金面上项目（2016M602923XB）；广西自然科学基金资助项目（2017GXNSFBA198212）；广西科技基地和人才专项（AD19110060）；认知无线电与信息处理教育部重点实验室资助项目（CRKL190107，CRKL160104）；桂林电子科技大学研究生教育创新计划项目（2019YCXS022）。

Multi-pose feature fusion generative adversarial network based face reconstruction method

LIN Leping^1,2, LI Sanfeng², OUYANG Ning^1,2

1. Key Laboratory of Cognitive Radio and Information Processing, Ministry of Education;(Guilin University of Electronic Technology), Guilin Guangxi 541004, China;
2. School of Information and Communication, Guilin University of Electronic Technology, Guilin Guangxi 541004, China

Received:2020-02-28 Revised:2020-05-14 Online:2020-10-10 Published:2020-10-17
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61661017), the Surface Program of China Postdoctoral Science Foundation (2016M602923XB), the Natural Science Foundation of Guangxi (2017GXNSFBA198212), the Science and Technology Base and Talent Project of Guangxi (AD19110060), the Project of the Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education (CRKL190107, CRKL160104), the Graduate Education Innovation Program of Guilin University of Electronic Technology (2019YCXS022).

摘要/Abstract

摘要： 针对人脸校正中单幅图像难以解决大姿态侧脸的问题，提出一种基于多姿态特征融合生成对抗网络（MFFGAN）的人脸校正方法，利用多幅不同姿态侧脸之间的相关信息来进行人脸校正，并采用对抗机制对网络参数进行调整。该方法设计了一种新的网络，包括由多姿态特征提取、多姿态特征融合、正脸合成三个模块组成的生成器，以及用于对抗训练的判别器。多姿态特征提取模块利用多个卷积层提取侧脸图像的多姿态特征；多姿态特征融合模块将多姿态特征融合成包含多姿态侧脸信息的融合特征；而正脸合成模块在进行姿态校正的过程中加入融合特征，通过探索多姿态侧脸图像之间的特征依赖关系来获取相关信息与全局结构，可以有效提高校正结果。实验结果表明，与现有基于深度学习的人脸校正方法相比，所提方法恢复出的正脸图像不仅轮廓清晰，而且从两幅侧脸中恢复出的正脸图像的识别率平均提高了1.9个百分点，并且输入侧脸图像越多，恢复出的正脸图像的识别率越高，表明所提方法可以有效融合多姿态特征来恢复出轮廓清晰的正脸图像。

关键词: 多幅人脸校正, 多姿态特征融合, 特征依赖关系, 深度学习, 生成对抗网络

Abstract: Concerning the problem that single face image is difficult to solve the large-pose profile face in face reconstruction, a face reconstruction method based on Multi-pose Feature Fusion Generative Adversarial Network (MFFGAN) was proposed. In this method, the relevant information between multiple profile faces with different poses was used for face reconstruction, and the adversarial mechanism was used to adjust network parameters. A new network was designed in the method, which consisted of a generator including multi-pose feature extraction, multi-pose feature fusion and frontal face synthesis, and a discriminator for adversarial training. In the multi-pose feature extraction module, multiple convolution layers were used to extract the multi-pose features of profile face images. In the multi-pose feature fusion module, the multi-pose features were fused into a fusion feature containing multi-pose face information. And, the fusion feature was added during the face reconstruction process in the frontal face synthesis module. Obtaining the relevant information and global structure by exploring the feature dependency between multi-pose profile face images can effectively improve the reconstruction results. Experimental results show that, compared with those of the state-of-the-art deep learning based face reconstruction methods, the contours of the frontal face recovered by the proposed method are clear, and the recognition rate of the frontal face recovered from two profile faces is increased by 1.9 percentage points on average; and the more profile faces are input, the higher the recognition rate of the recovered frontal face is, which indicates that the proposed method can effectively fuse multi-pose features to recover a clear frontal face.

Key words: multi-face reconstruction, multi-pose feature fusion, feature dependency, deep learning, generative adversarial network

中图分类号:

TP183

林乐平, 李三凤, 欧阳宁. 基于多姿态特征融合生成对抗网络的人脸校正方法[J]. 计算机应用, 2020, 40(10): 2856-2862.

LIN Leping, LI Sanfeng, OUYANG Ning. Multi-pose feature fusion generative adversarial network based face reconstruction method[J]. Journal of Computer Applications, 2020, 40(10): 2856-2862.

参考文献

[1] ZHOU E,CAO Z,SUN J. GridFace:face rectification via learning local homography transformations[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11220. Cham:Springer,2018:3-20.
[2] CAO J,HU Y,ZHANG H,et al. Learning a high fidelity pose invariant model for high-resolution face frontalization[C]//Proceedings of the 32nd International Conference on Neural Information Processing Systems. Red Hook, NY:Curran Associates Inc.,2018:2872-2882.
[3] DING C,TAO D. Pose-invariant face recognition with homographybased normalization[J]. Pattern Recognition,2017,66:144-152.
[4] YU Y,MORA K A F,ODOBEZ J M. Robust and accurate 3D head pose estimation through 3DMM and online head model reconstruction[C]//Proceedings of the 12th IEEE International Conference on Automatic Face and Gesture Recognition. Piscataway:IEEE,2017:711-718.
[5] TRAN L,YIN X,LIU X. Representation learning by rotating your faces[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2019,41(12):3007-3021.
[6] ZHAO J,CHENG Y,XU Y,et al. Towards pose invariant face recognition in the wild[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:2207-2216.
[7] YIN X, YU X, SOHN K, et al. Towards large-pose face frontalization in the wild[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:4010-4019.
[8] KITTLER J,HUBER P,FENG Z,et al. 3D morphable face models and their applications[C]//Proceedings of the 2016 International Conference on Articulated Motion and Deformable Objects,LNCS 9756. Cham:Springer,2016:185-206.
[9] ZHANG Z,CHEN X,WANG B,et al. Face frontalization using an appearance-flow-based convolutional neural network[J]. IEEE Transactions on Image Processing,2019,28(5):2187-2199.
[10] HUANG R,ZHANG S,LI T,et al. Beyond face rotation:global and local perception GAN for photorealistic and identity preserving frontal view synthesis[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:2458-2467.
[11] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:2672-2680.
[12] SHI X,CHEN Z,WANG H,et al. Convolutional LSTM network:a machine learning approach for precipitation nowcasting[C]//Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2015:802-810.
[13] HUANG G,HU H. c-RNN:a fine-grained language model for image captioning[J]. Neural Processing Letters,2019,49(2):683-691.
[14] LIU Y,HOU D,BAO J,et al. Multi-step ahead time series forecasting for different data patterns based on LSTM recurrent neural network[C]//Proceedings of the 14th Web Information Systems and Applications Conference. Piscataway:IEEE,2017:305-310.
[15] SHRIVASTAVA A,PFISTER T,TUZEL O,et al. Learning from simulated and unsupervised images through adversarial training[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:2242-2251.
[16] CAO J,HU Y,YU B,et al. Load balanced GANs for multi-view face image synthesis[EB/OL].[2020-02-27]. http://arxiv.org/pdf/1802.07447.pdf.
[17] 徐海月, 姚乃明, 彭晓兰, 等. 基于编解码网络的多姿态人脸图像正面化方法[J]. 中国科学:信息科学,2019,49(4):450-463.(XU H Y,YAO N M,PENG X L,et al. A multi-pose face frontalization method based on encoder-decoder network[J]. SCIENTIA SINICA Informationis,2019,49(4):450-463.)

基于多姿态特征融合生成对抗网络的人脸校正方法

Multi-pose feature fusion generative adversarial network based face reconstruction method

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	郑志强, 胡鑫, 翁智, 王雨禾, 程曦. 基于改进DenseNet的牛眼图像特征提取方法[J]. 计算机应用, 2021, 41(9): 2780-2784.
[2]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[3]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[4]	陈成瑞, 孙宁, 何世彪, 廖勇. 面向C-V2X通信的基于深度学习的联合信道估计与均衡算法[J]. 计算机应用, 2021, 41(9): 2687-2693.
[5]	谢德峰, 吉建民. 融入句法感知表示进行句法增强的语义解析[J]. 计算机应用, 2021, 41(9): 2489-2495.
[6]	代雨柔, 杨庆, 张凤荔, 周帆. 基于自监督学习的社交网络用户轨迹预测模型[J]. 计算机应用, 2021, 41(9): 2545-2551.
[7]	管其杰, 张挺, 李德亚, 周绍景, 杜奕. 基于多分辨率生成对抗网络的空间数据不确定性重建方法[J]. 计算机应用, 2021, 41(8): 2306-2311.
[8]	孙潇, 徐金东. 基于级联生成对抗网络的遥感图像去雾方法[J]. 计算机应用, 2021, 41(8): 2440-2444.
[9]	何正海, 线岩团, 王蒙, 余正涛. 融合句法指导与字符注意力机制的案情阅读理解方法[J]. 计算机应用, 2021, 41(8): 2427-2431.
[10]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 计算机应用, 2021, 41(8): 2273-2287.
[11]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[12]	高钦泉, 黄炳城, 刘文哲, 童同. 基于改进CenterNet的竹条表面缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1933-1938.
[13]	汤桂花, 孙磊, 毛秀青, 戴乐育, 胡永进. 基于深度对齐网络的生成对抗网络伪造人脸检测[J]. 计算机应用, 2021, 41(7): 1922-1927.
[14]	李亚芳, 梁烨, 冯韦玮, 祖宝开, 康玉健. 基于社区优化的深度网络嵌入方法[J]. 计算机应用, 2021, 41(7): 1956-1963.
[15]	杜炎, 吕良福, 焦一辰. 基于模糊推理的模糊原型网络[J]. 计算机应用, 2021, 41(7): 1885-1890.