Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (10): 2856-2862.DOI: 10.11772/j.issn.1001-9081.2020020205

• Artificial intelligence • Previous Articles     Next Articles

Multi-pose feature fusion generative adversarial network based face reconstruction method

LIN Leping1,2, LI Sanfeng2, OUYANG Ning1,2   

  1. 1. Key Laboratory of Cognitive Radio and Information Processing, Ministry of Education;(Guilin University of Electronic Technology), Guilin Guangxi 541004, China;
    2. School of Information and Communication, Guilin University of Electronic Technology, Guilin Guangxi 541004, China
  • Received:2020-02-28 Revised:2020-05-14 Online:2020-10-10 Published:2020-10-17
  • Supported by:
    This work is partially supported by the National Natural Science Foundation of China (61661017), the Surface Program of China Postdoctoral Science Foundation (2016M602923XB), the Natural Science Foundation of Guangxi (2017GXNSFBA198212), the Science and Technology Base and Talent Project of Guangxi (AD19110060), the Project of the Key Laboratory of Cognitive Radio and Information Processing of Ministry of Education (CRKL190107, CRKL160104), the Graduate Education Innovation Program of Guilin University of Electronic Technology (2019YCXS022).

基于多姿态特征融合生成对抗网络的人脸校正方法

林乐平1,2, 李三凤2, 欧阳宁1,2   

  1. 1. 认知无线电与信息处理省部共建教育部重点实验室(桂林电子科技大学), 广西 桂林 541004;
    2. 桂林电子科技大学 信息与通信学院, 广西 桂林 541004
  • 通讯作者: 欧阳宁
  • 作者简介:林乐平(1980-),女,广西桂平人,副教授,博士,主要研究方向:机器学习、智能信息处理、图像信号处理;李三凤(1993-),女,安徽亳州人,硕士研究生,主要研究方向:模式识别、深度学习;欧阳宁(1972-),男,湖南宁远人,教授,硕士,主要研究方向:数字图像处理、智能信息处理。
  • 基金资助:
    国家自然科学基金资助项目(61661017);中国博士后科学基金面上项目(2016M602923XB);广西自然科学基金资助项目(2017GXNSFBA198212);广西科技基地和人才专项(AD19110060);认知无线电与信息处理教育部重点实验室资助项目(CRKL190107,CRKL160104);桂林电子科技大学研究生教育创新计划项目(2019YCXS022)。

Abstract: Concerning the problem that single face image is difficult to solve the large-pose profile face in face reconstruction, a face reconstruction method based on Multi-pose Feature Fusion Generative Adversarial Network (MFFGAN) was proposed. In this method, the relevant information between multiple profile faces with different poses was used for face reconstruction, and the adversarial mechanism was used to adjust network parameters. A new network was designed in the method, which consisted of a generator including multi-pose feature extraction, multi-pose feature fusion and frontal face synthesis, and a discriminator for adversarial training. In the multi-pose feature extraction module, multiple convolution layers were used to extract the multi-pose features of profile face images. In the multi-pose feature fusion module, the multi-pose features were fused into a fusion feature containing multi-pose face information. And, the fusion feature was added during the face reconstruction process in the frontal face synthesis module. Obtaining the relevant information and global structure by exploring the feature dependency between multi-pose profile face images can effectively improve the reconstruction results. Experimental results show that, compared with those of the state-of-the-art deep learning based face reconstruction methods, the contours of the frontal face recovered by the proposed method are clear, and the recognition rate of the frontal face recovered from two profile faces is increased by 1.9 percentage points on average; and the more profile faces are input, the higher the recognition rate of the recovered frontal face is, which indicates that the proposed method can effectively fuse multi-pose features to recover a clear frontal face.

Key words: multi-face reconstruction, multi-pose feature fusion, feature dependency, deep learning, generative adversarial network

摘要: 针对人脸校正中单幅图像难以解决大姿态侧脸的问题,提出一种基于多姿态特征融合生成对抗网络(MFFGAN)的人脸校正方法,利用多幅不同姿态侧脸之间的相关信息来进行人脸校正,并采用对抗机制对网络参数进行调整。该方法设计了一种新的网络,包括由多姿态特征提取、多姿态特征融合、正脸合成三个模块组成的生成器,以及用于对抗训练的判别器。多姿态特征提取模块利用多个卷积层提取侧脸图像的多姿态特征;多姿态特征融合模块将多姿态特征融合成包含多姿态侧脸信息的融合特征;而正脸合成模块在进行姿态校正的过程中加入融合特征,通过探索多姿态侧脸图像之间的特征依赖关系来获取相关信息与全局结构,可以有效提高校正结果。实验结果表明,与现有基于深度学习的人脸校正方法相比,所提方法恢复出的正脸图像不仅轮廓清晰,而且从两幅侧脸中恢复出的正脸图像的识别率平均提高了1.9个百分点,并且输入侧脸图像越多,恢复出的正脸图像的识别率越高,表明所提方法可以有效融合多姿态特征来恢复出轮廓清晰的正脸图像。

关键词: 多幅人脸校正, 多姿态特征融合, 特征依赖关系, 深度学习, 生成对抗网络

CLC Number: