Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (3): 842-846.DOI: 10.11772/j.issn.1001-9081.2019071283

• Virtual reality and multimedia computing • Previous Articles     Next Articles

Portrait inpainting based on generative adversarial networks

YUAN Linjun1,2, JIANG Min1,2, LUO Dunlang1,2, JIANG Jiajun1,2, GUO Jia1,2   

  1. 1. College of Computer Science and Technology, Wuhan University of Science and Technology, Wuhan Hubei 430065, China;
    2. Hubei Province Key Laboratory of Intelligent Information Processing and Real-time Industrial System;(Wuhan University of Science and Technology), Wuhan Hubei 430065, China
  • Received:2019-07-23 Revised:2019-11-11 Online:2020-03-10 Published:2019-11-20
  • Supported by:
    This work is partially supported by the National Natural Science Foundation of China (41571396).

基于生成对抗网络的人像修复

袁琳君1,2, 蒋旻1,2, 罗敦浪1,2, 江佳俊1,2, 郭嘉1,2   

  1. 1. 武汉科技大学 计算机科学与技术学院, 武汉 430065;
    2. 智能信息处理与实时工业系统湖北省重点实验室(武汉科技大学), 武汉 430065
  • 通讯作者: 蒋旻
  • 作者简介:袁琳君(1994-),女,湖北襄阳人,硕士研究生,主要研究方向:计算机视觉、深度学习;蒋旻(1975-),女,湖南隆回人,教授,博士,主要研究方向:计算机视觉、机器人自动导航;罗敦浪(1994-),男,湖北武汉人,硕士研究生,主要研究方向:计算机视觉、深度学习;江佳俊(1996-),男,湖北天门人,硕士研究生,主要研究方向:计算机视觉、深度学习;郭嘉(1996-),女,河南安阳人,硕士研究生,主要研究方向:计算机视觉、深度学习。
  • 基金资助:
    国家自然科学基金资助项目(41571396)。

Abstract: Portrait inpainting was widely used in the photo editing based on image rendering and computational photography. A lot of factors including the variety in clothing, different body types such as tall, short, fat and thin size, the high freedom degree of human body pose, bring difficulties to portrait inpainting. Therefore, an efficient portrait inpainting method based on Generating Adversarial Network (GAN) was proposed. The algorithm consists two stages. During the first stage, the image was roughly inpainted based on an encoder-decoder network, and then the body pose information in the image was estimated. During the second stage, the portrait was accurately inpainted based on the pose information and GAN. Besides, the key points of the portrait pose were connected by using portrait pose information to form the pose framework and perform the dilation operation, and the portrait pose mask was obtained. Thereby, a portrait pose loss function was constructed for network training. The experimental results show that: compared with the Contextual Attention inpainting method, the proposed method has the SSIM (Structural SIMilarity index) increased by one percentage point. The method, by adding the portrait pose information into the portrait inpainting process, effectively constrains the solution space range of portrait data in the zone to be inpainted, and strengthens the network's attention to the portrait pose information.

Key words: portrait pose information, Generative Adversarial Network (GAN), image inpainting, pose mask, portrait pose loss

摘要: 人像修复广泛用于基于图像渲染和计算摄影的照片编辑。针对衣着的不同、高矮胖瘦的区别以及姿态的高自由度等因素给人像修复带来的困难,提出了一种基于生成对抗网络(GAN)的高效人像修复方法。算法分为两阶段:第一阶段基于编码器-解码器网络粗略修复图像,然后估计其中人体姿态信息;第二阶段基于姿态信息和GAN来精确修复人像。利用人像姿态信息来连接人像姿态关键点,形成姿态框架并执行膨胀操作,得到人像姿态掩码,以此构造人像姿态损失函数进行网络训练。实验结果表明,与Contextual Attention修复方法相比,所提方法的修复结果在结构相似度(SSIM)上提升了1%。该方法将人像姿态信息加入到修复过程中,有效地约束了待修复区域人像数据的解空间范围,加强了网络对人像姿态信息的关注程度。

关键词: 人像姿态信息, 生成对抗网络, 图像修复, 姿态掩码, 人像姿态损失

CLC Number: