Abstract:

Aiming at the problem that the details are not realistic enough when images with random irregular masks and complex semantic content were repaired by existing models， an image inpainting model of dual-discriminator generative adversarial network based on gated convolution and SENet（Squeeze and Excitation Network） was proposed. Firstly， the damaged image and masks were input into the coarse network composed of several gated convolution stacks， Squeeze and Excitation （SE） attention was added during upsampling， and L1 reconstruction loss was applied to obtain a preliminary repair map. Secondly， the preliminary repair result was input into the refine network， which was composed of several gated convolution blocks and SE attention blocks， reconstruction loss， perceptual loss and adversarial loss were combined to improve important features and details， and the repair result of the refine network was covered by the intact area of ??the damaged image to obtain the completed repair result. Finally， the dual-discriminator network structure was used for training， so that the output of the refine network and the completed result were more realistic. Experimental results on celebA dataset show that the inpainting result of the proposed model for images with large-area irregular masks achieves 27.39 dB on Peak Signal-to-Noise Ratio （PSNR） which is 6.74% higher than partial convolution， and 0.921 6 on Structural Similarity Index Meaturement （SSIM） which is 2.95% higher than partial convolution. Experimental results show that SE attention and dual discriminator help to improve the details of image inpainting.

Key words: gated convolution, dual-discriminator, Generative Adversarial Network (GAN), image inpainting, Squeeze and Excitation (SE) attention

摘要：

针对现有模型修复带有随机不规则掩码且语义内容复杂的图片时细节不够真实这一问题，提出了一种基于门控卷积和SENet的双判别生成对抗网络图像修复模型。首先，将破损图片掩码输入由若干门控卷积堆叠成的粗网络中，在上采样时添加通道注意力（SE），结合L1重建损失，得到初步修复图；然后，将初步修复图输入精细网络，精细网络由若干门控卷积块和通道注意力块构成，结合重构损失、感知损失和对抗损失完善重要特征和细节，将破损图像的完好区域覆盖到精细网络的修复图上，得到完成修复的图片；最后，使用双判别网络结构进行训练，使精细网络的输出与完成修复的图片更加真实。在celebA数据集上进行实验，所提模型对带有大面积不规则掩码图片的修复结果在峰值信噪比（PSNR）上达到了27.39 dB，相较于部分卷积提升了6.74%，在结构相似性（SSIM）上达到了0.921 6，较部分卷积提升了2.95%。实验结果表明，引入通道注意力和双判别结构有助于提升图像修复的细节。

关键词: 门控卷积, 双判别器, 生成对抗网络, 图像修复, 通道注意力