基于前景语义信息的图像着色算法

doi:10.11772/j.issn.1001-9081.2020081184

计算机应用 ›› 2021, Vol. 41 ›› Issue (7): 2048-2053.DOI: 10.11772/j.issn.1001-9081.2020081184

所属专题：多媒体计算与计算机仿真

• 多媒体计算与计算机仿真 • 上一篇下一篇

基于前景语义信息的图像着色算法

吴丽丹^1,2, 薛雨阳³, 童同^1,2,4, 杜民², 高钦泉^1,2,4

1. 福州大学物理与信息工程学院, 福州 350108;
2. 福建省医疗器械与医药技术重点实验室(福州大学), 福州 350108;
3. 筑波大学计算机科学学院, 筑波 3058577, 日本;
4. 福建帝视信息科技有限公司, 福州 350001

收稿日期:2020-08-10 修回日期:2020-12-11 出版日期:2021-07-10 发布日期:2020-12-29
通讯作者: 童同
作者简介:吴丽丹(1996-),女,福建龙岩人,硕士研究生,主要研究方向:计算机视觉、图像处理;薛雨阳(1994-),男,福建福州人,博士研究生,主要研究方向:计算机视觉、图像处理;童同(1986-),男,安徽安庆人,教授,博士,主要研究方向:计算机视觉、医学影像处理、脑疾病辅助诊断;杜民(1955-),女,福建泉州人,教授,博士,主要研究方向:传感技术、生物医学仪器;高钦泉(1986-),男,福建福州人,副教授,博士,主要研究方向:计算机视觉、增强现实、医学影像处理。
基金资助:
国家自然科学基金资助项目（61901120）；福建省科技厅重大专项（2019YZ016006）。

Image colorization algorithm based on foreground semantic information

WU Lidan^1,2, XUE Yuyang³, TONG Tong^1,2,4, DU Min², GAO Qinquan^1,2,4

1. College of Physics and Information Engineering, Fuzhou University, Fuzhou Fujian 350108, China;
2. Key Laboratory of Medical Instrumentation & Pharmaceutical Technology of Fujian Province(Fuzhou University), Fuzhou Fujian 350108, China;
3. Department of Computer Science, University of Tsukuba, Tsukuba 3058577, Japan;
4. Imperial Vision Technology Company Limited, Fuzhou Fujian 350001, China

Received:2020-08-10 Revised:2020-12-11 Online:2021-07-10 Published:2020-12-29
Supported by:
This work is partially supported by the National Natural Science Foundation of China (61901120), the Science and Technology Major Project of Fujian Province (2019YZ016006).

摘要/Abstract

摘要： 图像可分为前景部分与背景部分，而前景往往是视觉中心。在图像着色任务上，由于前景的类别多且情况复杂，着色困难，以至于图像中的前景部分会存在着色暗淡和细节丢失等问题。针对这些问题，提出了基于前景语义信息的图像着色算法，以改善图像着色效果，达到图像整体颜色自然、内容颜色丰富的目的。首先利用前景子网提取前景部分的低级特征和高级特征；然后将这些特征融合到全景子网训练中，以排除背景颜色信息影响并强调前景颜色信息；最后用生成损失和像素级别的颜色损失来不断优化网络，指导生成高质量图像。实验结果表明，引入前景语义信息后，所提算法在峰值信噪比（PSNR）和感知相似度（LPIPS）上有所提升，可有效改善视觉中心区域着色中的色泽暗淡、细节丢失、对比度低等问题；相比其他算法，该算法在图像整体上取得了更自然的着色效果，在内容部分上取得了显著的改进。

关键词: 图像着色, 特征融合, 灰度图像, 前景语义信息

Abstract: An image can be divided into foreground part and background part, while the foreground is often the visual center. Due to the large categories and complex situations of foreground part, the image colorization is difficult, thus the foreground part of an image may suffer from poor colorization and detail loss problems. To solve these problems, an image colorization algorithm based on foreground semantic information was proposed to improve the image colorization effect and achieve the purpose of natural overall image color and rich content color. First, the foreground network was used to extract the low-level features and high-level features of the foreground part. Then these features were integrated into the foreground subnetwork to eliminate the influence of background color information and emphasize the foreground color information. Finally, the network was continuously optimized by the generation loss and pixel-level color loss, so as to guide the generation of high-quality images. Experimental results show that after introducing the foreground semantic information, the proposed algorithm improves Peak Signal-to-Noise Ratio (PSNR) and Learned Perceptual Image Patch Similarity (LPIPS), effectively solving the problems of dull color, detail loss and low contrast in the colorization of the central visual regions; compared with other algorithms, the proposed algorithm achieves a more natural colorization effect on the overall image and a significant improvement on the content part.

Key words: image colorization, feature fusion, grayscale image, foreground semantic information

中图分类号:

TP391.4

吴丽丹, 薛雨阳, 童同, 杜民, 高钦泉. 基于前景语义信息的图像着色算法[J]. 计算机应用, 2021, 41(7): 2048-2053.

WU Lidan, XUE Yuyang, TONG Tong, DU Min, GAO Qinquan. Image colorization algorithm based on foreground semantic information[J]. Journal of Computer Applications, 2021, 41(7): 2048-2053.

参考文献

[1] GONZALEZ R C,WOODS R E. 数字图像处理[M]. 3版. 阮秋琦, 阮宇智, 等译. 北京:电子工业出版社,2017:484-486. (GONZALEZ R C,WOODS R E. Digital Image Processing[M]. 3rd ed. RUAN Q Q,RUAN Y Z,translated. Beijing:Publishing House of Electronics Industry,2017:484-486.)
[2] LEVIN A, LISCHINSKI D, WEISS Y. Colorization using optimization[J]. ACM Transactions on Graphics,2004,23(3):689-694.
[3] HEU J H,HYUN D Y,KIM C S,et al. Image and video colorization based on prioritized source propagation[C]//Proceedings of the 16th IEEE International Conference on Image Processing. Piscataway:IEEE,2009:465-468.
[4] WELSH T,ASHIKHMIN M,MUELLER K. Transferring color to greyscale images[J]. ACM Transactions on Graphics,2002,21(3):277-280.
[5] CHANG H,FRIED O,LIU Y,et al. Palette-based photo recoloring[J]. ACM Transactions on Graphics,2015,34(4):No. 139.
[6] CHEN J,SHEN Y,GAO J,et al. Language-based image editing with recurrent attentive models[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:8721-8729.
[7] CHENG Z, YANG Q, SHENG B. Deep colorization[C]//Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway:IEEE,2015:415-423.
[8] ZHANG R,ISOLA P,EFROS A A. Colorful image colorization[C]//ECCV2016:Proceedings of the 2016 European Conference on Computer Vision,LNCS 9907. Cham:Springer,2016:649-666.
[9] ZHAO J,HAN J G,SHAO L,et al. Pixelated semantic colorization[J]. International Journal of Computer Vision,2020,128(4):818-834.
[10] 张娜, 秦品乐, 曾建潮, 等. 基于密集神经网络的灰度图像着色算法[J]. 计算机应用,2019,39(6):1816-1823.(ZHANG N, QIN P L,ZENG J C,et al. Grayscale image colorization algorithm based on dense neural network[J]. Journal of Computer Applications,2019,39(6):1816-1823.)
[11] HE K,GKIOXARI G,DOLLÁR P,et al. Mask R-CNN[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:2980-2988.
[12] HE K,ZHANG X,REN S,et al. Deep residual learning for image recognition[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2016:770-778.
[13] DUTA I C,LIU L,ZHU F,et al. Improved residual networks for image and video recognition[EB/OL].[2020-05-04]. https://arxiv.org/pdf/2004.04989.pdf.
[14] GOODFELLOW I J,POUGET-ABADIE J,MIRZA M,et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press,2014:2672-2680.
[15] ARJOVSKY M,CHINTALA S,BOTTOU L,et al. Wasserstein GAN[EB/OL].[2020-05-04]. https://arxiv.org/pdf/1701.07875.pdf.
[16] GULRAJANI I,AHMED F,ARJOVSKY M,et al. Improved training of Wasserstein GANs[C]//Proceedings of the 31st International Conference on Neural Information Processing Systems. Red Hook,NY:Curran Associates Inc.,2017:5769-5779.
[17] WU J,HUANG Z,THOMA J,et al. Wasserstein divergence for GANs[C]//Proceedings of the 2018 European Conference on Computer Vision,LNCS 11209. Cham:Springer,2018:673-688.
[18] RONNEBERGER O, FISCHER P, BROX T. U-Net:convolutional networks for biomedical image segmentation[C]//Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention, LNCS 9351. Cham:Springer,2015:234-241.
[19] LIN T Y,MAIRE M,BELONGIE S,et al. Microsoft COCO:common objects in context[C]//Proceedings of the 2014 European Conference on Computer Vision,LNCS 8693. Cham:Springer, 2014:740-755.
[20] EVERINGHAM M,VAN GOOL L,WILLIAMS C K I,et al. The PASCAL Visual Object Classes (VOC) Challenge[J]. International Journal of Computer Vision,2010,88(2):303-338.
[21] ZHANG R,ISOLA P,EFROS A A,et al. The unreasonable effectiveness of deep features as a perceptual metric[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:586-595.
[22] LEDIG C,THEIS L,HUSZÁR F,et al. Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceeding of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:105-114.
[23] BEVILACQUA M,ROUMY A,GUILLEMOT C,et al. Lowcomplexity single image super-resolution based on nonnegative neighbor embedding[C]//Proceedings of the 2012 British Machine Vision Conference. Durham:BMVA Press,2012:No. 135.
[24] ISOLA P,ZHU J Y,ZHOU T,et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2017:5967-5976.
[25] YOO S,BAHNG H,CHUNG S,et al. Coloring with limited data:few-shot colorization via memory-augmented networks[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2019:11275-11284.

基于前景语义信息的图像着色算法

Image colorization algorithm based on foreground semantic information

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	周险兵, 樊小超, 任鸽, 杨勇. 基于多层次语义特征的英文作文自动评分方法[J]. 计算机应用, 2021, 41(8): 2205-2211.
[2]	王伟, 赵尔平, 崔志远, 孙浩. 基于HowNet义原和Word2vec词向量表示的多特征融合消歧方法[J]. 计算机应用, 2021, 41(8): 2193-2198.
[3]	杜炎, 吕良福, 焦一辰. 基于模糊推理的模糊原型网络[J]. 计算机应用, 2021, 41(7): 1885-1890.
[4]	章荪, 尹春勇. 基于多任务学习的时序多模态情感分析模型[J]. 计算机应用, 2021, 41(6): 1631-1639.
[5]	赖雪梅, 唐宏, 陈虹羽, 李珊珊. 基于注意力机制的特征融合-双向门控循环单元多模态情感分析[J]. 计算机应用, 2021, 41(5): 1268-1274.
[6]	卞鹏程, 郑忠龙, 李明禄, 何依然, 王天翔, 张大伟, 陈丽媛. 基于注意力融合网络的视频超分辨率重建[J]. 计算机应用, 2021, 41(4): 1012-1019.
[7]	胡屹杉, 秦品乐, 曾建潮, 柴锐, 王丽芳. 基于特征融合和动态多尺度空洞卷积的超声甲状腺分割网络[J]. 计算机应用, 2021, 41(3): 891-897.
[8]	姜倩玉, 王凤英, 贾立鹏. 基于感知哈希算法和特征融合的恶意代码检测方法[J]. 计算机应用, 2021, 41(3): 780-785.
[9]	后云龙, 朱磊, 陈琴, 吕燧栋. 基于高斯差分特征网络的显著目标检测[J]. 计算机应用, 2021, 41(3): 706-713.
[10]	邱宁佳, 王晓霞, 王鹏, 王艳春. 融合语法规则的双通道中文情感模型分析[J]. 计算机应用, 2021, 41(2): 318-323.
[11]	刘紫燕, 朱明成, 袁磊, 马珊珊, 陈霖周廷. 基于非局部关注和多重特征融合的视频行人重识别[J]. 计算机应用, 2021, 41(2): 530-536.
[12]	常峥, 罗萍, 杨波, 张晓晓. 基于HHT-MFCC和短时能量的慢性阻塞性肺病患者呼吸声识别[J]. 计算机应用, 2021, 41(2): 598-603.
[13]	孟祥瑞, 杨文忠, 王婷. 基于图文融合的情感分析研究综述[J]. 计算机应用, 2021, 41(2): 307-317.
[14]	杜培德, 严华. 基于多尺度空间注意力特征融合的人群计数网络[J]. 计算机应用, 2021, 41(2): 537-543.
[15]	许学斌, 张佳达, 刘伟, 路龙宾, 赵雨晴. 融合空间和通道特征的高精度乳腺癌分类方法[J]. 计算机应用, 2021, 41(10): 3025-3032.