Neural style transfer algorithm based on Laplacian operator and color retention

doi:10.11772/j.issn.1001-9081.2021081457

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (10): 3209-3216.DOI: 10.11772/j.issn.1001-9081.2021081457

• Multimedia computing and computer simulation • Previous Articles

Neural style transfer algorithm based on Laplacian operator and color retention

Yongqian TAN, Fanju ZENG

School of Big Data Engineering，Kaili University，Kaili Guizhou 556011，China

Received:2021-08-10 Revised:2021-12-17 Accepted:2021-12-17 Online:2022-01-07 Published:2022-10-10
Contact: Fanju ZENG
About author:TAN Yongqian, born in 1984， M. S. ， associate professor. His research interests include graphics and image processing， optoelectronic materials and devices.
ZENG Fanju, born in 1986， Ph. D. ， associate professor. Her research interests include optical image processing， optoelectronic materials and devices.
Supported by:
Basic Research Program of Guizhou Province in 2022 （Natural Science）（Qiankehe Jichu-ZK［2022］ General 526）, Guizhou Provincial Department of Education’s Youth Science and Technology Talent Growth Project （Qianjiaohe KY ［2017］ 335）, Guizhou Provincial Science and Technology Cooperation Program （Qiankehe LH ［2017］ 7161）, Special Project of "First-class Construction and Cultivation Discipline in Guizhou Province?Ethnology"(YLXKJS0071)

基于拉普拉斯算子和颜色保留的神经风格迁移算法

谭永前, 曾凡菊

凯里学院大数据工程学院，贵州凯里 556011

通讯作者: 曾凡菊
作者简介:第一联系人：谭永前（1984—），男（苗族），贵州黄平人，副教授，硕士，主要研究方向：图形图像处理、光电材料与器件
曾凡菊（1986—），女（土族），贵州黄平人，副教授，博士，主要研究方向：光学图像处理、光电材料与器件。z‑f‑j‑1@163.com
基金资助:
贵州省2022年度基础研究计划（自然科学）项目（黔科合基础-ZK［2022］一般526）;贵州省教育厅青年科技人才成长项目(黔教合KY字［2017］335);贵州省科技合作计划项目(黔科合LH字［2017］7161号);“贵州省区域内一流建设培育学科?民族学”专项课题(YLXKJS0071)

Abstract

Abstract:

There are some problems in results of the neural style transfer algorithms， such as artifacts， color loss， and blurred contour， which are negative to the overall artistic effect. Therefore， a neural style transfer algorithm based on Laplacian operator and Color Retention （LCR） was proposed. Content loss term， style loss， histogram loss， and Laplacian loss are utilized in the proposed LCR algorithm to construct the total loss function. Because histogram loss and Laplacian loss are used in the LCR algorithm， the proposed algorithm has better overall artistic effect on the stylized result images than Image Style Transfer using Convolutional Neural Networks （IST-CNN） algorithm and Deep Feature Perturbation （DFP） algorithm. Firstly， the influence of image noise on latter calculation of each loss was reduced by denoising input content image and style image. Secondly， the separation of image brightness channel L and color channel a， b was achieved by converting content image and style image from RGB space to Lab space. And the brightness information of content image was transferred to style image to preserve the color of content image. Finally， in Convolutional Neural Network （CNN）， the total loss function was iteratively optimized， and then the stylized result image was output. Compared with IST-CNN and DFP algorithms， the proposed LCR algorithm has the Peak Signal-to-Noise Ratio （PSNR） improved by 12.418 dB and 8.038 dB approximately and respectively， the Structural SIMilarity （SSIM） improved by about 0.348 06 and 0.258 54 approximately and respectively， and the Mean Square Error （MSE） decreased by about 0.653 76 and 0.296 00 respectively. Experimental results show that LCR algorithm has advantage in the overall visual effect of stylized drawing.

Key words: histogram loss, image style transfer, Laplacian operator, style loss, content loss

摘要：

针对神经风格迁移算法结果中存在伪影、颜色丢失、轮廓模糊不清等影响整体艺术效果的问题，提出了一种基于拉普拉斯算子和颜色保留（LCR）的神经风格迁移算法。所提LCR算法使用内容损失项、风格损失项、直方图损失项以及拉普拉斯损失项构建总损失函数。由于在LCR算法中使用了直方图损失项和拉普拉斯损失项，因此，LCR算法与基于卷积神经网络的图像风格迁移（IST-CNN）算法、基于深度特征扰动（DFP）算法相比，对风格化结果图有更好的整体艺术效果。首先，通过对输入内容图像和风格图像进行去噪处理，减小了图像噪声对后续各个损失项计算的影响；其次，对内容图像和风格图像进行RGB空间到Lab空间的转换，以实现图像亮度通道L和颜色通道a、b的分离，并把内容图像的亮度信息迁移到风格图像上，从而达到内容图像颜色保留的目的；最后，在卷积神经网络（CNN）中对总损失函数进行迭代优化并输出风格化结果图。与IST-CNN和DFP算法相比，所提LCR算法的峰值信噪比（PSNR）平均分别提高了约12.418 dB和8.038 dB，结构相似性（SSIM）平均分别提高了约0.348 06和0.258 54，均方差（MSE）平均分别降低了0.653 76和0.296 00。实验结果表明，LCR算法有更好的风格化绘制整体视觉效果。

关键词: 直方图损失, 图像风格迁移, 拉普拉斯算子, 风格损失, 内容损失

CLC Number:

TP391.4

Yongqian TAN, Fanju ZENG. Neural style transfer algorithm based on Laplacian operator and color retention[J]. Journal of Computer Applications, 2022, 42(10): 3209-3216.

谭永前, 曾凡菊. 基于拉普拉斯算子和颜色保留的神经风格迁移算法[J]. 《计算机应用》唯一官方网站, 2022, 42(10): 3209-3216.

Figures/Tables 12

Fig. 1 Structure of VGG model

Fig. 2 Style transfer algorithm flowchart

Fig. 3 Gray-scale images with different mean and variance but same Gram matrix

Fig. 4 Schematic diagram of stylized drawing

Tab. 1 Main parameters

参数	含义	数值	参数	含义	数值
$α$	内容损失权重	5	N	迭代次数	900
$β$	风格损失权重	500	M	学习率	10
$γ$	直方图损失权重	100

Tab. 1 Main parameters

参数	含义	数值	参数	含义	数值
$α$	内容损失权重	5	N	迭代次数	900
$β$	风格损失权重	500	M	学习率	10
$γ$	直方图损失权重	100

Fig. 5 Experiment result comparison of color retention

Fig. 6 Experiment result comparison of adding histogram loss

Fig. 7 Experiment result comparison of adding Laplacian loss

Fig. 8 Comparison of overall stylized effect of different algorithms

Tab. 2 Experimental image parameters

图像类别	图像名称	图像大小
风格图像	Cartoon	736×704
	Starrynight	1 024×640
	Muse	1 088×873
	Feathers	450×563
	Smallworld	391×500
内容图像	Girl-1	591×804
	Megan	800×587
	Goat	777×777
	Model	1 920×1 285
	Girl-2	319×384

Fig. 9 Comparison of local magnification effect

Tab. 3 Evaluation index statistics

算法	均方差/10³					峰值信噪比/dB
算法	Girl-1_Cartoon	Megan_Starrynight	Goat_Muse	Model-2_Feathers	Girl-2_Smallworld	Girl-1_Cartoon	Megan_Starrynight	Goat_Muse	Model-2_Feathers	Girl-2_Smallworld
IST-CNN^［2］	1.527	1.268	1.468	1.480	1.487	61	58.2	62.5	62.5	61.8
DFP^［7］	1.168	1.008	1.015	1.130	1.118	65	65.8	65.9	65.8	65.4
本文算法	1.057	0.687	0.856	0.651	0.710	72	71.8	75.6	73.9	74.8
算法	结构相似性					算法运行时间/s
算法	Girl-1_Cartoon	Megan_Starrynight	Goat_Muse	Model-2_Feathers	Girl-2_Smallworld	Girl-1_Cartoon	Megan_Starrynight	Goat_Muse	Model-2_Feathers	Girl-2_Smallworld
IST-CNN^［2］	0.253	0.358	0.259	0.389	0.439	88	86	94	85	56
DFP^［7］	0.336	0.416	0.368	0.458	0.567	75	80	86	80	45
本文算法	0.786	0.557	0.648	0.688	0.759	93	96	97	95	65

References 17

1	GATYS L A， ECKER A S， BETHGE M. Texture synthesis using convolutional neural networks［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2015： 262-270. 10.1109/cvpr.2016.265
2	GATYS L A， ECKER A S， BETHGE M. Image style transfer using convolutional neural networks［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：2414-2423. 10.1109/cvpr.2016.265
3	GATYS L A， BETHGE M， HERTZMANN A， et al. Preserving color in neural artistic style transfer［EB/OL］. （2016-06-19）［2021-10-07］.. 10.1109/cvpr.2017.397
4	谢斌，汪宁，范有伟. 相关对齐的总变分风格迁移新模型［J］. 中国图象图形学报， 2020， 25（2）：241-254. 10.11834/jig.190199
	XIE B， WANG N， FAN Y W. Correlation alignment total variation model and algorithm for style transfer［J］. Journal of Image and Graphics， 2020， 25（2）：241-254. 10.11834/jig.190199
5	KOTOVENKO D， SANAKOYEU A， MA P C， et al. A content transformation block for image style transfer［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 10024-10033. 10.1109/cvpr.2019.01027
6	CHENG M M， LIU X C， WANG J， et al. Structure-preserving neural style transfer［J］. IEEE Transactions on Image Processing， 2020， 29： 909-920. 10.1109/tip.2019.2936746
7	WANG Z Z， ZHAO L， CHEN H B， et al. Diversified arbitrary style transfer via deep feature perturbation［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 7786-7795. 10.1109/cvpr42600.2020.00781
8	LI S H， XU X X， NIE L Q， et al. Laplacian-steered neural style transfer［C］// Proceedings of the 25th ACM International Conference on Multimedia. New York： ACM， 2017： 1716-1724. 10.1145/3123266.3123425
9	SIMONYAN K， ZISSERMAN A. Very deep convolutional networks for large-scale image recognition［EB/OL］. （2015-04-10）［2021-10-07］..
10	MISRA D， LANDSKAPE. Mish： a self regularized non-monotonic neural activation function［C］// Proceedings of the 2020 British Machine Vision Conference. Durham： BMVA Press， 2020： No.928.
11	RISSER E， WILMOT P， BARNES C. Stable and controllable neural texture synthesis and style transfer using histogram losses［EB/OL］. （2017-02-01）［2021-10-12］..
12	杨慧炯，韩燕丽，郭芸俊. 基于亮度与色度信息的深度学习图像风格迁移算法研究［J］. 重庆理工大学学报（自然科学）， 2019， 33（7）：145-151， 159.
	YANG H J， HAN Y L， GUO Y J. Deep learning algorithm for image style transfer based on luminance and color channels［J］. Journal of Chongqing University of Technology （Natural Science）， 2019， 33（7）：145-151， 159.
13	田会娟，蔡敏鹏，关涛，等. 基于YCbCr颜色空间的Retinex低照度图像增强方法研究［J］. 光子学报， 2020， 49（2）： 173-184. 10.3788/gzxb20204902.0210002
	TIAN H J， CAI M P， GUAN T， et al Low-light image enhancement method using Retinex method based on YCbCr color space［J］. Acta Photonica Sinica， 2020， 49（2）： 173-184. 10.3788/gzxb20204902.0210002
14	曾宪华，陆宇喆，童世玥，等. 结合马尔科夫场和格拉姆矩阵特征的写实类图像风格迁移［J］. 南京大学学报（自然科学）， 2021， 57（1）：1-9.
	ZENG X H， LU Y Z， TONG S Y， et al. Photorealism style transfer combining MRFs-based and gram-based features［J］. Journal of Nanjing University （Natural Science）， 2021， 57（1）：1-9.
15	HERTZMANN A， JACOBS C E， OLIVER N， et al. Image analogies［C］// Proceedings of the 2001 28th Annual Conference on Computer Graphics and Interactive Techniques. New York： ACM， 2001：327-340. 10.1145/383259.383295
16	王明印，韦群，何国林，等. 基于拉普拉斯图像锐化算法研究［C］// 中国电子学会第十六届信息论学术年会论文集. 北京：电子工业出版社， 2009：464-468.
	WANG M Y， WEI Q， HE G L， et al. Research on the algorithm of Laplacian sharpening［C］// Proceedings of the 16th Annual Conference on Information Theory Technology of Chinese Institute of Electronics. Beijing： Publishing House of Electronics Industry， 2009：464-468.
17	贾海鹏，张云泉，龙国平，等. 基于OpenCL的拉普拉斯图像增强算法优化研究［J］. 计算机科学， 2012， 39（5）：271-277. 10.3969/j.issn.1002-137X.2012.05.065
	JIA H P， ZHANG Y Q， LONG G P， et al. Research on Laplace image enhancement algorithm optimization based on OpenCL［J］. Computer Science， 2012， 39（5）：271-277. 10.3969/j.issn.1002-137X.2012.05.065

[1]	YU Yingdong, YANG Yi, LIN Lan. Image style transfer network based on texture feature analysis [J]. Journal of Computer Applications, 2020, 40(3): 638-644.
[2]	TIAN Su-yun WANG Xiao-ming ZHAO Xue-qing. Image denoising algorithms based on Laplacian operator and image inpainting [J]. Journal of Computer Applications, 2012, 32(10): 2793-2797.
[3]	JIANG Wei DING Zhi-quan LIU Ya-wei. New image edge detection model based on fractional-order partial differention [J]. Journal of Computer Applications, 2012, 32(10): 2848-2850.
[4]	. Laplacian editing and its improvement on OpenMesh platform [J]. Journal of Computer Applications, 2009, 29(12): 3449-3452.

Neural style transfer algorithm based on Laplacian operator and color retention

基于拉普拉斯算子和颜色保留的神经风格迁移算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 12

References 17

Related Articles 4

Recommended Articles

Metrics