Single image shadow removal method based on multistage generative adversarial network

doi:10.11772/j.issn.1001-9081.2019122146

Journal of Computer Applications ›› 2020, Vol. 40 ›› Issue (8): 2378-2385.DOI: 10.11772/j.issn.1001-9081.2019122146

• Virtual reality and multimedia computing • Previous Articles Next Articles

Single image shadow removal method based on multistage generative adversarial network

ZHANG Shuping¹, WU Wen¹, WAN Yi²

1. Department of Information Engineering, Xinjiang Institute of Technology, Aksu Xinjiang 843100, China;
2. School of Electrical and Electronic Engineering, Wenzhou University, Wenzhou Zhejiang 325035, China

Received:2019-12-23 Revised:2020-03-24 Online:2020-08-10 Published:2020-05-14
Supported by:
This work is partially supported by the Technology Plan Public Project of Zhejiang Province in China(LGG18F040002), the Natural Science Foundation of Zhejiang Province (LY19F020035).

基于多阶段生成对抗网络的单幅图像阴影去除方法

张淑萍¹, 吴文¹, 万毅²

1. 新疆理工学院信息工程系, 新疆阿克苏 843100;
2. 温州大学电气与电子工程学院, 浙江温州 325035

通讯作者: 万毅(1974-),男,浙江温州人,教授,博士,主要研究方向:系统可行性、智能检测,1119764335@qq.com
作者简介:张淑萍(1974-),女,甘肃武威人,讲师,硕士,主要研究方向:图像处理、深度学习;吴文(1994-),男,湖北武汉人,硕士,主要研究方向:图像处理、深度学习。
基金资助:
浙江省基础公益研究计划项目（LGG18F040002）；浙江省自然科学基金资助项目（LY19F020035）。

Abstract

Abstract: Traditional deep learning shadow removal methods often change the pixels in non-shadow areas and cannot obtain results with smooth boundary transition. In order to solve these problems, a new multistage shadow removal framework based on Generative Adversarial Network (GAN) was proposed. Firstly, shadow mask and shadow matte of the input image were generated by multitask driven generator via shadow detection subnet and shadow matter generation subnet respectively. Secondly, under the guidance of shadow mask and shadow matte, an umbra module and a penumbra module were designed respectively to remove different types of shadows successively. Thirdly, a new compose loss function dominated by least squares loss was created to obtain a better result. Compared with state-of-the-art shadow removal methods based on deep learning, the proposed method has the Balanced Error Rate (BER) averagely reduced by 4.39%, the Structural SIMilarity index (SSIM) averagely improved by 0.44%, and the Root Mean Square Error (RMSE) averagely reduced by 13.32%. Experimental results show that the boundary transition of shadow removal result of the proposed method is smoother.

Key words: deep learning, Generative Adversarial Network (GAN), image processing, shadow removal, shadow detection

摘要： 传统的深度学习阴影去除方法常常会改变非阴影区域的像素且无法得到边界过渡自然的阴影去除结果。为了解决该问题，基于生成对抗网络（GAN）提出一种新颖的多阶段阴影去除框架。首先，多任务驱动的生成器分别通过阴影检测子网和蒙版生成子网为输入图像生成相应的阴影掩膜和阴影蒙版；其次，在阴影掩膜和阴影蒙版的引导下，分别设计全影模块和半影模块，分阶段去除图像中不同类型的阴影；然后，以最小二乘损失为主导构建一种新的组合损失函数以得到更好的结果。与最新的深度学习阴影去除方法相比，在筛选数据集上，所提方法的平衡误差率（BER）减小约4.39%，结构相似性（SSIM）提高约0.44%，像素均方根误差（RMSE）减小约13.32%。实验结果表明该方法得到的阴影去除结果边界过渡更加平滑。

关键词: 深度学习, 生成对抗网络, 图像处理, 阴影去除, 阴影检测

CLC Number:

TP391.41

ZHANG Shuping, WU Wen, WAN Yi. Single image shadow removal method based on multistage generative adversarial network[J]. Journal of Computer Applications, 2020, 40(8): 2378-2385.

张淑萍, 吴文, 万毅. 基于多阶段生成对抗网络的单幅图像阴影去除方法[J]. 计算机应用, 2020, 40(8): 2378-2385.

References

[1] GUO R, DAI Q Y, HOIEM D. Paired regions for shadow detection and removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2013, 35(12):2956-2967.
[2] KHAN S H, BENNAMOUN M, SOHEL F, et al. Automatic shadow detection and removal from a single image[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2016, 38(3):431-446.
[3] VICENTE T F Y, HOAI M, SAMMARAS D. Leave-one-out kernel optimization for shadow detection and removal[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2018, 40(3):682-695.
[4] 廖斌,吴文.区域配对引导的光照传播视频阴影去除方法[J].计算机应用,2019,39(2):556-563. (LIAO B, WU W. Video shadow removal using region matching guided by illumination transfer[J]. Journal of Computer Applications, 2019, 39(2):556-563.)
[5] ZHANG L, ZHU Y, LIAO B, et al. Video shadow removal using spatio-temporal illumination transfer[J]. Computer Graphics Forum, 2017, 36(7):125-134.
[6] SHEN L, CHUA T W, LEMAN K. Shadow optimization from structured deep edge detection[C]//Proceedings of the 2015 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2015:2067-2074.
[7] LIU F, GLEICHER M. Texture-consistent shadow removal[C]//Proceedings of the 10th European Conference on Computer Vision, LNCS 5305. Berlin:Springer, 2008:437-450.
[8] FINLAYSON G D, HORDLEY S D, DREW M S. Removing shadows from images[C]//Proceedings of the 7th European Conference on Computer Vision, LNCS 2353. Berlin:Springer, 2002:823-836.
[9] FINLAYSON G D, HORDLEY S D, LU C, et al. On the removal of shadows from images[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2006, 28(1):59-68.
[10] MOHAN A, TUMBLIN J, CHOUDHURY P. Editing soft shadows in a digital photograph[J]. IEEE Computer Graphics and Applications, 2007, 27(2):23-31.
[11] ZHANG L, YAN Q, ZHU Y, et al. Effective shadow removal via multi-scale image decomposition[J]. The Visual Computer, 2019, 35(6/7/8):1091-1104.
[12] ARBEL E, HEL-OR H. Shadow removal using intensity surfaces and texture anchor points[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2011, 33(6):1202-1216.
[13] BARRON J T, MALIK J. Shape, illumination, and reflectance from shading[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2015, 37(8):1670-1687.
[14] QU L, TIAN J, HE S, et al. DeshadowNet:a multi-context embedding deep network for shadow removal[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:2308-2316.
[15] WANG J, LI X, YANG J. Stacked conditional generative adversarial networks for jointly learning shadow detection and shadow removal[C]//Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:1788-1797.
[16] 廖斌,谭道强,吴文. 基于衰减式生成对抗网络的单幅图像阴影去除[J]. 计算机应用, 2019, 39(9):2712-2718. (LIAO B, TAN D Q, WU W. Single image shadow removal based on attenuated generative adversarial networks[J]. Journal of Computer Applications, 2019, 39(9):2712-2718.)
[17] HU X, JIANG Y, FU C W, et al. Mask-ShadowGAN:learning to remove shadows from unpaired data[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:2472-2481.
[18] GOODFELLOW I J, POUGET-ABADIE J, MIRZA M, et al. Generative adversarial nets[C]//Proceedings of the 27th International Conference on Neural Information Processing Systems. Cambridge:MIT Press, 2014:2672-2680.
[19] MIRZA M, OSINDERO S. Conditional generative adversarial nets[EB/OL].[2019-12-25]. https://arxiv.org/pdf/1411.1784.pdf.
[20] ISOLA P, ZHU J Y, ZHOU T, et al. Image-to-image translation with conditional adversarial networks[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2017:5967-5976.
[21] FAN H, HAN M, LI J. Image shadow removal using end-to-end deep convolutional neural networks[J]. Applied Sciences, 2019, 9(5):No.1009.
[22] JOHNSON J, ALAHI A, LI F F. Perceptual losses for real-time style transfer and super-resolution[C]//Proceedings of the 14th European Conference on Computer Vision, LNCS 9906. Berlin:Springer, 2016:694-711.
[23] CHO D,TAI Y W,KWEON I S. Deep convolutional neural network for natural image matting using initial alpha mattes[J]. IEEE Transactions on Image Processing,2019,28(3):1054-1067.
[24] LOFFE S, SZEGEDY C. Batch normalization:accelerating deep network training by reducing internal covariate shift[C]//Proceedings of the 32nd International Conference on Machine Learning. New York:JMLR.org, 2015:448-456.
[25] MAO X, LI Q, XIE H, et al. Least squares generative adversarial networks[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE, 2017:2813-2821.
[26] WONGSUPHASAWAT K, SMILKOV D, WEXLER J, et al. Visualizing dataflow graphs of deep learning models in TensorFlow[J]. IEEE Transactions on Visualization and Computer Graphics, 2018, 24(1):1-12.

Single image shadow removal method based on multistage generative adversarial network

基于多阶段生成对抗网络的单幅图像阴影去除方法

PDF

Knowledge

Abstract

Cite this article

share this article

References

Related Articles 15

Recommended Articles

Metrics

[1]	ZHAO Hong, KONG Dongyi. Chinese description of image content based on fusion of image feature attention and adaptive attention [J]. Journal of Computer Applications, 2021, 41(9): 2496-2503.
[2]	XU Jianglang, LI Linyan, WAN Xinjun, HU Fuyuan. Indoor scene recognition method combined with object detection [J]. Journal of Computer Applications, 2021, 41(9): 2720-2725.
[3]	ZHENG Zhiqiang, HU Xin, WENG Zhi, WANG Yuhe, CHENG Xi. Cattle eye image feature extraction method based on improved DenseNet [J]. Journal of Computer Applications, 2021, 41(9): 2780-2784.
[4]	XIE Defeng, JI Jianmin. Syntax-enhanced semantic parsing with syntax-aware representation [J]. Journal of Computer Applications, 2021, 41(9): 2489-2495.
[5]	DAI Yurou, YANG Qing, ZHANG Fengli, ZHOU Fan. Trajectory prediction model of social network users based on self-supervised learning [J]. Journal of Computer Applications, 2021, 41(9): 2545-2551.
[6]	CHEN Chengrui, SUN Ning, HE Shibiao, LIAO Yong. Deep learning-based joint channel estimation and equalization algorithm for C-V2X communications [J]. Journal of Computer Applications, 2021, 41(9): 2687-2693.
[7]	GUAN Qijie, ZHANG Ting, LI Deya, ZHOU Shaojing, DU Yi. Indefinite reconstruction method of spatial data based on multi-resolution generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2306-2311.
[8]	SUN Xiao, XU Jindong. Remote sensing image dehazing method based on cascaded generative adversarial network [J]. Journal of Computer Applications, 2021, 41(8): 2440-2444.
[9]	HE Zhenghai, XIAN Yantuan, WANG Meng, YU Zhengtao. Case reading comprehension method combining syntactic guidance and character attention mechanism [J]. Journal of Computer Applications, 2021, 41(8): 2427-2431.
[10]	CAO Yuhong, XU Hai, LIU Sun'ao, WANG Zixiao, LI Hongliang. Review of deep learning-based medical image segmentation [J]. Journal of Computer Applications, 2021, 41(8): 2273-2287.
[11]	QIN Binbin, PENG Liangkang, LU Xiangming, QIAN Jiangbo. Research progress on driver distracted driving detection [J]. Journal of Computer Applications, 2021, 41(8): 2330-2337.
[12]	GAO Qinquan, HUANG Bingcheng, LIU Wenzhe, TONG Tong. Bamboo strip surface defect detection method based on improved CenterNet [J]. Journal of Computer Applications, 2021, 41(7): 1933-1938.
[13]	LI Yafang, LIANG Ye, FENG Weiwei, ZU Baokai, KANG Yujian. Deep network embedding method based on community optimization [J]. Journal of Computer Applications, 2021, 41(7): 1956-1963.
[14]	HOU Xiaohan, JIN Guodong, TAN Lining, XUE Yuanliang. Synthetic aperture radar ship detection method based on self-adaptive and optimal features [J]. Journal of Computer Applications, 2021, 41(7): 2150-2155.
[15]	TANG Guihua, SUN Lei, MAO Xiuqing, DAI Leyu, HU Yongjin. Generative adversarial network synthesized face detection based on deep alignment network [J]. Journal of Computer Applications, 2021, 41(7): 1922-1927.