Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network

doi:10.11772/j.issn.1001-9081.2021122091

Abstract

Abstract:

The pictures taken on hazy days have color distortion and blurry details， which will affect the quality of the pictures to a certain extent. Many deep learning based methods have good results on synthetic homogeneous haze images， but they have poor results on the real nonhomogeneous dehazing dataset introduced in the latest NTIRE （New Trends in Image Restoration and Enhancement） challenge. The main reason is that the non-uniform distribution of haze is complicated， and the texture details are easily lost in the process of dehazing. Moreover， the sample number of this dataset is limited， which is easy to lead to overfitting. Therefore， a Conditional Generative Adversarial Network with Dual-Branch generators （DB-CGAN） was proposed. Among them， in one branch， with U-net used as the basic architecture， through the strategy of "Strengthen-Operate-Subtract"， enhancement modules were added to the decoder to enhance the recovery of features in the decoder， and the dense feature fusion was used to build enough connections for non-adjacent levels. In the other branch， a multi-layer residual structure was used to speed up the training of the network， and a large number of channel attention modules were concatenated to extract more high-frequency detailed features as many as possible. Finally， a simple and efficient fusion subnet was used to fuse the two branches. In the experiment， this model is significantly better than the previous Dark Channel Prior （DCP）， All-in-One Dehazing Network （AODNet）， Gated Context Aggregation Network （GCANet）， and Multi-Scale Boosted Dehazing Network （MSBDN） dehazing models in the evaluation index Peak Signal-to-Noise Ratio （PSNR） and Structural SIMilarity （SSIM）. Experimental results show that the proposed network has better performance on nonhomogeneous dehazing datasets.

Key words: deep learning, nonhomogeneous image dehazing, Generative Adversarial Network (GAN), enhanced U-net, channel attention

摘要：

雾天拍摄的图片存在颜色失真、细节模糊等问题，会对图片的质量造成一定影响。许多基于深度学习的方法虽然在去除合成的均匀雾霾图片上具有很好的效果，但在最新的NTIRE挑战赛中引入的真实非均匀去雾数据集上效果较差。主要原因是非均匀雾霾的分布较复杂，纹理细节在去雾过程中很容易丢失，并且该数据集的样本数量有限，容易产生过拟合。因此提出了一种双分支生成器的条件生成对抗网络（DB-CGAN）。其中，一条分支以U-net为基础架构，通过“加强-整合-减去”的策略在解码器中加入增强模块，从而增强解码器中特征的恢复，并使用密集特征融合为非相邻层级建立足够的连接。另一分支使用多层残差的结构来加快网络的训练，并串联大量的通道注意力模块，以最大限度地提取更多的高频细节特征。最后，使用一个简单有效的融合子网来融合两个分支。在实验中，所提模型在评价指标峰值信噪比（PSNR）和结构相似性（SSIM）上明显优于先前的暗通道先验（DCP）、一体化去雾网络（AODNet）、门控上下文聚合网络（GCANet）、多尺度增强去雾网络（MSBDN）去雾模型。实验结果表明，所提出的网络能够在非均匀去雾数据集上具有更好的性能。

关键词: 深度学习, 非均匀图像去雾, 生成对抗网络, 增强U-net, 通道注意力

CLC Number:

TP391.4

Li’an ZHU, Hong ZHANG. Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network[J]. Journal of Computer Applications, 2023, 43(2): 567-574.

朱利安, 张鸿. 基于双分支条件生成对抗网络的非均匀图像去雾[J]. 《计算机应用》唯一官方网站, 2023, 43(2): 567-574.

Figures/Tables 12

Fig. 1 Structure of generator

Tab. 1 Structure information of residual block

网络层	输出矩阵	参数个数
ReflectionPad2d	77×102×256	0
Conv2d	75×100×256	590 080
PReLU	75×100×256	1
ReflectionPad2d	77×102×256	0
Conv2d	75×100×256	590 080
ResidualBlock	75×100×256	0

Fig. 2 Structure of DFF module

Tab. 2 Structure information of RCAB

网络层	输出矩阵	参数个数
Conv2d	1 200×1 600×32	9 248
ReLU	1 200×1 600×32	0
Conv2d	1 200×1 600×32	9 248
AdaptiveAvgPool2d	1×1×32	0
Conv2d	1×1×4	132
ReLU	1×1×4	0
Conv2d	1×1×32	160
Sigmoid	1×1×32	0
RCAB	1 200×1 600×32	0

Fig. 3 Structure of discriminator

Tab. 3 Comparison of different methods on NH-HAZE and NH-HAZE2 datasets

方法	NH-HAZE		NH-HAZE2		参数量/MB	运行时间/s
方法	PSNR/dB	SSIM	PSNR/dB	SSIM	参数量/MB	运行时间/s
OUN	17.32（100%）	0.587（100%）	17.99（100%）	0.742（100%）	29.69	0.018
EUN	17.68（102%）	0.611（104%）	18.67（103%）	0.771（103%）	140.90	0.024
AN	17.64（101%）	0.625（106%）	19.06（105%）	0.785（105%）	1.00	0.028
EUN+AN	18.12（104%）	0.639（108%）	19.21（106%）	0.788（106%）	141.90	0.066
DB⁃CGAN	18.26（105%）	0.640（109%）	19.33（107%）	0.791（106%）	141.90	0.069

Fig. 4 Dehazing performance of GAN loss function and L1 loss function

Tab. 4 Comparison of experimental results of loss function ablation

$L s m o o t h ⁃ L 1$	$L L 2$	$L G ⁃ C G A N$	$L p e r$	$L M S ⁃ S S I M$	PSNR/dB	SSIM
		√			12.21	0.233
√					19.12	0.771
√		√			19.19	0.782
√		√	√		19.28	0.783
√		√	√	√	19.33	0.791
	√	√	√	√	19.25	0.779

Tab. 4 Comparison of experimental results of loss function ablation

$L s m o o t h ⁃ L 1$	$L L 2$	$L G ⁃ C G A N$	$L p e r$	$L M S ⁃ S S I M$	PSNR/dB	SSIM
		√			12.21	0.233
√					19.12	0.771
√		√			19.19	0.782
√		√	√		19.28	0.783
√		√	√	√	19.33	0.791
	√	√	√	√	19.25	0.779

Tab. 5 Experimental results of weight of mixed loss function

$λ 1$	$λ 2$	$λ 3$	PSNR/dB	SSIM
0.001	0.2	0.005	19.33	0.791
0.010	0.2	0.050	19.31	0.784

Tab. 5 Experimental results of weight of mixed loss function

$λ 1$	$λ 2$	$λ 3$	PSNR/dB	SSIM
0.001	0.2	0.005	19.33	0.791
0.010	0.2	0.050	19.31	0.784

Tab. 6 Quantitative comparison of different methods on NH-HAZE and NH-HAZE2 datasets

方法	NH-HAZE		NH-HAZE2		参数量/MB	运行时间/s
方法	PSNR/dB	SSIM	PSNR/dB	SSIM	参数量/MB	运行时间/s
DCP^［2］	12.01（100%）	0.505（122%）	11.78（100%）	0.673（105%）		>1
AODNet^［10］	12.72（105%）	0.413（100%）	13.64（115%）	0.635（100%）	0.01	0.006
GCANet^［4］	16.12（134%）	0.579（140%）	17.11（145%）	0.763（120%）	2.68	0.184
MSBDN^［5］	18.27（152%）	0.615（148%）	18.67（158%）	0.742（116%）	140.60	0.058
MPSHAN^［14］	18.13（150%）	0.641（155%）	18.97（161%）	0.781（122%）	109.80	0.042
本文方法	18.29（152%）	0.633（153%）	19.33（164%）	0.791（124%）	141.90	0.069

Fig. 5 Qualitative comparison on NH-HAZE2 dataset

Fig. 6 Qualitative comparison on NH-HAZE dataset

References 24

1	FATTAL R. Single image dehazing［J］. ACM Transactions on Graphics， 2008， 27（3）： 1-9. 10.1145/1360612.1360671
2	HE K M， SUN J， TANG X O. Single image haze removal using dark channel prior［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2011， 33（12）： 2341-2353. 10.1109/tpami.2010.168
3	CAI B L， XU X M， JIA K， et al. DehazeNet： an end-to-end system for single image haze removal［J］. IEEE Transactions on Image Processing， 2016， 25（11）： 5187-5198. 10.1109/tip.2016.2598681
4	CHEN D D， HE M M， FAN Q N， et al. Gated context aggregation network for image dehazing and deraining［C］// Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2019： 1375-1383. 10.1109/wacv.2019.00151
5	DONG H， PAN J S， XIANG L， et al. Multi-Scale boosted dehazing network with dense feature fusion［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2154-2164. 10.1109/cvpr42600.2020.00223
6	GOODFELLOW I J， POUGHT-ABADIE J， MIRZA M， et al. Generative adversarial nets［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. Cambridge： MIT Press， 2014： 2672-2680.
7	边小勇，江沛龄，赵敏，等. 基于多分支神经网络模型的弱监督细粒度图像分类方法［J］. 计算机应用， 2020， 40（5）： 1295-1300.
	BIAN X Y， JIANG P L， ZHAO M， et al. Multi-branch neural network model based weakly supervised fine-grained image classification method［J］. Journal of Computer Applications， 2020， 40（5）： 1295-1300.
8	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
9	ZHU Q S， MAI J M， SHAO L. A fast single image haze removal algorithm using color attenuation prior［J］. IEEE Transactions on Image Processing， 2015， 24（11）： 3522-3533. 10.1109/tip.2015.2446191
10	LI B Y， PENG X L， WANG Z Y， et al. AOD-Net： all-in-one dehazing network［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 4780-4788. 10.1109/iccv.2017.511
11	QIN X， WANG Z L， BAI Y C， et al. FFA-Net： feature fusion attention network for single image dehazing［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 11908-11915. 10.1609/aaai.v34i07.6865
12	DONG Y， LIU Y H， ZHANG H， et al. FD-GAN： generative adversarial networks with fusion-discriminator for single image dehazing［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 10729-10736. 10.1609/aaai.v34i07.6701
13	ANCUTI C O， ANCUTI C， VASLUIANU F A， et al. NTIRE 2021 nonhomogeneous dehazing challenge report［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2021： 627-646.
14	杨坤，张娟，方志军. 基于多补丁和多尺度层级聚合网络的快速非均匀图像去雾［J］. 计算机科学， 2021， 48（11）： 250-257. 10.11896/jsjkx.200900058
	YANG K， ZHANG J， FANG Z J. Multi-patch and multi-scale hierarchical aggregation network for fast nonhomogeneous image dehazing［J］. Computer Science， 2021， 48（11）： 250-257. 10.11896/jsjkx.200900058
15	MIRZA M， OSINDERO S. Conditional generative adversarial nets［EB/OL］. （2014-11-06）［2021-11-21］..
16	肖进胜，申梦瑶，雷俊锋，等. 基于生成对抗网络的雾霾场景图像转换算法［J］. 计算机学报， 2020， 43（1）： 165-176. 10.11897/SP.J.1016.2020.00165
	XIAO J S， SHEN M Y， LEI J F， et al. Image conversion algorithm for haze scene based on generative adversarial networks［J］. Chinese Journal of Computers， 2020， 43（1）： 165-176. 10.11897/SP.J.1016.2020.00165
17	LEDIG C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
18	ZHANG H， SINDAGI V， PATEL V M. Image de-raining using a conditional generative adversarial network［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（11）： 3943-3956. 10.1109/tcsvt.2019.2920407
19	ZHANG Y L， LI K P， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 294-310.
20	ISOLA P， ZHU J Y， ZHOU T H， et al. Image-to-image translation with conditional adversarial networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 5967-5976. 10.1109/cvpr.2017.632
21	JOHNSON J， ALAHI A， LI F F. Perceptual losses for real-time style transfer and super-resolution［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9906. Cham： Springer， 2016： 694-711.
22	WANG Z， SIMONCELLI E P， BOVIK A C. Multiscale structural similarity for image quality assessment［C］// Proceedings of the 37th Asilomar Conference on Signals， Systems and Computers - Volume 2. Piscataway： IEEE， 2003： 1398-1402. 10.1109/acssc.2003.1292181
23	FU M H， LIU H， YU Y K， et al. DW-GAN： a discrete wavelet transform GAN for NonHomogeneous Dehazing［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2021： 203-212. 10.1109/cvprw53098.2021.00029
24	ANCUTI C O， ANCUTI C， VASLUIANU F A， et al. NTIRE 2020 Challenge on NonHomogeneous Dehazing［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 2029-2044.

[1]	Gang CHEN, Yongwei LIAO, Zhenguo YANG, Wenying LIU. Image inpainting algorithm of multi-scale generative adversarial network based on multi-feature fusion [J]. Journal of Computer Applications, 2023, 43(2): 536-544.
[2]	Ping WANG, Nan CHEN, Lei LU. Fall detection algorithm based on scene prior and attention guidance [J]. Journal of Computer Applications, 2023, 43(2): 529-535.
[3]	Keyou GUO, Xue LI, Min YANG. Real‑time detection method of traffic information based on lightweight YOLOv4 [J]. Journal of Computer Applications, 2023, 43(1): 74-80.
[4]	Zhijun SHEN, Lina MU, Jing GAO, Yuanhang SHI, Zhiqiang LIU. Review of fine-grained image categorization [J]. Journal of Computer Applications, 2023, 43(1): 51-60.
[5]	Youxin WANG, Bin CHEN. Print defect detection method based on deep comparison network [J]. Journal of Computer Applications, 2023, 43(1): 250-258.
[6]	Yangping LIN, Jia LIU, Pei CHEN, Mingshu ZHANG, Xiaoyuan YANG. Semi-generative video steganography scheme based on deep convolutional generative adversarial net [J]. Journal of Computer Applications, 2023, 43(1): 169-175.
[7]	Ziqi HU, Kai XIE, Chang WEN, Meiran LI, Jianbiao HE. Low dose CT image enhancement based on generative adversarial network [J]. Journal of Computer Applications, 2023, 43(1): 280-288.
[8]	Jun ZHANG, Pengli WU, Lukui SHI, Jin SHI, Bin PAN. Deep learning model for multi-station temperature prediction combined with MOD11A1 and surface meteorological station data [J]. Journal of Computer Applications, 2023, 43(1): 321-328.
[9]	Jinghu LI, Qianguo XING, Xiangyang ZHENG, Lin LI, Lili WANG. Noctiluca scintillans red tide extraction method from UAV images based on deep learning [J]. Journal of Computer Applications, 2022, 42(9): 2969-2974.
[10]	Jiaxuan WEI, Shikang DU, Zhixuan YU, Ruisheng ZHANG. Review of white-box adversarial attack technologies in image classification [J]. Journal of Computer Applications, 2022, 42(9): 2732-2741.
[11]	Zanxia QIANG, Xianfu BAO. Residual attention deraining network based on convolutional long short-term memory [J]. Journal of Computer Applications, 2022, 42(9): 2858-2864.
[12]	Jinghan YIN, Shaojun QU, Zekai YAO, Xuanye HU, Xiaoyu QIN, Pujing HUA. Traffic sign recognition model in haze weather based on YOLOv5 [J]. Journal of Computer Applications, 2022, 42(9): 2876-2884.
[13]	Yajiao LIU, Haitao YU, Jiang WANG, Lifeng YU, Chunhui ZHANG. Surface detection algorithm of multi-shape small defects for section steel based on deep learning [J]. Journal of Computer Applications, 2022, 42(8): 2601-2608.
[14]	Yining WANG, Qingshan ZHAO, Pinle QIN, Yulan HU, Chunmei ZONG. Super-resolution reconstruction algorithm of medical image based on lightweight dense neural network [J]. Journal of Computer Applications, 2022, 42(8): 2586-2592.
[15]	Xianjie ZHANG, Zhiming ZHANG. Handwritten English text recognition based on convolutional neural network and Transformer [J]. Journal of Computer Applications, 2022, 42(8): 2394-2400.