Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network

doi:10.11772/j.issn.1001-9081.2021122091

Abstract

Abstract:

The pictures taken on hazy days have color distortion and blurry details， which will affect the quality of the pictures to a certain extent. Many deep learning based methods have good results on synthetic homogeneous haze images， but they have poor results on the real nonhomogeneous dehazing dataset introduced in the latest NTIRE （New Trends in Image Restoration and Enhancement） challenge. The main reason is that the non-uniform distribution of haze is complicated， and the texture details are easily lost in the process of dehazing. Moreover， the sample number of this dataset is limited， which is easy to lead to overfitting. Therefore， a Conditional Generative Adversarial Network with Dual-Branch generators （DB-CGAN） was proposed. Among them， in one branch， with U-net used as the basic architecture， through the strategy of "Strengthen-Operate-Subtract"， enhancement modules were added to the decoder to enhance the recovery of features in the decoder， and the dense feature fusion was used to build enough connections for non-adjacent levels. In the other branch， a multi-layer residual structure was used to speed up the training of the network， and a large number of channel attention modules were concatenated to extract more high-frequency detailed features as many as possible. Finally， a simple and efficient fusion subnet was used to fuse the two branches. In the experiment， this model is significantly better than the previous Dark Channel Prior （DCP）， All-in-One Dehazing Network （AODNet）， Gated Context Aggregation Network （GCANet）， and Multi-Scale Boosted Dehazing Network （MSBDN） dehazing models in the evaluation index Peak Signal-to-Noise Ratio （PSNR） and Structural SIMilarity （SSIM）. Experimental results show that the proposed network has better performance on nonhomogeneous dehazing datasets.

Key words: deep learning, nonhomogeneous image dehazing, Generative Adversarial Network (GAN), enhanced U-net, channel attention

摘要：

雾天拍摄的图片存在颜色失真、细节模糊等问题，会对图片的质量造成一定影响。许多基于深度学习的方法虽然在去除合成的均匀雾霾图片上具有很好的效果，但在最新的NTIRE挑战赛中引入的真实非均匀去雾数据集上效果较差。主要原因是非均匀雾霾的分布较复杂，纹理细节在去雾过程中很容易丢失，并且该数据集的样本数量有限，容易产生过拟合。因此提出了一种双分支生成器的条件生成对抗网络（DB-CGAN）。其中，一条分支以U-net为基础架构，通过“加强-整合-减去”的策略在解码器中加入增强模块，从而增强解码器中特征的恢复，并使用密集特征融合为非相邻层级建立足够的连接。另一分支使用多层残差的结构来加快网络的训练，并串联大量的通道注意力模块，以最大限度地提取更多的高频细节特征。最后，使用一个简单有效的融合子网来融合两个分支。在实验中，所提模型在评价指标峰值信噪比（PSNR）和结构相似性（SSIM）上明显优于先前的暗通道先验（DCP）、一体化去雾网络（AODNet）、门控上下文聚合网络（GCANet）、多尺度增强去雾网络（MSBDN）去雾模型。实验结果表明，所提出的网络能够在非均匀去雾数据集上具有更好的性能。

关键词: 深度学习, 非均匀图像去雾, 生成对抗网络, 增强U-net, 通道注意力

CLC Number:

TP391.4

Li’an ZHU, Hong ZHANG. Nonhomogeneous image dehazing based on dual-branch conditional generative adversarial network[J]. Journal of Computer Applications, 2023, 43(2): 567-574.

朱利安, 张鸿. 基于双分支条件生成对抗网络的非均匀图像去雾[J]. 《计算机应用》唯一官方网站, 2023, 43(2): 567-574.

Figures/Tables 12

Fig. 1 Structure of generator

Tab. 1 Structure information of residual block

网络层	输出矩阵	参数个数
ReflectionPad2d	77×102×256	0
Conv2d	75×100×256	590 080
PReLU	75×100×256	1
ReflectionPad2d	77×102×256	0
Conv2d	75×100×256	590 080
ResidualBlock	75×100×256	0

Fig. 2 Structure of DFF module

Tab. 2 Structure information of RCAB

网络层	输出矩阵	参数个数
Conv2d	1 200×1 600×32	9 248
ReLU	1 200×1 600×32	0
Conv2d	1 200×1 600×32	9 248
AdaptiveAvgPool2d	1×1×32	0
Conv2d	1×1×4	132
ReLU	1×1×4	0
Conv2d	1×1×32	160
Sigmoid	1×1×32	0
RCAB	1 200×1 600×32	0

Fig. 3 Structure of discriminator

Tab. 3 Comparison of different methods on NH-HAZE and NH-HAZE2 datasets

方法	NH-HAZE		NH-HAZE2		参数量/MB	运行时间/s
方法	PSNR/dB	SSIM	PSNR/dB	SSIM	参数量/MB	运行时间/s
OUN	17.32（100%）	0.587（100%）	17.99（100%）	0.742（100%）	29.69	0.018
EUN	17.68（102%）	0.611（104%）	18.67（103%）	0.771（103%）	140.90	0.024
AN	17.64（101%）	0.625（106%）	19.06（105%）	0.785（105%）	1.00	0.028
EUN+AN	18.12（104%）	0.639（108%）	19.21（106%）	0.788（106%）	141.90	0.066
DB⁃CGAN	18.26（105%）	0.640（109%）	19.33（107%）	0.791（106%）	141.90	0.069

Fig. 4 Dehazing performance of GAN loss function and L1 loss function

Tab. 4 Comparison of experimental results of loss function ablation

$L s m o o t h ⁃ L 1$	$L L 2$	$L G ⁃ C G A N$	$L p e r$	$L M S ⁃ S S I M$	PSNR/dB	SSIM
		√			12.21	0.233
√					19.12	0.771
√		√			19.19	0.782
√		√	√		19.28	0.783
√		√	√	√	19.33	0.791
	√	√	√	√	19.25	0.779

Tab. 4 Comparison of experimental results of loss function ablation

$L s m o o t h ⁃ L 1$	$L L 2$	$L G ⁃ C G A N$	$L p e r$	$L M S ⁃ S S I M$	PSNR/dB	SSIM
		√			12.21	0.233
√					19.12	0.771
√		√			19.19	0.782
√		√	√		19.28	0.783
√		√	√	√	19.33	0.791
	√	√	√	√	19.25	0.779

Tab. 5 Experimental results of weight of mixed loss function

$λ 1$	$λ 2$	$λ 3$	PSNR/dB	SSIM
0.001	0.2	0.005	19.33	0.791
0.010	0.2	0.050	19.31	0.784

Tab. 5 Experimental results of weight of mixed loss function

$λ 1$	$λ 2$	$λ 3$	PSNR/dB	SSIM
0.001	0.2	0.005	19.33	0.791
0.010	0.2	0.050	19.31	0.784

Tab. 6 Quantitative comparison of different methods on NH-HAZE and NH-HAZE2 datasets

方法	NH-HAZE		NH-HAZE2		参数量/MB	运行时间/s
方法	PSNR/dB	SSIM	PSNR/dB	SSIM	参数量/MB	运行时间/s
DCP^［2］	12.01（100%）	0.505（122%）	11.78（100%）	0.673（105%）		>1
AODNet^［10］	12.72（105%）	0.413（100%）	13.64（115%）	0.635（100%）	0.01	0.006
GCANet^［4］	16.12（134%）	0.579（140%）	17.11（145%）	0.763（120%）	2.68	0.184
MSBDN^［5］	18.27（152%）	0.615（148%）	18.67（158%）	0.742（116%）	140.60	0.058
MPSHAN^［14］	18.13（150%）	0.641（155%）	18.97（161%）	0.781（122%）	109.80	0.042
本文方法	18.29（152%）	0.633（153%）	19.33（164%）	0.791（124%）	141.90	0.069

Fig. 5 Qualitative comparison on NH-HAZE2 dataset

Fig. 6 Qualitative comparison on NH-HAZE dataset

References 24

1	FATTAL R. Single image dehazing［J］. ACM Transactions on Graphics， 2008， 27（3）： 1-9. 10.1145/1360612.1360671
2	HE K M， SUN J， TANG X O. Single image haze removal using dark channel prior［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2011， 33（12）： 2341-2353. 10.1109/tpami.2010.168
3	CAI B L， XU X M， JIA K， et al. DehazeNet： an end-to-end system for single image haze removal［J］. IEEE Transactions on Image Processing， 2016， 25（11）： 5187-5198. 10.1109/tip.2016.2598681
4	CHEN D D， HE M M， FAN Q N， et al. Gated context aggregation network for image dehazing and deraining［C］// Proceedings of the 2019 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2019： 1375-1383. 10.1109/wacv.2019.00151
5	DONG H， PAN J S， XIANG L， et al. Multi-Scale boosted dehazing network with dense feature fusion［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2154-2164. 10.1109/cvpr42600.2020.00223
6	GOODFELLOW I J， POUGHT-ABADIE J， MIRZA M， et al. Generative adversarial nets［C］// Proceedings of the 27th International Conference on Neural Information Processing Systems - Volume 2. Cambridge： MIT Press， 2014： 2672-2680.
7	边小勇，江沛龄，赵敏，等. 基于多分支神经网络模型的弱监督细粒度图像分类方法［J］. 计算机应用， 2020， 40（5）： 1295-1300.
	BIAN X Y， JIANG P L， ZHAO M， et al. Multi-branch neural network model based weakly supervised fine-grained image classification method［J］. Journal of Computer Applications， 2020， 40（5）： 1295-1300.
8	RONNEBERGER O， FISCHER P， BROX T. U-net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention， LNCS 9351. Cham： Springer， 2015： 234-241.
9	ZHU Q S， MAI J M， SHAO L. A fast single image haze removal algorithm using color attenuation prior［J］. IEEE Transactions on Image Processing， 2015， 24（11）： 3522-3533. 10.1109/tip.2015.2446191
10	LI B Y， PENG X L， WANG Z Y， et al. AOD-Net： all-in-one dehazing network［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 4780-4788. 10.1109/iccv.2017.511
11	QIN X， WANG Z L， BAI Y C， et al. FFA-Net： feature fusion attention network for single image dehazing［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 11908-11915. 10.1609/aaai.v34i07.6865
12	DONG Y， LIU Y H， ZHANG H， et al. FD-GAN： generative adversarial networks with fusion-discriminator for single image dehazing［C］// Proceedings of the 34th AAAI Conference on Artificial Intelligence. Palo Alto， CA： AAAI Press， 2020： 10729-10736. 10.1609/aaai.v34i07.6701
13	ANCUTI C O， ANCUTI C， VASLUIANU F A， et al. NTIRE 2021 nonhomogeneous dehazing challenge report［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2021： 627-646.
14	杨坤，张娟，方志军. 基于多补丁和多尺度层级聚合网络的快速非均匀图像去雾［J］. 计算机科学， 2021， 48（11）： 250-257. 10.11896/jsjkx.200900058
	YANG K， ZHANG J， FANG Z J. Multi-patch and multi-scale hierarchical aggregation network for fast nonhomogeneous image dehazing［J］. Computer Science， 2021， 48（11）： 250-257. 10.11896/jsjkx.200900058
15	MIRZA M， OSINDERO S. Conditional generative adversarial nets［EB/OL］. （2014-11-06）［2021-11-21］..
16	肖进胜，申梦瑶，雷俊锋，等. 基于生成对抗网络的雾霾场景图像转换算法［J］. 计算机学报， 2020， 43（1）： 165-176. 10.11897/SP.J.1016.2020.00165
	XIAO J S， SHEN M Y， LEI J F， et al. Image conversion algorithm for haze scene based on generative adversarial networks［J］. Chinese Journal of Computers， 2020， 43（1）： 165-176. 10.11897/SP.J.1016.2020.00165
17	LEDIG C， THEIS L， HUSZÁR F， et al. Photo-realistic single image super-resolution using a generative adversarial network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 105-114. 10.1109/cvpr.2017.19
18	ZHANG H， SINDAGI V， PATEL V M. Image de-raining using a conditional generative adversarial network［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（11）： 3943-3956. 10.1109/tcsvt.2019.2920407
19	ZHANG Y L， LI K P， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 294-310.
20	ISOLA P， ZHU J Y， ZHOU T H， et al. Image-to-image translation with conditional adversarial networks［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 5967-5976. 10.1109/cvpr.2017.632
21	JOHNSON J， ALAHI A， LI F F. Perceptual losses for real-time style transfer and super-resolution［C］// Proceedings of the 2016 European Conference on Computer Vision， LNCS 9906. Cham： Springer， 2016： 694-711.
22	WANG Z， SIMONCELLI E P， BOVIK A C. Multiscale structural similarity for image quality assessment［C］// Proceedings of the 37th Asilomar Conference on Signals， Systems and Computers - Volume 2. Piscataway： IEEE， 2003： 1398-1402. 10.1109/acssc.2003.1292181
23	FU M H， LIU H， YU Y K， et al. DW-GAN： a discrete wavelet transform GAN for NonHomogeneous Dehazing［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2021： 203-212. 10.1109/cvprw53098.2021.00029
24	ANCUTI C O， ANCUTI C， VASLUIANU F A， et al. NTIRE 2020 Challenge on NonHomogeneous Dehazing［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition Workshops. Piscataway： IEEE， 2020： 2029-2044.

[1]	Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877.
[2]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[3]	Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918.
[4]	Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703.
[5]	Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969.
[6]	Tong CHEN, Fengyu YANG, Yu XIONG, Hong YAN, Fuxing QIU. Construction method of voiceprint library based on multi-scale frequency-channel attention fusion [J]. Journal of Computer Applications, 2024, 44(8): 2407-2413.
[7]	Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557.
[8]	Yanjie GU, Yingjun ZHANG, Xiaoqian LIU, Wei ZHOU, Wei SUN. Traffic flow forecasting via spatial-temporal multi-graph fusion [J]. Journal of Computer Applications, 2024, 44(8): 2618-2625.
[9]	Qianhong SHI, Yan YANG, Yongquan JIANG, Xiaocao OUYANG, Wubo FAN, Qiang CHEN, Tao JIANG, Yuan LI. Multi-granularity abrupt change fitting network for air quality prediction [J]. Journal of Computer Applications, 2024, 44(8): 2643-2650.
[10]	Zheng WU, Zhiyou CHENG, Zhentian WANG, Chuanjian WANG, Sheng WANG, Hui XU. Deep learning-based classification of head movement amplitude during patient anaesthesia resuscitation [J]. Journal of Computer Applications, 2024, 44(7): 2258-2263.
[11]	Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072.
[12]	Zhi ZHANG, Xin LI, Naifu YE, Kaixi HU. DKP： defending against model stealing attacks based on dark knowledge protection [J]. Journal of Computer Applications, 2024, 44(7): 2080-2086.
[13]	Yiqun ZHAO, Zhiyu ZHANG, Xue DONG. Anisotropic travel time computation method based on dense residual connection physical information neural networks [J]. Journal of Computer Applications, 2024, 44(7): 2310-2318.
[14]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[15]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.