Residual attention deraining network based on convolutional long short-term memory

doi:10.11772/j.issn.1001-9081.2021081379

Abstract

Abstract:

Unmanned driving vehicles driving in rainy environment face the following problems： the images collected by the car on-board camera contain rain streak noise， which reduces the target detection accuracy and difficulty in identifying key targets of the unmanned driving system. In order to solve these problems， a residual attention deraining network based on convolutional long short-term memory was proposed. Firstly， the Convolutional Long Short-Term Memory （CLSTM） units were proposed to learn the distribution of different scales of rain streaks. Then， the residual channel attention mechanism was used to extract the rain streaks. Finally， the extracted rain streak information was subtracted from the rain image to obtain the restored background image. To determine the optimal network structure， the ablation experiments of each network module were carried out， and the structure with best rain removal effect was selected as the deraining network. Through the continuous optimization of network parameters， the proposed algorithm was tested on Rain100H， Rain100L and Real100 datasets， the results illustrate that the Peak Signal-to-Noise Ratio （PSNR） of the proposed algorithm reaches 29.1 dB， 33.1 dB and 32.4 dB respectively， and the Structural SIMilarity （SSIM） of the algorithm reaches 0.89， 0.94 and 0.93 respectively. Experimental results show that through the additional supervision of the Generative Adversarial Network （GAN） discriminator， the proposed algorithm achieves an visible rain streak removal effect and enhances the environmental perception ability of unmanned driving system under complex rainfall condition.

Key words: deraining, Generative Adversarial Network (GAN), Convolutional Long Short-Term Memory (CLSTM) network, residual channel attention, multi-scale feature fusion

摘要：

无人驾驶汽车在雨天环境中行驶，由于车载相机采集的图片包含雨纹噪声，导致无人驾驶系统的目标检测精度降低，关键目标识别困难。为解决这些问题，提出了一种基于卷积长短期记忆的残差注意力去雨网络。首先提出卷积长短期记忆（CLSTM）单元对不同尺度的雨纹分布进行学习，然后使用残差通道注意力机制对雨纹进行提取，最后将雨图与雨纹提取信息相减得到修复后的背景图。为确定最优的网络结构，对各网络模块进行消融实验，然后选择去雨效果最优的结构作为去雨网络。通过对网络参数的不断优化，所提算法在数据集Rain100H、Rain100L、Real200上进行测试，结果显示该算法的峰值信噪比（PSNR）分别达到29.1 dB、33.1 dB、32.4 dB，结构相似性（SSIM）分别达到0.89、0.94和0.93。实验结果表明，通过生成对抗网络（GAN）判别器对雨纹去除效果的额外监督，所提算法取得了明显的雨纹去除效果，增强了无人驾驶系统在复杂降雨条件下的环境感知能力。

关键词: 去雨, 生成对抗网络, 卷积长短期记忆网络, 残差通道注意力, 多尺度特征融合

CLC Number:

TP391.4

Zanxia QIANG, Xianfu BAO. Residual attention deraining network based on convolutional long short-term memory[J]. Journal of Computer Applications, 2022, 42(9): 2858-2864.

强赞霞, 鲍先富. 基于卷积长短期记忆的残差注意力去雨网络[J]. 《计算机应用》唯一官方网站, 2022, 42(9): 2858-2864.

Figures/Tables 10

References 22

1	CRESWELL A， WHITE T， DUMOULIN V， et al. Generative adversarial networks： an overview［J］. IEEE Signal Processing Magazine， 2018， 35（1）： 53-65. 10.1109/msp.2017.2765202
2	REYNOLDS D A， QUATIERI T F， DUNN R B. Speaker verification using adapted Gaussian mixture models［J］. Digital Signal Processing， 2000， 10（1/2/3）：19-41. 10.1006/dspr.1999.0361
3	KANG L W， LIN C W， FU Y H. Automatic single-image-based rain streaks removal via image decomposition［J］. IEEE Transactions on Image Processing， 2012， 21（4）： 1742-1755. 10.1109/tip.2011.2179057
4	LUO Y， XU Y， JI H. Removing rain from a single image via discriminative sparse coding［C］// Proceedings of the 2015 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2015： 3397-3405. 10.1109/iccv.2015.388
5	LI Y， TAN R T， GUO X J， et al. Rain streak removal using layer priors［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2736-2744. 10.1109/cvpr.2016.299
6	QIAN R， TAN R T， YANG W H， et al. Attentive generative adversarial network for raindrop removal from a single image［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 2482-2491. 10.1109/cvpr.2018.00263
7	CHEN B H， HUANG S C， KUO S Y. Error-optimized sparse representation for single image rain removal［J］. IEEE Transactions on Industrial Electronics， 2017， 64（8）： 6573-6581. 10.1109/tie.2017.2682036
8	FU X Y， HUANG J B， DING X H， et al. Clearing the skies： a deep network architecture for single-image rain removal［J］. IEEE Transactions on Image Processing， 2017， 26（6）： 2944-2956. 10.1109/tip.2017.2691802
9	BARZEGAR S， SHARIFI A， MANTHOURI M. Super-resolution using lightweight detailnet network［J］. Multimedia Tools and Applications， 2020， 79（1/2）： 1119-1136. 10.1007/s11042-019-08218-4
10	ZHANG H， SINDAGI V， PATEL V M. Image de-raining using a conditional generative adversarial network［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（11）： 3943-3956. 10.1109/tcsvt.2019.2920407
11	JIANG K， WANG Z Y， YI P， et al. Multi-scale progressive fusion network for single image deraining［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 8343-8352. 10.1109/cvpr42600.2020.00837
12	邱永茹，姚光乐，冯杰，等. 基于半监督学习的单幅图像去雨算法［J］. 计算机应用， 2022， 42（5）： 1577-1582. 10.11772/j.issn.1001-9081.2021030492
	QIU Y R， YAO G L， FENG J， et al. Single image de-raining algorithm based on semi-supervised learning［J］. Journal of Computer Applications， 2022， 42（5）： 1577-1582. 10.11772/j.issn.1001-9081.2021030492
13	LAI W S， HUANG J B， AHUJA N， et al. Deep Laplacian pyramid networks for fast and accurate super-resolution［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern recognition. Piscataway： IEEE， 2017： 5835-5843. 10.1109/cvpr.2017.618
14	ZHANG H， PATEL V M. Density-aware single image de-raining using a multi-stream dense network［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 695-704. 10.1109/cvpr.2018.00079
15	SHI X J， CHEN Z R， WANG H， et al. Convolutional LSTM network： a machine learning approach for precipitation nowcasting［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2015： 802-810.
16	GERS F A， SCHMIDHUBER J， CUMMINS F. Learning to forget： continual prediction with LSTM［J］. Neural Computation， 2000， 12（10）： 2451-2471. 10.1162/089976600300015015
17	ZHANG Y L， LI K P， LI K， et al. Image super-resolution using very deep residual channel attention networks［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 294-310.
18	FU X Y， HUANG J B， ZENG D L， et al. Removing rain from single images via a deep detail network［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1715-1723. 10.1109/cvpr.2017.186
19	YANG W H， TAN R T， FENG J S， et al. Deep joint rain detection and removal from a single image［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1685-1694. 10.1109/cvpr.2017.183
20	LI X， WU J L， LIN Z C， et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 262-277.
21	REN D E， ZUO W M， HU Q H， et al. Progressive image deraining networks： a better and simpler baseline［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3932-3941. 10.1109/cvpr.2019.00406
22	BOCHKOVSKIY A， WANG C Y， LIAO H Y M. YOLOv4： optimal speed and accuracy of object detection［EB/OL］. （2020-04-23）［2021-07-25］..

特征尺寸	CLSTM数	RCAB数	RG数
1	1（核大小：3×3）	4	10
1/2	1（核大小：3×3）	4	10
1/4	1（核大小：3×3）	4	10

特征尺寸	CLSTM数	RCAB数	RG数
1	1（核大小：3×3）	4	10
1/2	1（核大小：3×3）	4	10
1/4	1（核大小：3×3）	4	10

数据集名	测试集名	训练集样本数	测试集样本数
合计	—	15 400	2 800
Rain14000^［18］	Test2000	12 000	2 000
Rain1800^［19］	Rain1800	1 800	0
Rain800^［10］	Test100	700	100
Rain100H^［19］	Rain100H	0	100
Rain100L^［19］	Rain100L	0	100
Rain1200^［14］	Test300	900	300
Real200^［17］	Real200	0	200

数据集名	测试集名	训练集样本数	测试集样本数
合计	—	15 400	2 800
Rain14000^［18］	Test2000	12 000	2 000
Rain1800^［19］	Rain1800	1 800	0
Rain800^［10］	Test100	700	100
Rain100H^［19］	Rain100H	0	100
Rain100L^［19］	Rain100L	0	100
Rain1200^［14］	Test300	900	300
Real200^［17］	Real200	0	200

模型	结构	PSNR/dB	SSIM	推理时间/s
Model1	t=2，B=4，C=10	25.19	0.782	0.245
Model2	t=0，B=4，C=10	23.61	0.727	0.156
Model3	t=1，B=4，C=1	25.15	0.791	0.164
Model4	t=1，B=4，C=5	27.31	0.831	0.184
Model5	t=1，B=4，C=10	29.06	0.896	0.194
Model6	t=1，B=4，C=15	27.98	0.856	0.213
Model7	t=1，B=3，C=10	25.61	0.812	0.182
Model8	t=1，B=5，C=10	24.40	0.829	0.206