De-raining algorithm based on joint attention mechanism for single image

doi:10.11772/j.issn.1001-9081.2021061072

Journal of Computer Applications ›› 2022, Vol. 42 ›› Issue (8): 2578-2585.DOI: 10.11772/j.issn.1001-9081.2021061072

• Multimedia computing and computer simulation • Previous Articles Next Articles

De-raining algorithm based on joint attention mechanism for single image

Chengxia XU¹, Qing YAN¹, Teng LI¹^,², Kaichao MIAO³()

^1.School of Electrical Engineering and Automation，Anhui University，Hefei Anhui 230601，China
^2.Key Laboratory of Intelligent Computing & Signal Processing，Ministry of Education （Anhui University），Hefei Anhui 230601，China
^3.Anhui Public Meteorological Service Center，Anhui Meteorology Service，Hefei Anhui 230091，China

Received:2021-06-24 Revised:2022-01-01 Accepted:2022-01-20 Online:2022-03-02 Published:2022-08-10
Contact: Kaichao MIAO
About author:XU Chengxia， born in 1995， M. S. candidate. Her research interests include image denoising based on deep learning.
YAN Qing， born in 1978， Ph. D.， associate professor. Her research interests include sparse subspace clustering， image processing.
LI Teng， born in 1980， Ph. D.， professor. His research interests include speech recognition， image processing.
MIAO Kaichao， born in 1973， M. S.， senior engineer （research fellow）. His research interests include professional meteorological service.

基于联合注意力机制的单幅图像去雨算法

徐成霞¹, 阎庆¹, 李腾¹^,², 苗开超³()

^1.安徽大学电气工程与自动化学院, 合肥 230601
^2.计算智能与信号处理教育部重点实验室(安徽大学), 合肥 230601
^3.安徽省气象局安徽省公共气象服务中心, 合肥 230091

通讯作者: 苗开超
作者简介:徐成霞（1995—），女，安徽合肥人，硕士研究生，主要研究方向：基于深度学习的图像去噪；
阎庆（1978—），女，安徽合肥人，副教授，博士，主要研究方向：稀疏子空间聚类、图像处理；
李腾（1980—），男，安徽合肥人，教授，博士，CCF会员，主要研究方向：语音识别、图像处理；
苗开超（1973—），男，安徽合肥人，正研级高级工程师，硕士，主要研究方向：专业气象服务。

Abstract

Abstract:

It is challenging for the existing single image de-raining algorithms to fully explore the interaction of attention mechanisms in different dimensions. Therefore， an algorithm based on joint attention mechanism was proposed to realize single image de-raining. The algorithm contains a channel attention mechanism and a spatial attention mechanism. Specifically， in the channel attention mechanism， the distribution of rain streak features in each channel was detected and the importance of each feature channel was differentiated. In the spatial attention mechanism， aiming at the spatial relationship of rain streak distribution within channels， the context information was accumulated in a local to global manner to realize efficient and accurate de-raining. Additionally， a deep residual shrinkage network with a soft threshold nonlinear transformation sub-network embedded in the residual module was used to zero out redundant information via a soft threshold function， thereby improving the ability of the CNN in retaining image details in noise. Experiments were carried out on open rainfall data sets and self constructed rainfall data sets. Compared with spatial attention， the joint attention rain removal algorithm improved Peak Signal-to-Noise Ratio （PSNR） by 4.5% and the Structural SIMilarity （SSIM） by 0.3%. Experimental results show that the proposed algorithm can effectively perform single image de-raining and image detail preserving. At the same time， this algorithm outperforms the comparison algorithms in terms of visual effect and quantitative metrics.

Key words: image de-raining, Convolutional Neural Network (CNN), attention mechanism, deep residual shrinkage network, soft threshold

摘要：

现有的单幅图像去雨算法难以充分发掘不同维度注意力机制的相互作用，因此提出一种基于联合注意力机制的单幅图像去雨算法。该算法包含通道注意力机制和空间注意力机制：通道注意力机制检测各通道雨线特征的分布，并差异化各个特征通道的重要程度；空间注意力机制则针对通道内雨线分布的空间关系，以局部到全局的方式积累上下文信息，从而高效准确地去雨。此外，引入深度残差收缩网络，以利用残差模块中嵌入的软阈值非线性变换子网络来通过软阈值函数将冗余信息置零，从而提升CNN在噪声中保留图像细节的能力。在公开降雨数据集与自构建的降雨数据集上进行实验，相较于单一空间注意力算法，联合注意力去雨算法的峰值信噪比（PSNR）提升4.5%，结构相似性（SSIM）提升0.3%。实验结果表明，所提算法可以有效地进行单幅图像去雨和图像细节的信息保留，在目视效果和定量指标上均优于对比算法。

关键词: 图像去雨, 卷积神经网络, 注意力机制, 深度残差收缩网络, 软阈值

CLC Number:

TP751.1

Chengxia XU, Qing YAN, Teng LI, Kaichao MIAO. De-raining algorithm based on joint attention mechanism for single image[J]. Journal of Computer Applications, 2022, 42(8): 2578-2585.

徐成霞, 阎庆, 李腾, 苗开超. 基于联合注意力机制的单幅图像去雨算法[J]. 《计算机应用》唯一官方网站, 2022, 42(8): 2578-2585.

Figures/Tables 13

Fig. 1 Examples of rain image in MSR

Fig. 2 Residual module structure

Fig. 3 Attention mechanism modules

Fig. 4 Joint attention de-raining network architecture

Fig. 5 Rain streak detection results for spatial attention and joint attention

Tab. 1 Comparison of results of different de-raining algorithms

算法	SSIM/VIFP/PSNR
算法	Rain100H	DID-MDN	SPANet-Data
原始雨图	0.357/0.172/12.13	0.706/0.363/22.16	0.904/0.753/32.63
PReNet^［19］	0.858/0.679/26.77	0.945/0.861/32.44	0.937/0.810/35.60
SPANet^［6］	0.847/0.702/27.34	0.947/0.847/32.06	0.975/0.826/36.83
Syn2Real^［20］	0.860/0.715/28.77	0.947/0.861/34.33	0.962/0.817/36.79
本文算法	0.856/0.715/27.20	0.951/0.859/32.15	0.975/0.838/36.97

Fig. 6 Visualization results of different de-raining algorithms

Tab. 2 Cross-validation results on synthetic rain data

测试数据集	SSIM/VIFP/PSNR
测试数据集	原始雨图	Rain100H&100L	DID-MDN	MSR-Synthetic
Rain100H	0.357/0.172/13.13	0.917/0.863/29.15	0.731/0.761/24.71	0.786/0.764/26.85
Rain100L	0.713/0.402/23.64	0.921/0.865/29.34	0.911/0.806/27.61	0.913/0.807/29.37
DID-MDN	0.706/0.363/22.16	0.844/0.841/26.33	0.947/0.860/33.87	0.958/0.860/34.25
MSR-Synthetic	0.762/0.397/23.07	0.908/0.843/26.97	0.944/0.861/33.63	0.960/0.861/36.65

Fig. 7 De-raining results on synthetic rain data

Tab. 3 Cross-validation results on real rain data

测试数据集	SSIM/VIFP/PSNR
测试数据集	原图	SPANet-Data	MSR-Real
SPANet-Data	0.904/0.753/32.63	0.986/0.878/38.06	0.985/0.875/37.87
MSR-Real	0.910/0.802/32.83	0.985/0.880/37.81	0.988/0.881/38.90

Fig. 8 De-raining results on real rain data

Fig. 9 Denoising experiment of residual shrinkage module

Tab. 4 Network module analysis

模块	PSNR/dB	SSIM	运行时间/s
模块	PSNR/dB	SSIM	512×512	1 024×1 024
$B a$	37.23	0.985	0.103	0.341
$B b$	37.43	0.985	0.126	0.423
$B c$	37.31	0.984	0.117	0.351
$B d$	38.90	0.988	0.132	0.441

Tab. 4 Network module analysis

模块	PSNR/dB	SSIM	运行时间/s
模块	PSNR/dB	SSIM	512×512	1 024×1 024
$B a$	37.23	0.985	0.103	0.341
$B b$	37.43	0.985	0.126	0.423
$B c$	37.31	0.984	0.117	0.351
$B d$	38.90	0.988	0.132	0.441

References 20

1	何新宇，张晓龙.基于深度神经网络的肺炎图像识别模型［J］.计算机应用， 2019， 39（6）： 1680-1684. 10.11772/j.issn.1001-9081.2018102112
	HE X Y， ZHANG X L. Pneumonia image recognition model based on deep neural network［J］. Journal of Computer Applications， 2019， 39（6）： 1680-1684. 10.11772/j.issn.1001-9081.2018102112
2	ZAMIR S W， ARORA A， KHAN S， et al. Multi-stage progressive image restoration ［C］// Proceedings of the 2021 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2021： 14816-14826. 10.1109/cvpr46437.2021.01458
3	FU X Y， HUANG J B， ZENG D L， et al. Removing rain from single images via a deep detail network ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1715-1723. 10.1109/cvpr.2017.186
4	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
5	ZHANG H， PATEL V M. Density-aware single image de-raining using a multi-stream dense network ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 695-704. 10.1109/cvpr.2018.00079
6	WANG T Y， YANG X， XU K， et al. Spatial attentive single-image deraining with a high quality real rain dataset ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 12262-12271. 10.1109/cvpr.2019.01255
7	YANG W H， TAN R T， FENG J S， et al. Deep joint rain detection and removal from a single image ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1685-1694. 10.1109/cvpr.2017.183
8	ZHANG H， SINDAGI H， PATEL V M. Image de-raining using a conditional generative adversarial network［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（11）： 3943-3956. 10.1109/tcsvt.2019.2920407
9	ZHAO M H， ZHONG S S， FU X Y， et al. Deep residual shrinkage networks for fault diagnosis［J］. IEEE Transactions on Industrial Informatics， 2020， 16（7）： 4681-4690. 10.1109/tii.2019.2943898
10	YU F， KOLTUN V， FUNKHOUSER T. Dilated residual networks ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 636-644. 10.1109/cvpr.2017.75
11	WANG X L， GIRSHICK R， GUPTA A， et al. Non-local neural networks ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 7794-7803. 10.1109/cvpr.2018.00813
12	JADERBERG M， SIMONYAN K， ZISSERMAN A， et al. Spatial transformer networks ［C］// Proceedings of the 28th International Conference on Neural Information Processing Systems. Cambridge： MIT Press， 2015： 2017-2025. 10.1007/s11263-015-0823-z
13	WOO S， PARK J， LEE J Y， et al. CBAM： convolutional block attention module ［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 3-19.
14	LE Q V， JAITLY N， HINTON G E. A simple way to initialize recurrent networks of rectified linear units［EB/OL］. （2015-04-07）［2021-04-10］. .
15	ZAREMBA W， SUTSKEVER I， VINYALS O. Recurrent neural network regularization［EB/OL］. （2015-02-19）［2021-04-10］. .
16	WANG Z， BOVIK A C， SHEIKH H R， et al. Image quality assessment： from error visibility to structural similarity［J］. IEEE Transactions on Image Processing， 2004， 13（4）： 600-612. 10.1109/tip.2003.819861
17	ZHANG L， ZHANG L， MOU X Q， et al. FSIM： a feature similarity index for image quality assessment［J］. IEEE Transactions on Image Processing， 2011， 20（8）： 2378-2386. 10.1109/tip.2011.2109730
18	SHEIKH H R， BOVIK A C， DE VECIANA G. An information fidelity criterion for image quality assessment using natural scene statistics［J］. IEEE Transactions on Image Processing， 2005， 14（12）： 2117-2128. 10.1109/tip.2005.859389
19	REN D W， ZOU W M， HU Q H， et al. Progressive image deraining networks： a better and simpler baseline ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3932-3941. 10.1109/cvpr.2019.00406
20	YASARLA R， SINDAGI V A， PATEL V M. Syn2Real transfer learning for image deraining using Gaussian processes ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2723-2733. 10.1109/cvpr42600.2020.00280

[1]	Yun LI, Fuyou WANG, Peiguang JING, Su WANG, Ao XIAO. Uncertainty-based frame associated short video event detection method [J]. Journal of Computer Applications, 2024, 44(9): 2903-2910.
[2]	Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892.
[3]	Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974.
[4]	Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738.
[5]	Hong CHEN, Bing QI, Haibo JIN, Cong WU, Li’ang ZHANG. Class-imbalanced traffic abnormal detection based on 1D-CNN and BiGRU [J]. Journal of Computer Applications, 2024, 44(8): 2493-2499.
[6]	Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392.
[7]	Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406.
[8]	Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594.
[9]	Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617.
[10]	Yangyi GAO, Tao LEI, Xiaogang DU, Suiyong LI, Yingbo WANG, Chongdan MIN. Crowd counting and locating method based on pixel distance map and four-dimensional dynamic convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2233-2242.
[11]	Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109.
[12]	Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199.
[13]	Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182.
[14]	Wenliang WEI, Yangping WANG, Biao YUE, Anzheng WANG, Zhe ZHANG. Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention [J]. Journal of Computer Applications, 2024, 44(7): 2183-2191.
[15]	Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232.

De-raining algorithm based on joint attention mechanism for single image

基于联合注意力机制的单幅图像去雨算法

RichHTML

PDF

Knowledge

Abstract

Cite this article

share this article

Figures/Tables 13

References 20

Related Articles 15

Recommended Articles

Metrics