基于半监督学习的单幅图像去雨算法

doi:10.11772/j.issn.1001-9081.2021030492

《计算机应用》唯一官方网站 ›› 2022, Vol. 42 ›› Issue (5): 1577-1582.DOI: 10.11772/j.issn.1001-9081.2021030492

• 多媒体计算与计算机仿真 • 上一篇下一篇

基于半监督学习的单幅图像去雨算法

邱永茹, 姚光乐(), 冯杰, 崔昊宇

成都理工大学计算机与网络安全学院（牛津布鲁克斯学院），成都 610059

收稿日期:2021-04-02 修回日期:2021-06-08 接受日期:2021-06-08 发布日期:2022-06-11 出版日期:2022-05-10
通讯作者: 姚光乐
作者简介:邱永茹（1998—），女，广东湛江人，主要研究方向：图像复原、深度学习网络结构
姚光乐（1985—），男，河南三门峡人，副教授，博士，主要研究方向：人工智能，计算机视觉 yaoguangle19@cdut.edu.cn
冯杰（1995—），男，四川南充人，硕士研究生，主要研究方向：图像去雨、遥感图像处理、半监督学习
崔昊宇（2000—），男，吉林伊通人，主要研究方向：计算机视觉、人工智能。
基金资助:
四川省重点研发计划项目(2020YFG0169)

Single image de-raining algorithm based on semi-supervised learning

Yongru QIU, Guangle YAO(), Jie FENG, Haoyu CUI

College of Computer Science and Cyber Security （Oxford Brookes College），Chengdu University of Technology，Chengdu Sichuan 610059，China

Received:2021-04-02 Revised:2021-06-08 Accepted:2021-06-08 Online:2022-06-11 Published:2022-05-10
Contact: Guangle YAO
About author:QIU Yongru， born in 1998. Her research interests include imagerestoration，deep learning network structure.
YAO Guangle， born in 1985，Ph. D.，associate professor. Hisresearch interests include artificial intelligence，computer vision.
FENG Jie， born in 1995，M. S. candidate. His research interestsinclude image de-raining， remote sensing image processing， semisupervised learning.
CUI Haoyu， born in 2000. His research interests include computervision，artificial intelligence.
Supported by:
Key Science and Technology Program of Sichuan Province(2020YFG0169)

摘要/Abstract

摘要：

在雨天采集的图像通常存在背景物体被雨纹遮挡、图像变形等影响图像质量的现象，对后续图像分析及应用造成严重影响。近年来，已经提出了许多基于深度学习的去雨算法并获得了较好的效果。由于真实雨图的无雨纹干净背景图采集非常困难，大多数算法都采用监督学习即在含有配对标签的合成雨图数据集上进行模型训练。由于合成雨图和真实雨图中雨纹的亮度、透明度、形状等存在巨大差异，基于监督学习的去雨算法对真实雨图的泛化能力普遍较差。为提高去雨模型对真实雨图的去雨效果，提出了一种基于半监督学习的单幅图像去雨算法。该算法在模型训练过程中加入合成雨图和真实雨图并最小化两个输入图像转换成的特征向量的一阶信息和二阶统计信息差异，使两者特征分布一致。同时，针对雨纹复杂多样的特点，引入多尺度网络以获取更丰富的图像特征，并提高模型性能。实验结果表明，所提算法在Rain100H合成雨图测试集上相较JDNet、Syn2Real等算法在峰值信噪比（PSNR）和结构相似度（SSIM）上分别至少提升了0.66 dB、0.01，在去除雨纹的同时能最大限度保留图像细节和颜色信息；并且由于减少了分布差异，该算法在真实雨图测试集上的去雨效果明显优于现有的JDNet、Syn2Real等去雨算法，具有较强的泛化能力。所提算法可以应用于现有的基于监督学习的去雨算法并显著提高其去雨效果，拥有较高的独立性。

关键词: 单幅图像去雨, 半监督学习, 多尺度网络, 深度学习, 密集残差连接

Abstract:

The images collected in rainy days usually have some phenomena that affect the image quality， such as the background object blocked by rain streaks and the image deformation， which have serious impact on the subsequent image analysis and application. Recently， numerous de-raining algorithms based on deep learning have been proposed and achieve good results. Most algorithms adopt supervised learning， that is， training the model on the synthetic rainy image dataset with paired labels due to the difficulty in acquiring clean background images without rain streaks from real-world rainy images. However， there are differences between synthetic and real-world rainy images on brightness， transparency， and shape of rain streaks. Thus， most de-raining algorithms based on supervised learning have poor generalization ability to real-world rainy images. Therefore， in order to improve the rain removal effect of de-raining models on the real-world rainy images， a single image de-raining algorithm based on semi-supervised learning was proposed. In the model training process of the proposed algorithm， the synthetic and real-world rainy images were added， and the difference of the first-order and second-order statistic information of feature vectors converted from the both input images were minimized to make the features of the both have same distribution. Meanwhile， in view of the complex and diverse characteristics of rain streaks， a multi-scale network was introduced to obtain richer image features and improve the performance of model. Experimental results show that， on the Rain100H dataset of synthetic rainy images， compared with Joint Deraining Network （JDNet）， Synthetic-to-Real transfer learning （Syn2Real）， the proposed algorithm improves the Peak Signal-to-Noise Ratio （PSNR） and Structural SIMilarity （SSIM） by at least 0.66 dB and 0.01 respectively. While removing rain streaks， the proposed algorithm can retain image details and color information to the greatest extent. At the same time， with the reduction of distribution discrepancy， the proposed algorithm achieves better performance on the real-world rainy images with strong generalization ability， compared with the de-raining algorithms such as JDNet and Syn2Real. The proposed algorithm is highly independent， can be applied to the existing de-raining algorithms based on supervised learning and significantly improve their de-raining effects.

Key words: single image de-raining, semi-supervised learning, multi-scale network, deep learning, dense residual connection

中图分类号:

TP391.41

邱永茹, 姚光乐, 冯杰, 崔昊宇. 基于半监督学习的单幅图像去雨算法[J]. 计算机应用, 2022, 42(5): 1577-1582.

Yongru QIU, Guangle YAO, Jie FENG, Haoyu CUI. Single image de-raining algorithm based on semi-supervised learning[J]. Journal of Computer Applications, 2022, 42(5): 1577-1582.

图/表 8

图1 基于半监督学习的多尺度网络结构

Fig. 1 Multi-scale network structure based on semi-supervised learning

图2 合成雨图和真实雨图的差异

Fig. 2 Difference between synthetic and real-world rainy images

图3 半监督匹配网络的详细结构

Fig. 3 Detailed structure of semi-supervised matching network

图4 半监督匹配网络中的特征分布对齐

Fig. 4 Feature distribution alignment in semi-supervised matching network

图5 多尺度去雨网络的详细结构

Fig. 5 Detailed structure of multi-scale de-raining network

表1 不同去雨算法在测试数据集上的图像质量评估指标（PSNR/SSIM）对比

Tab. 1 Comparison of image quality evaluation indicators （PSNR/SSIM） among different de-raining algorithms on test datasets

算法	Rain100H		Rain100L		Rain12
算法	PSNR/dB	SSIM	PSNR/dB	SSIM	PSNR/dB	SSIM
RESCAN	25.92	0.84	36.12	0.97	32.35	0.89
PreNet	27.89	0.89	36.69	0.98	34.77	0.96
SSIR	22.47	0.71	32.37	0.92	24.12	0.78
JDNet	30.02	0.92	38.65	0.99	37.02	0.97
本文算法	30.68	0.93	39.53	0.99	36.61	0.97

图6 不同算法在真实雨图上的视觉比较

Fig. 6 Visual comparison of different algorithms on real-world rainy images

图7 RESCAN和JDNet应用半监督匹配网络的去雨实验结果

Fig. 7 Experimental results of de-raining by RESCAN and JDNet using semi-supervised matching network

参考文献 20

1	KANG L W， LIN C W， FU Y H. Automatic single-image-based rain streaks removal via image decomposition ［J］. IEEE Transactions on Image Processing， 2012， 21（4）： 1742-1755. 10.1109/tip.2011.2179057
2	LI Y， TAN R T， GUO X J， et al. Rain streak removal using layer priors ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 2736-2744. 10.1109/cvpr.2016.299
3	黄靖，姜文，肖长诗，等.一种深度学习的单幅图像去雾方法［J］.小型微型计算机系统，2018，39（8）：1882-1888. 10.3969/j.issn.1000-1220.2018.08.044
	HUANG J， JIANG W， XIAO C S， et al. Single image haze removal algorithm based on deep learning ［J］. Journal of Chinese Computer Systems， 2018， 39（8）： 1882-1888. 10.3969/j.issn.1000-1220.2018.08.044
4	徐爱生，唐丽娟，陈冠楠.注意力残差网络的单图像去雨方法研究［J］.小型微型计算机系统，2020，41（6）：1281-1285.
	XU A S， TANG L J， CHEN G N. Single image rain removal method based on attention mechanism ［J］. Journal of Chinese Computer Systems， 2020， 41（6）： 1281-1285.
5	FU X Y， HUANG J B， DING X H， et al. Clearing the skies： a deep network architecture for single-image rain removal ［J］. IEEE Transactions on Image Processing， 2017， 26（6）： 2944-2956. 10.1109/tip.2017.2691802
6	FU X Y， HUANG J B， ZENG D L， et al. Removing rain from single images via a deep detail network ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1715-1723. 10.1109/cvpr.2017.186
7	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
8	LI X， WU J L， LIN Z C， et al. Recurrent squeeze-and-excitation context aggregation net for single image deraining ［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018： 262-277.
9	ZHANG H， PATEL V M. Convolutional sparse and low-rank coding-based rain streak removal ［C］// Proceedings of the 2017 IEEE Winter Conference on Applications of Computer Vision. Piscataway： IEEE， 2017： 1259-1267. 10.1109/wacv.2017.145
10	ZHANG H， PATEL V M. Density-aware single image de-raining using a multi-stream dense network ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 695-704. 10.1109/cvpr.2018.00079
11	ZHANG H， SINDAGI V A， PATEL V M. Image de-raining using a conditional generative adversarial network ［J］. IEEE Transactions on Circuits and Systems for Video Technology， 2020， 30（11）： 3943-3956. 10.1109/tcsvt.2019.2920407
12	李烨，许乾坤，李克东.面向图像复原的残差密集生成对抗网络新方法［J］.小型微型计算机系统，2020，41（4）：830-836. 10.3969/j.issn.1000-1220.2020.04.028
	LI Y， XU Q K， LI K D. New method of residual dense generative adversarial networks for image restoration ［J］. Journal of Chinese Computer Systems， 2020， 41（4）： 830-836. 10.3969/j.issn.1000-1220.2020.04.028
13	WEI W， MENG D Y， ZHAO Q， et al. Semi-supervised transfer learning for image rain removal ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3872-3881. 10.1109/cvpr.2019.00400
14	YASARLA R， SINDAGI V A， PATEL V M. Syn2Real transfer learning for image deraining using Gaussian processes ［C］// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2020： 2723-2733. 10.1109/cvpr42600.2020.00280
15	HUANG G， LIU Z， L van der MAATEN， et al. Densely connected convolutional networks ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2261-2269. 10.1109/cvpr.2017.243
16	YANG W H， TAN R T， FENG J S， et al. Deep joint rain detection and removal from a single image ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 1685-1694. 10.1109/cvpr.2017.183
17	WANG T Y， YANG X， XU K， et al. Spatial attentive single-image deraining with a high quality real rain dataset ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 12262-12271. 10.1109/cvpr.2019.01255
18	WANG Z， BOVIK A C， SHEIKH H R， et al. Image quality assessment： from error visibility to structural similarity ［J］. IEEE Transactions on Image Processing， 2004， 13（4）： 600-612. 10.1016/j.jvcir.2019.102655
19	REN D W， ZUO W M， HU Q H， et al. Progressive image deraining networks： a better and simpler baseline ［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 3937-3946. 10.1109/cvpr.2019.00406
20	WANG C， WU Y T， SU Z X， et al. Joint self-attention and scale-aggregation for self-calibrated deraining network ［C］// Proceedings of the 2020 28th ACM International Conference on Multimedia. New York： ACM， 2020： 2517-2525. 10.1145/3394171.3413559

[1]	屈震, 李堃婷, 冯志玺. 基于有效通道注意力的遥感图像场景分类[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1431-1439.
[2]	谢新林, 肖毅, 续欣莹. 基于神经网络架构搜索的肺结节分类算法[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1424-1430.
[3]	任炜, 白鹤翔. 基于全局与局部标签关系的多标签图像分类方法[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1383-1390.
[4]	鲁永帅, 唐英杰, 马鑫然. 基于深度特征融合的无纺布低对比度浆丝缺陷检测方法[J]. 《计算机应用》唯一官方网站, 2022, 42(5): 1440-1446.
[5]	陈浩杰, 范江亭, 刘勇. 深度强化学习解决动态旅行商问题[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1194-1200.
[6]	汪祖民, 张志豪, 秦静, 季长清. 基于卷积神经网络的机械故障诊断技术综述[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1036-1043.
[7]	董永峰, 邓亚晗, 董瑶, 王雅琮. 基于深度学习的聚类综述[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1021-1028.
[8]	王颖洁, 朱久祺, 汪祖民, 白凤波, 弓箭. 自然语言处理在文本情感分析领域应用综述[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1011-1020.
[9]	季长清, 高志勇, 秦静, 汪祖民. 基于卷积神经网络的图像分类算法综述[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1044-1049.
[10]	张锦, 屈佩琪, 孙程, 罗蒙. 基于改进YOLOv5的安全帽佩戴检测算法[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1292-1300.
[11]	刘志华, 陈文洁, 陈爱斌. 基于自注意力机制时频谱同源特征融合的鸟鸣声分类[J]. 《计算机应用》唯一官方网站, 2022, 42(4): 1260-1268.
[12]	殷雨昌, 王洪元, 陈莉, 冯尊登, 肖宇. 基于单标注样本的多损失学习与联合度量视频行人重识别[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 764-769.
[13]	于婉莹, 梁美玉, 王笑笑, 陈徵, 曹晓雯. 基于深度注意力网络的课堂教学视频中学生表情识别与智能教学评估[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 743-749.
[14]	李讷, 徐光柱, 雷帮军, 马国亮, 石勇涛. 交通道路行驶车辆车标识别算法[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 810-817.
[15]	陈亭秀, 尹建芹. 基于关键帧筛选网络的视听联合动作识别[J]. 《计算机应用》唯一官方网站, 2022, 42(3): 731-735.

基于半监督学习的单幅图像去雨算法

Single image de-raining algorithm based on semi-supervised learning

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 20

相关文章 15

编辑推荐

Metrics