Segmentation network for day and night ground-based cloud images based on improved Res-UNet

doi:10.11772/j.issn.1001-9081.2023040453

Abstract

Abstract:

Aiming at the problems of detail information loss and low segmentation accuracy in the segmentation of day and night ground-based cloud images， a segmentation network called CloudResNet-UNetwork （CloudRes-UNet） for day and night ground-based cloud images based on improved Res-UNet （Residual network-UNetwork） was proposed， in which the overall network structure of encoder-decoder was adopted. Firstly， ResNet50 was used by the encoder to extract features to enhance the feature extraction ability. Then， a Multi-Stage feature extraction （Multi-Stage） module was designed， which combined three techniques of group convolution， dilated convolution and channel shuffle to obtain high-intensity semantic information. Secondly， Efficient Channel Attention Network （ECA?Net） module was added to focus on the important information in the channel dimension， strengthen the attention to the cloud region in the ground-based cloud image， and improve the segmentation accuracy. Finally， bilinear interpolation was used by the decoder to upsample the features， which improved the clarity of the segmented image and reduced the loss of object and position information. The experimental results show that， compared with the state-of-the-art ground-based cloud image segmentation network Cloud-UNetwork （Cloud-UNet） based on deep learning， the segmentation accuracy of CloudRes-UNet on the day and night ground-based cloud image segmentation dataset is increased by 1.5 percentage points， and the Mean Intersection over Union （MIoU） is increased by 1.4 percentage points， which indicates that CloudRes-UNet obtains cloud information more accurately. It has positive significance for weather forecast， climate research， photovoltaic power generation and so on.

Key words: ground-based cloud image, semantic segmentation, deep learning, Efficient Channel Attention Network (ECA-Net), ResNet50, Res-UNet (Residual network-UNetwork)

摘要：

针对昼夜地基云图在分割中细节信息丢失、分割精度低等问题，提出一种基于改进Res-UNet（Residual network-UNetwork）的昼夜地基云图分割网络CloudRes-UNet（Cloud ResNet-UNetwork），整体采用编码器-解码器的网络结构。首先，编码器使用ResNet50提取特征，增强特征提取能力；其次，设计多级特征提取（Multi-Stage）模块，该模块结合分组卷积、膨胀卷积和通道打乱这3种技巧，获取高强度语义信息；再次，加入高效通道注意力（ECA?Net）模块，在通道维度上聚焦重要信息，加强对地基云图中云区域的关注，提高分割精度；最后，解码器使用双线性插值对特征进行上采样，提高分割图像的清晰度并减少目标和位置信息丢失。实验结果表明，与当前基于深度学习表现较好的地基云图分割网络（Cloud-UNet）相比，CloudRes-UNet在昼夜地基云图分割数据集上的分割准确率提升了1.5个百分点，平均交并比（MIoU）上升了1.4个百分点，更准确地获取了云量信息，对天气预报、气候研究和光伏发电等方面具有积极意义。

关键词: 地基云图, 语义分割, 深度学习, 高效通道注意力网络, ResNet50, Res-UNet

CLC Number:

TP391

Boyue WANG, Yingxiang LI, Jiandan ZHONG. Segmentation network for day and night ground-based cloud images based on improved Res-UNet[J]. Journal of Computer Applications, 2024, 44(4): 1310-1316.

王铂越, 李英祥, 钟剑丹. 基于改进Res-UNet的昼夜地基云图分割网络[J]. 《计算机应用》唯一官方网站, 2024, 44(4): 1310-1316.

Figures/Tables 15

References 35

1	STEPHENS G L. Cloud feedbacks in the climate system： a critical review［J］. Journal of Climate， 2005， 18（2）：237-273. 10.1175/jcli-3243.1
2	刘翼飞，崔承刚. 基于地基云图云量特征的光伏发电功率区间预测［J］. 南方电网技术， 2023， 17（2）：92-100.
	LIU Y F， CUI C G. Interval prediction for short-term solar power based on cloud features of ground-based cloud images［J］. Southern Power System Technology， 2023， 17（2）： 92-100.
3	顾轶，韩潮，刘建勋，等.基于卫星云图的大区域云层预测方法［J］.中国空间科学技术， 2023，43（2）：165-173.
	GU Y， HAN C， LIU J X， et al. Research on large area cloud forecasting method based on satellite cloud images［J］. Chinese Space Science and Technology， 2023， 43（2）：165-173.
4	WANG Y， WANG C， SHI C， et al. Short-term cloud coverage prediction using the ARIMA time series model［J］. Remote Sensing Letters， 2018， 9（3）：274-283. 10.1080/2150704x.2017.1418992
5	LONG C N， SABBURG J M， CALBÓ J， et al. Retrieving cloud characteristics from ground-based daytime color all-sky images［J］. Journal Atmospheric Oceanic Technolog， 2006， 23：633-652. 10.1175/jtech1875.1
6	HEINLE A， MACKE A， SRIVASTAV A. Automatic cloud classification of whole sky images［J］. Atmospheric Measurement Techniques， 2010， 3（3）：557-567. 10.5194/amt-3-557-2010
7	LI Q， LU W， YANG J. A hybrid thresholding algorithm for cloud detection on ground-based color images［J］. Journal of Atmospheric & Oceanic Technology， 2011， 28：1286-1296. 10.1175/jtech-d-11-00009.1
8	LIU S， ZHANG L， ZHANG Z， et al. Automatic cloud detection for all-sky images using superpixel segmentation［J］. IEEE Geoscience and Remote Sensing Letters， 2015， 12（2）：354-358. 10.1109/lgrs.2014.2341291
9	SHI C， WANG Y， WANG C， et al. Ground-based cloud detection using graph model built upon superpixels［J］. IEEE Geoscience & Remote Sensing Letters， 2017， 14（5）： 719-723. 10.1109/lgrs.2017.2676007
10	吉茹，张银胜，杨宇龙，等. 基于多尺度特征融合的改进型云图分割方法［J］. 国外电子测量技术， 2022， 41（11）：37-44.
	JI R， ZHANG Y S， YANG Y L，et al. Improved cloud image segmentation method based on multi-scale feature fusion［J］. Foreign Electronic Measurement Technology， 2022， 41（11）：37-44.
11	张雪，贾克斌，刘钧，等. 面向轻量化的地基云图分割技术研究［J］. 测控技术， 2022， 41（9）：37-43.
	ZHANG X， JIA K B， LIU J，et al. Segmentation technology of ground-based cloud image for lightweight［J］. Measurement & Control Technology， 2022， 41（9）：37-43.
12	GACAL G F B， ANTIOQUIA C， LAGROSAS N. Ground-based detection of nighttime clouds above Manila Observatory （14.64°N， 121.07°E） using a digital camera［J］. Applied Optics， 2016， 55（22）：6040-6045. 10.1364/ao.55.006040
13	DEV S， LEE Y H， WINKLER S. Color-based segmentation of sky/cloud images from ground-based cameras［J］. IEEE Journal Selected Topics in Applied Earth Observations and Remote Sensing， 2017， 10（1）：231-242. 10.1109/jstars.2016.2558474
14	SHI C， ZHOU Y， QIU B， et al. Diurnal and nocturnal cloud segmentation of All-Sky Imager （ASI） images using enhancement fully convolutional networks［J］. Atmospheric Measurement Techniques， 2019， 12（9）：4713-4724. 10.5194/amt-12-4713-2019
15	DEV S， NAUTIYAL A， LEE Y H， et al. CloudSegNet： a deep network for nychthemeron cloud image segmentation［J］. IEEE Geoscience and Remote Sensing Letters， 2019， 16（12）：1814-1818. 10.1109/lgrs.2019.2912140
16	张男男，李丽莎，王宝珠，等. 基于MobileNet的地基云图分割方法研究［J］. 电子技术与软件工程， 2022， 18：129-132.
	ZHANG N N， LI L S， WANG B Z，et al. Research on MobileNet-based ground-based cloud segmentation method［J］. Electronic Technology & Software Engineering， 2022， 18：129-132.
17	HE K， ZHANG X， REN S， et al. Deep residual learning for image recognition［C］// Proceeding of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016：770-778. 10.1109/cvpr.2016.90
18	WANG Q， WU B， ZHU P， et al. ECA-Net： efficient channel attention for deep convolutional neural networks［C］// Proceedings of the 2020 IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Washington， DC： IEEE Computer Society， 2020：11531-11539. 10.1109/cvpr42600.2020.01155
19	SHI C， ZHOU Y， QIU B. CloudU-Netv2： a cloud segmentation method for ground-based cloud images based on deep learning［J］. Neural Processing Letters， 2021， 53：2715-2728. 10.1007/s11063-021-10457-2
20	SHELHAMER E， LONG J， DARRELL T. Fully convolutional networks for semantic segmentation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（4）：640-651. 10.1109/tpami.2016.2572683
21	RONNEBERGER O， FISCHER P， BROX T. U-Net： convolutional networks for biomedical image segmentation［C］// Proceedings of the 2015 International Conference on Medical Image Computing and Computer-Assisted Intervention，LNCS 9351. Cham： Springer， 2015： 234-241.
22	BADRINARAYANAN V， KENDALL A， CIPOLLA R. SegNet： a deep convolutional encoder-decoder architecture for image segmentation［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2017， 39（12）：2481-2495. 10.1109/tpami.2016.2644615
23	ZHAO H， SHI J， QI X， et al. Pyramid scene parsing network ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017：6230-6239. 10.1109/cvpr.2017.660
24	CHEN L-C， ZHU Y， PAPANDREOU G， et al. Encoder-decoder with atrous separable convolution for semantic image segmentation［C］// Proceedings of the 2018 European Conference on Computer Vision， LNCS 11211. Cham： Springer， 2018：833-851.
25	LIU Z， LIN Y， CAO Y， et al. Swin Transformer： hierarchical vision Transformer using shifted windows［EB/OL］. ［2023-05-10］. . 10.1109/iccv48922.2021.00986
26	LIU Z， MAO H， WU C-Y， et al. A ConvNet for the 2020s［EB/OL］. ［2023-05-10］.. 10.1109/cvpr52688.2022.01167
27	HE K， CHEN X， XIE S， et al. Masked autoencoders are scalable vision learners［EB/OL］. ［2023-05-10］. . 10.1109/cvpr52688.2022.01553
28	CARON M， TOUVRON H， MISRA I， et al. Emerging properties in self-supervised vision Transformers［EB/OL］. ［2023-05-10］. . 10.1109/iccv48922.2021.00951
29	STEPHAN M， SANTRA A. Radar-based human target detection using deep residual U-Net for smart home applications［C］// Proceedings of the 2019 18th IEEE International Conference on Machine Learning and Applications. Piscataway： IEEE， 2019： 175-182. 10.1109/icmla.2019.00035
30	冯兴杰，张天泽. 基于分组卷积进行特征融合的全景分割算法［J］. 计算机应用， 2021， 41（7）：2054-2061. 10.11772/j.issn.1001-9081.2020091523
	FENG X J， ZHANG T Z. Panoptic segmentation algorithm based on grouped convolution for feature fusion［J］. Journal of Computer Applications， 2021， 41（7）： 2054-2061. 10.11772/j.issn.1001-9081.2020091523
31	谷静，吴怡宁，孟鑫昊. 基于膨胀卷积的多尺度焊缝缺陷检测算法［J］. 光电子·激光， 2022， 33（1）：61-66.
	GU J， WU Y N， MENG X H. Weld defect detection based on expansion convolution multi-scale fusion［J］. Journal of Optoelectronics·Laser， 2022， 33（1）：61-66.
32	YU F， KOLYUN V. Multi-scale context aggregation by dilated convolutions ［EB/OL］. ［2023-03-23］. . 10.1109/cvpr.2017.75
33	ZHANG X， ZHOU X， LIN M， et al. ShuffleNet： an extremely efficient convolutional neural network for mobile devices ［C］// Proceeding of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018：6548-6856. 10.1109/cvpr.2018.00716
34	HU J， SHEN L， ALBANIE S， et al. Squeeze-and-excitation networks［J］. IEEE Transactions on Pattern Analysis and Machine Intelligence， 2020， 42（8）：2011-2023. 10.1109/tpami.2019.2913372
35	SHI C， ZHOU Y， QIU B， et al. CloudU-Net： a deep convolutional neural network architecture for daytime and nighttime cloud images’segmentation［J］. IEEE Geoscience and Remote Sensing Letters， 2021， 18（10）：1688-1692. 10.1109/lgrs.2020.3009227

卷积层名称	输出尺寸	层内部结构
Conv1	160×160	7×7，64，步长=2
Conv2_x	80×80	残差块×3
Conv3_x	40×40	残差块×4
Conv4_x	20×20	残差块×6
Conv5_x	10×10	残差块×3

卷积层名称	输出尺寸	层内部结构
Conv1	160×160	7×7，64，步长=2
Conv2_x	80×80	残差块×3
Conv3_x	40×40	残差块×4
Conv4_x	20×20	残差块×6
Conv5_x	10×10	残差块×3

数据集	分割网络	准确率	精确率	召回率	F1值	ER	MIoU
白天地基云图	U-Net	0.942	0.918	0.917	0.916	0.058	0.847
	PSPNet	0.941	0.941	0.941	0.941	0.059	0.889
	DeepLabv3+	0.934	0.933	0.934	0.933	0.066	0.875
	CloudU-Net	0.951	0.951	0.951	0.951	0.049	0.906
	CloudRes-UNet	0.958	0.958	0.949	0.953	0.042	0.912
夜间地基云图	U-Net	0.952	0.951	0.951	0.951	0.048	0.907
	PSPNet	0.963	0.962	0.963	0.962	0.037	0.927
	DeepLabv3+	0.958	0.958	0.957	0.958	0.042	0.919
	CloudU-Net	0.973	0.972	0.973	0.972	0.027	0.947
	CloudRes-UNet	0.986	0.983	0.981	0.982	0.014	0.965
昼夜地基云图	U-Net	0.931	0.931	0.931	0.931	0.069	0.871
	PSPNet	0.947	0.947	0.947	0.947	0.053	0.900
	DeepLabv3+	0.937	0.936	0.937	0.936	0.063	0.882
	CloudU-Net	0.951	0.954	0.954	0.954	0.049	0.908
	CloudRes-UNet	0.966	0.964	0.958	0.961	0.034	0.922

数据集	分割网络	准确率	精确率	召回率	F1值	ER	MIoU
白天地基云图	U-Net	0.942	0.918	0.917	0.916	0.058	0.847
	PSPNet	0.941	0.941	0.941	0.941	0.059	0.889
	DeepLabv3+	0.934	0.933	0.934	0.933	0.066	0.875
	CloudU-Net	0.951	0.951	0.951	0.951	0.049	0.906
	CloudRes-UNet	0.958	0.958	0.949	0.953	0.042	0.912
夜间地基云图	U-Net	0.952	0.951	0.951	0.951	0.048	0.907
	PSPNet	0.963	0.962	0.963	0.962	0.037	0.927
	DeepLabv3+	0.958	0.958	0.957	0.958	0.042	0.919
	CloudU-Net	0.973	0.972	0.973	0.972	0.027	0.947
	CloudRes-UNet	0.986	0.983	0.981	0.982	0.014	0.965
昼夜地基云图	U-Net	0.931	0.931	0.931	0.931	0.069	0.871
	PSPNet	0.947	0.947	0.947	0.947	0.053	0.900
	DeepLabv3+	0.937	0.936	0.937	0.936	0.063	0.882
	CloudU-Net	0.951	0.954	0.954	0.954	0.049	0.908
	CloudRes-UNet	0.966	0.964	0.958	0.961	0.034	0.922

网络模型	参数量/MB	训练时间/h	测试时间/s
CloudRes-UNet	156.8	2.8	105
U-Net	94.9	3.3	218
PSPNet	9.3	2.1	47
DeepLabv3+	22.4	0.9	63
CloudU-Net	138.6	2.6	91