Improved remote sensing image fusion algorithm based on channel attention feedback network

doi:10.11772/j.issn.1001-9081.2020071064

Abstract

Abstract: Aiming at the problems of feedforward Convolutional Neural Network(CNN), such as small receptive field, insufficient context information acquirement and that only shallow features can be extracted by the feature extraction convolutional layer of the network, an improved remote sensing image fusion algorithm based on channel attention feedback network was proposed. Firstly, the detail features of PANchromatic(PAN) images and the spectral features of Low-resolution MultiSpectral(LMS) images were initially extracted through two convolutional layers. Secondly, the extracted features were combined with the deep features fed back from the network and inputted to the channel attention mechanism module to obtain the initially refined features. Thirdly, the deep features with stronger characterization capability were generated by feedback module. Finally, High-resolution MultiSpectral(HMS) images were obtained by putting the generated deep features into the reconstruction layer with deconvolution. Experimental results on three different satellite image datasets show that the proposed algorithm can well extract the detail features of PAN images and the spectral features of LMS images, and the HMS images recovered by this algorithm are clearer subjectively and better than the comparison algorithms objectively; at the same time, the Root Mean Square Error(RMSE) index of the proposed method is more than 50% lower than that the traditional methods, and more than 10% lower than that the feedforward convolutional network methods.

Key words: remote sensing image fusion, channel attention, feedback network, Convolutional Neural Network (CNN), deep learning

摘要： 针对前馈卷积神经网络（CNN）感受野较小、获取上下文信息不足、其特征提取卷积层只能提取到浅层特征的问题，提出改进的基于通道注意力反馈网络的遥感图像融合算法。首先，通过两层卷积层分别初步提取全色（PAN）图像的细节特征和低分辨率多光谱（LMS）图像的光谱特征；其次，将提取的特征和网络反馈的深层特征相结合，并将其输入到通道注意力机制模块中以得到初步精细化特征；然后，经过反馈模块生成表征能力更强的深层特征；最后，将生成的深层特征经过含有反卷积的重建层，从而得到高分辨率多光谱（HMS）图像。在三个不同卫星图像数据集上的实验结果表明：所提算法能很好地提取PAN图像的细节特征和LMS图像的光谱特征，同时其恢复出来的HMS图像在主观视觉上更加清晰，并且在客观评价指标上优于对比算法，同时在均方根误差（RMSE）指标上，所提算法比传统算法降低了50%以上，比前馈卷积神经网络算法降低了10%以上。

关键词: 遥感图像融合, 通道注意力, 反馈网络, 卷积神经网络, 深度学习

CLC Number:

TP751.1

WU Lei, YANG Xiaomin. Improved remote sensing image fusion algorithm based on channel attention feedback network[J]. Journal of Computer Applications, 2021, 41(4): 1172-1178.

吴蕾, 杨晓敏. 改进的基于通道注意力反馈网络的遥感图像融合算法[J]. 计算机应用, 2021, 41(4): 1172-1178.

References

[1] VIVONE G,ALPARONE L,CHANUSSOT J,et al. A critical comparison of pansharpening algorithms[C]//Proceedings of the 2014 IEEE Geoscience and Remote Sensing Symposium. Piscataway:IEEE,2014:191-194.
[2] ALPARONE L,WALD L,CHANUSSOT J,et al. Comparison of pansharpening algorithms:outcome of the 2006 GRS-S data-fusion contest[J]. IEEE Transactions on Geoscience and Remote Sensing, 2007,45(10):3012-3021.
[3] MENG X,SHEN H,LI H,et al. Review of the pansharpening methods for remote sensing images based on the idea of metaanalysis:practical discussion and challenges[J]. Information Fusion,2019,46:102-113.
[4] 王晓艳, 刘勇, 蒋志勇. 一种基于结构相似度的IHS变换融合算法[J]. 遥感技术与应用,2011,26(5):670-676.(WANG X Y, LIU Y,JIANG Z Y. An HIS fusion method based on structural similarity[J]. Remote Sensing Technology and Application,2011, 26(5):670-676.)
[5] 曾宇燕, 何建农. 基于区域小波统计特征的遥感图像融合方法[J]. 计算机工程,2011,37(19):198-200.(ZENG Y Y,HE J N. Remote sensing image fusion method based on regional wavelet statistical features[J]. Computer Engineering,2011,37(19):198-200.)
[6] MASI G,COZZOLINO D,VERDOLIVA L,et al. Pansharpening by convolutional neural networks[J]. Remote Sensing,2016,8(7):No. 594.
[7] WEI Y,YUAN Q,SHEN H,et al. Boosting the accuracy of multispectral image pansharpening by learning a deep residual network[J]. IEEE Geoscience and Remote Sensing Letters,2017, 14(10):1795-1799.
[8] SHAO Z, CAI J. Remote sensing image fusion with deep convolutional neural network[J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing,2018,11(5):1656-1669.
[9] YANG J,FU X,HU Y,et al. PanNet:a deep network architecture for pan-sharpening[C]//Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway:IEEE,2017:1753-1761.
[10] HUPÉ J M,JAMES A C,PAYNE B R,et al. Cortical feedback improves discrimination between figure and background by V1,V2 and V3 neurons[J]. Nature,1998,394(6695):784-787.
[11] HARIS M, SHAKHNAROVICH G, UKITA N. Deep backprojection networks for super-resolution[C]//Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2018:1664-1673.
[12] LI Z,YANG J,LIU Z,et al. Feedback network for image superresolution[C]//Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2019:3862-3871.
[13] HU J,SHEN L,SUN G. Squeeze-and-excitation networks[C]//Proceedings of the 2018 IEEE/CVF International Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE, 2018:7132-7141.
[14] ZHANG Y,LI K,LI K,et al. Image super-resolution using very deep residual channel attention networks[C]//Proceedings of the 15th European Conference on Computer Vision,LNCS 11211. Cham:Springer,2018:294-310.
[15] 胡高鹏, 陈子鎏, 王晓明, 等. 基于深度渐进式反投影注意力网络的图像超分辨率重建[J]. 计算机应用,2020,40(7):2077-2083.(HU G P,CHEN Z L,WANG X M,et al. Image superresolution reconstruction based on deep progressive backprojection attention network[J]. Journal of Computer Applications,2020,40(7):2077-2083.)
[16] ALPARONE L,WALD L,CHANUSSOT J,et al. Comparison of pansharpening algorithms:outcome of the 2006 GRS-S data-fusion contest[J]. IEEE Transactions on Geoscience and Remote Sensing,2007,45(10):3012-3021.
[17] WALD L. Quality of high resolution synthesized images:is there a simple criterion[C/OL]//Proceedings of the 3rd Conference "Fusion of Earth Data:Merging Point Measurements,Raster Maps and Remotely Sensed Images"[2020-03-20]. https://hal.archives-ouvertes.fr/hal-00395027/document.
[18] WALD L,RANCHIN T,MANGOLINI M. Fusion of satellite images of different spatial resolutions:assessing the quality of resulting images[J]. Photogrammetric Engineering and Remote Sensing,1997,63(6):691-699.
[19] YUHAS R H,GOETZ A F H,BOARDMAN J W. Discrimination among semi-arid landscape endmembers using the Spectral Angle Mapper(SAM)algorithm[C]//Proceedings of the Summaries of the 3rd Annual JPL Airborne Geoscience Workshop. Pasadena, CA:JPL,1992:147-149.
[20] CHOI M. A new intensity-hue-saturation fusion approach to image fusion with a tradeoff parameter[J]. IEEE Transactions on Geoscience and Remote Sensing,2006,44(6):1672-1682.
[21] JIN B,KIM G,CHO N I. Wavelet-domain satellite image fusion based on a generalized fusion equation[J]. Journal of Applied Remote Sensing,2014,8(1):No. 080599.
[22] KANG X, LI S, BENEDIKTSSON J A. Pansharpening with matting model[J]. IEEE Transactions on Geoscience and Remote Sensing,2014,52(8):5088-5099.
[23] SHAH V P,YOUNAN N H,KING R L. An efficient pansharpening method via a combined adaptive PCA approach and contourlets[J]. IEEE Transactions on Geoscience and Remote Sensing,2008,46(5):1323-1335.
[24] TU T M,HUANG P S,HUNG C L,et al. A fast intensity-huesaturation fusion technique with spectral adjustment for IKONOS imagery[J]. IEEE Geoscience and Remote Sensing Letters, 2004,1(4):309-312.