Dual-channel night vision image restoration method based on deep learning

doi:10.11772/j.issn.1001-9081.2020091411

Abstract

Abstract: Due to the low light level and low visibility of night scene, there are many problems in night vision image, such as low signal to noise ratio and low imaging quality. To solve the problems, a dual-channel night vision image restoration method based on deep learning was proposed. Firstly, two Convolutional Neural Network (CNN) based on Fully connected Multi-scale Residual learning Block (FMRB) were used to extract multi-scale features and fuse hierarchical features of infrared night vision images and low-light-level night vision images respectively, so as to obtain the reconstructed infrared image and enhanced low-light-level image. Then, the two processed images were fused by the adaptive weighted averaging algorithm, and the effective information of the more salient one in the two images was highlighted adaptively according to the different scenes. Finally, the night vision restoration images with high resolution and good visual effect were obtained. The reconstructed infrared night vision image obtained by the FMRB based deep learning network had the average values of Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) by 3.56 dB and 0.091 2 higher than the image obtained by Super-Resolution Convolutional Neural Network (SRCNN) reconstruction algorithm respectively, and the enhanced low-light-level night vision image obtained by the FMRB based deep learning network had the average values of PSNR and SSIM by 6.82dB and 0.132 1 higher than the image obtained by Multi-Scale Retinex with Color Restoration (MSRCR). Experimental results show that, by using the proposed method, the resolution of reconstructed image is improved obviously and the brightness of the enhanced image is also improved significantly, and the visual effect of the fusion image obtained by the above two images is better. It can be seen that the proposed algorithm can effectively restore the night vision images.

Key words: Convolution Neural Network (CNN), multi-scale residual learning, super-resolution reconstruction, image enhancement, image fusion, night vision image restoration, infrared night vision image, low-light-level night vision image

摘要： 针对夜间场景光线微弱、能见度低导致夜视图像信噪比低、成像质量差的问题，提出了基于深度学习的双通道夜视图像复原方法。首先，用两种基于全连接多尺度残差学习分块（FMRB）的卷积神经网络（CNN）分别对红外夜视图像和微光夜视图像进行多尺度特征提取和层次特征融合，从而得到重建的红外图像和增强的微光图像；然后，两种处理后的图像通过自适应加权平均算法进行融合，并根据场景的不同自适应地凸显两个图像中具有更高显著性的有效信息；最后，得到分辨率高且视觉效果好的夜视复原图像。使用基于FMRB的深度学习网络得到的红外夜视重建图像，相较于卷积神经网络超分辨率（SRCNN）重建算法得到的在峰值信噪比（PSNR）和结构相似性（SSIM）的平均值上分别提升了3.56 dB和0.091 2；相较于MSRCR，得到的微光夜视增强图像在PSNR和SSIM的平均值上分别提升了6.82 dB和0.132 1。实验结果表明，所提方法得到的重建图像的清晰度明显得到改善，获得的增强图像的亮度也明显得到提升，而且前二者的融合图像的视觉效果较好，可见所提方法能有效改善夜视图像的复原效果。

关键词: 卷积神经网络, 多尺度残差学习, 超分辨率重建, 图像增强, 图像融合, 夜视图像复原, 红外夜视图像, 微光夜视图像

CLC Number:

NIU Kangli, CHEN Yuzhang, SHEN Junfeng, ZENG Zhangfan, PAN Yongcai, WANG Yichong. Dual-channel night vision image restoration method based on deep learning[J]. Journal of Computer Applications, 2021, 41(6): 1775-1784.

牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲. 基于深度学习的双通道夜视图像复原方法[J]. 计算机应用, 2021, 41(6): 1775-1784.

References

[1] 石永彪, 张湧. 车载红外夜视技术发展研究综述[J]. 红外技术, 2019,41(6):504-510. (SHI Y B, ZHANG Y. Survey on development about vehicular infrared night vision technology[J]. Infrared Technology,2019,41(6):504-510.)
[2] 陈军, 张长江. 基于小波域反正切变换的红外图像增强算法[J]. 计算机工程, 2013, 39(1):248-251.(CHEN J,ZHANG C J. Infrared image enhancement algorithm based on arc-tangent transform in wavelet domain[J]. Computer Engineering,2013,39(1):248-251.)
[3] 曹海杰, 刘宁, 许吉, 等. 红外图像自适应逆直方图增强技术[J]. 红外与激光工程, 2020, 49(4):0426003.1-0426003.7.(CAO H J,LIU N,XU J, et al. Infrared image adaptive inverse histogram enhancement technology[J]. Infrared and Laser Engineering, 2020,49(4):0426003.1-0426003.7.)
[4] LIU X,CHEN Y,PENG Z,et al. Infrared image super-resolution reconstruction based on quaternion and high-order overlapping group sparse total variation[J]. Sensors,2019,19(23):Article No. 5139.
[5] 王瑞尧, 岳雪亭, 周志青, 等. 最大差值图决策的低照度图像自适应增强算法[J]. 计算机应用, 2020, 40(4):1164-1170.(WANG R Y, YUE X T, ZHOU Z Q, et al. Adaptive enhancement algorithm of low illumination image based on maximum difference image decision[J]. Journal of Computer Applications,2020,40(4):1164-1170.)
[6] 刘小园, 衣扬, 杨磊. 基于时域自适应滤波及非局部平均的夜视图像去噪算法[J]. 计算机应用研究, 2018, 35(6):1917-1920. (LIU X Y,YI Y,YANG L. Night vision image denoising algorithm based on time domain adaptive filtering and non-local means[J]. Application Research of Computers,2018,35(6):1917-1920.)
[7] BU L,XU Z,ZHANG G,et al. Night-light image restoration method based on night scattering model for Luojia 1-01 satellite[J]. Sensors,2019,19(17):Article No. 3761.
[8] MA S,MA H,XU Y,et al. A low-light sensor image enhancement algorithm based on HSI color model[J]. Sensors,2018,18(10):Article No. 3583.
[9] 朱浩然, 刘云清, 张文颖. 基于灰度变换与两尺度分解的夜视图像融合[J]. 电子与信息学报, 2019, 41(3):640-648.(ZHU H R, LIU Y Q,ZHANG W Y. Night-vision image fusion based on intensity transformation and two-scale decomposition[J]. Journal of Electronics and Information Technology,2019,41(3):640-648.)
[10] 杨钒, 钱立志, 刘晓, 等. 红外与微光图像开窗配准融合处理方法[J]. 激光与红外, 2018, 48(8):1060-1064.(YANG F,QIAN L Z,LIU X,et al. The window registration and fusion processing method for infrared and low light level images[J]. Laser and Infrared,2018,48(8):1060-1064.)
[11] BAMRUNGTHAI P,WONGKAMCHANG P. Development of a thermal/visible image fusion system for situation awareness[C]//Proceedings of the 2018 5th International Conference on Advanced Informatics:Concept Theory and Applications. Piscataway:IEEE,2018:96-100.
[12] LIU Y,DONG L,JI Y,et al. Infrared and visible image fusion through details preservation[J]. Sensors,2019,19(20):Article No. 4556.
[13] 邹鹏, 谌雨章, 陈龙彪, 等. 基于神经网络的光照分布预测夜视复原算法[J]. 计算机科学, 2019, 46(11A):329-333, 340.(ZOU P,CHEN Y Z,CHEN L B,et al. Night vision restoration algorithm based on neural network for illumination distribution prediction[J]. Computer Science, 2019, 46(11A):329-333,340.)
[14] 王丹, 陈亮. 基于深度学习的红外夜视图像超分辨率重建[J]. 红外技术, 2019, 41(10):963-969.(WANG D,CHEN L. Superresolution reconstruction of infrared images in night environments based on deep-learning[J]. Infrared Technology,2019,41(10):963-969.)
[15] HE K,ZHANG X,REN S,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2015, 37(9):1904-1916.
[16] 陈龙彪, 谌雨章, 王晓晨, 等. 基于深度学习的水下图像超分辨率重建方法[J]. 计算机应用, 2019, 39(9):2738-2743.(CHEN L B,CHEN Y Z,WANG X C,et al. Underwater image superresolution reconstruction method based on deep learning[J]. Journal of Computer Applications,2019,39(9):2738-2743.)
[17] SCHAEFER G,STICH M. UCID:an uncompressed color image database[C]//Proceedings of the SPIE 5307, Storage and Retrieval Methods and Applications for Multimedia 2004. Bellingham:SPIE,2004:472-480.
[18] CHEN Y, CHEN J. Spatial adaptive regularized MAP reconstruction for LD-based night vision[J]. Optik,2014,125(13):3162-3165.
[19] CHEN Y,YANG W,TAN H,et al. Image enhancement for LD based imaging in turbid water[J]. Optik,2016,127(2):517-521.
[20] DONG C,LOY C C,HE K,et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of the 2014 European Conference on Computer Vision,LNCS 8692. Cham:Springer,2014:184-199.
[21] KIM J,LEE J K,LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1646-1654.
[22] KIM J,LEE J K,LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1637-1645.
[23] LIM B,SON S,KIM H,et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2017:1132-1140.
[24] FU X,ZENG D,HUANG Y,et al. A weighted variational model for simultaneous reflectance and illumination estimation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:2782-2790.
[25] JOBSON D J,RAHMAN Z,WOODELL G A. A multiscale retinex for bridging the gap between color images and the human observation of scenes[J]. IEEE Transactions on Image Processing,1997,6(7):965-976.
[26] GUO X,LI Y,LING H. LIME:low-light image enhancement via illumination map estimation[J]. IEEE Transactions on Image Processing,2017,26(2):982-993.
[27] NGUYEN C T,HAVLICEK J P. Linear adaptive infrared image fusion[C]//Proceedings of the 2014 Southwest Symposium on Image Analysis and Interpretation. Piscataway:IEEE, 2014:117-120.