基于深度学习的双通道夜视图像复原方法

doi:10.11772/j.issn.1001-9081.2020091411

计算机应用 ›› 2021, Vol. 41 ›› Issue (6): 1775-1784.DOI: 10.11772/j.issn.1001-9081.2020091411

所属专题：多媒体计算与计算机仿真

• 多媒体计算与计算机仿真 • 上一篇下一篇

基于深度学习的双通道夜视图像复原方法

牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲

湖北大学计算机与信息工程学院, 武汉 430062

收稿日期:2020-09-11 修回日期:2020-11-11 出版日期:2021-06-10 发布日期:2020-11-26
通讯作者: 曾张帆
作者简介:牛康力(2000-),男,湖北宜昌人,主要研究方向:人工智能、深度学习;谌雨章(1984-),男,湖北武汉人,副教授,博士,主要研究方向:光电探测、图像处理;沈君凤(1977-),女,湖北随州人,副教授,博士,主要研究方向:光通信、信号处理;曾张帆(1983-),男,湖北武汉人,副教授,博士,主要研究方向:5G无线通信、信号处理;潘永才(1964-),男,湖北潜江人,教授,主要研究方向:光通信、信号处理;王绎冲(2000-),男,广东广州人,主要研究方向:图像处理、深度学习。
基金资助:
湖北省教育厅科学技术研究计划重点项目（D20181001）。

Dual-channel night vision image restoration method based on deep learning

NIU Kangli, CHEN Yuzhang, SHEN Junfeng, ZENG Zhangfan, PAN Yongcai, WANG Yichong

School of Computer Science and Information Engineering, Hubei University, Wuhan Hubei 430062, China

Received:2020-09-11 Revised:2020-11-11 Online:2021-06-10 Published:2020-11-26
Supported by:
This work is partially supported by the Key Project of Science and Technology Research Plan of Education Department of Hubei Province (D20181001).

摘要/Abstract

摘要： 针对夜间场景光线微弱、能见度低导致夜视图像信噪比低、成像质量差的问题，提出了基于深度学习的双通道夜视图像复原方法。首先，用两种基于全连接多尺度残差学习分块（FMRB）的卷积神经网络（CNN）分别对红外夜视图像和微光夜视图像进行多尺度特征提取和层次特征融合，从而得到重建的红外图像和增强的微光图像；然后，两种处理后的图像通过自适应加权平均算法进行融合，并根据场景的不同自适应地凸显两个图像中具有更高显著性的有效信息；最后，得到分辨率高且视觉效果好的夜视复原图像。使用基于FMRB的深度学习网络得到的红外夜视重建图像，相较于卷积神经网络超分辨率（SRCNN）重建算法得到的在峰值信噪比（PSNR）和结构相似性（SSIM）的平均值上分别提升了3.56 dB和0.091 2；相较于MSRCR，得到的微光夜视增强图像在PSNR和SSIM的平均值上分别提升了6.82 dB和0.132 1。实验结果表明，所提方法得到的重建图像的清晰度明显得到改善，获得的增强图像的亮度也明显得到提升，而且前二者的融合图像的视觉效果较好，可见所提方法能有效改善夜视图像的复原效果。

关键词: 卷积神经网络, 多尺度残差学习, 超分辨率重建, 图像增强, 图像融合, 夜视图像复原, 红外夜视图像, 微光夜视图像

Abstract: Due to the low light level and low visibility of night scene, there are many problems in night vision image, such as low signal to noise ratio and low imaging quality. To solve the problems, a dual-channel night vision image restoration method based on deep learning was proposed. Firstly, two Convolutional Neural Network (CNN) based on Fully connected Multi-scale Residual learning Block (FMRB) were used to extract multi-scale features and fuse hierarchical features of infrared night vision images and low-light-level night vision images respectively, so as to obtain the reconstructed infrared image and enhanced low-light-level image. Then, the two processed images were fused by the adaptive weighted averaging algorithm, and the effective information of the more salient one in the two images was highlighted adaptively according to the different scenes. Finally, the night vision restoration images with high resolution and good visual effect were obtained. The reconstructed infrared night vision image obtained by the FMRB based deep learning network had the average values of Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) by 3.56 dB and 0.091 2 higher than the image obtained by Super-Resolution Convolutional Neural Network (SRCNN) reconstruction algorithm respectively, and the enhanced low-light-level night vision image obtained by the FMRB based deep learning network had the average values of PSNR and SSIM by 6.82dB and 0.132 1 higher than the image obtained by Multi-Scale Retinex with Color Restoration (MSRCR). Experimental results show that, by using the proposed method, the resolution of reconstructed image is improved obviously and the brightness of the enhanced image is also improved significantly, and the visual effect of the fusion image obtained by the above two images is better. It can be seen that the proposed algorithm can effectively restore the night vision images.

Key words: Convolution Neural Network (CNN), multi-scale residual learning, super-resolution reconstruction, image enhancement, image fusion, night vision image restoration, infrared night vision image, low-light-level night vision image

中图分类号:

牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲. 基于深度学习的双通道夜视图像复原方法[J]. 计算机应用, 2021, 41(6): 1775-1784.

NIU Kangli, CHEN Yuzhang, SHEN Junfeng, ZENG Zhangfan, PAN Yongcai, WANG Yichong. Dual-channel night vision image restoration method based on deep learning[J]. Journal of Computer Applications, 2021, 41(6): 1775-1784.

参考文献

[1] 石永彪, 张湧. 车载红外夜视技术发展研究综述[J]. 红外技术, 2019,41(6):504-510. (SHI Y B, ZHANG Y. Survey on development about vehicular infrared night vision technology[J]. Infrared Technology,2019,41(6):504-510.)
[2] 陈军, 张长江. 基于小波域反正切变换的红外图像增强算法[J]. 计算机工程, 2013, 39(1):248-251.(CHEN J,ZHANG C J. Infrared image enhancement algorithm based on arc-tangent transform in wavelet domain[J]. Computer Engineering,2013,39(1):248-251.)
[3] 曹海杰, 刘宁, 许吉, 等. 红外图像自适应逆直方图增强技术[J]. 红外与激光工程, 2020, 49(4):0426003.1-0426003.7.(CAO H J,LIU N,XU J, et al. Infrared image adaptive inverse histogram enhancement technology[J]. Infrared and Laser Engineering, 2020,49(4):0426003.1-0426003.7.)
[4] LIU X,CHEN Y,PENG Z,et al. Infrared image super-resolution reconstruction based on quaternion and high-order overlapping group sparse total variation[J]. Sensors,2019,19(23):Article No. 5139.
[5] 王瑞尧, 岳雪亭, 周志青, 等. 最大差值图决策的低照度图像自适应增强算法[J]. 计算机应用, 2020, 40(4):1164-1170.(WANG R Y, YUE X T, ZHOU Z Q, et al. Adaptive enhancement algorithm of low illumination image based on maximum difference image decision[J]. Journal of Computer Applications,2020,40(4):1164-1170.)
[6] 刘小园, 衣扬, 杨磊. 基于时域自适应滤波及非局部平均的夜视图像去噪算法[J]. 计算机应用研究, 2018, 35(6):1917-1920. (LIU X Y,YI Y,YANG L. Night vision image denoising algorithm based on time domain adaptive filtering and non-local means[J]. Application Research of Computers,2018,35(6):1917-1920.)
[7] BU L,XU Z,ZHANG G,et al. Night-light image restoration method based on night scattering model for Luojia 1-01 satellite[J]. Sensors,2019,19(17):Article No. 3761.
[8] MA S,MA H,XU Y,et al. A low-light sensor image enhancement algorithm based on HSI color model[J]. Sensors,2018,18(10):Article No. 3583.
[9] 朱浩然, 刘云清, 张文颖. 基于灰度变换与两尺度分解的夜视图像融合[J]. 电子与信息学报, 2019, 41(3):640-648.(ZHU H R, LIU Y Q,ZHANG W Y. Night-vision image fusion based on intensity transformation and two-scale decomposition[J]. Journal of Electronics and Information Technology,2019,41(3):640-648.)
[10] 杨钒, 钱立志, 刘晓, 等. 红外与微光图像开窗配准融合处理方法[J]. 激光与红外, 2018, 48(8):1060-1064.(YANG F,QIAN L Z,LIU X,et al. The window registration and fusion processing method for infrared and low light level images[J]. Laser and Infrared,2018,48(8):1060-1064.)
[11] BAMRUNGTHAI P,WONGKAMCHANG P. Development of a thermal/visible image fusion system for situation awareness[C]//Proceedings of the 2018 5th International Conference on Advanced Informatics:Concept Theory and Applications. Piscataway:IEEE,2018:96-100.
[12] LIU Y,DONG L,JI Y,et al. Infrared and visible image fusion through details preservation[J]. Sensors,2019,19(20):Article No. 4556.
[13] 邹鹏, 谌雨章, 陈龙彪, 等. 基于神经网络的光照分布预测夜视复原算法[J]. 计算机科学, 2019, 46(11A):329-333, 340.(ZOU P,CHEN Y Z,CHEN L B,et al. Night vision restoration algorithm based on neural network for illumination distribution prediction[J]. Computer Science, 2019, 46(11A):329-333,340.)
[14] 王丹, 陈亮. 基于深度学习的红外夜视图像超分辨率重建[J]. 红外技术, 2019, 41(10):963-969.(WANG D,CHEN L. Superresolution reconstruction of infrared images in night environments based on deep-learning[J]. Infrared Technology,2019,41(10):963-969.)
[15] HE K,ZHANG X,REN S,et al. Spatial pyramid pooling in deep convolutional networks for visual recognition[J]. IEEE Transactions on Pattern Analysis and Machine Intelligence,2015, 37(9):1904-1916.
[16] 陈龙彪, 谌雨章, 王晓晨, 等. 基于深度学习的水下图像超分辨率重建方法[J]. 计算机应用, 2019, 39(9):2738-2743.(CHEN L B,CHEN Y Z,WANG X C,et al. Underwater image superresolution reconstruction method based on deep learning[J]. Journal of Computer Applications,2019,39(9):2738-2743.)
[17] SCHAEFER G,STICH M. UCID:an uncompressed color image database[C]//Proceedings of the SPIE 5307, Storage and Retrieval Methods and Applications for Multimedia 2004. Bellingham:SPIE,2004:472-480.
[18] CHEN Y, CHEN J. Spatial adaptive regularized MAP reconstruction for LD-based night vision[J]. Optik,2014,125(13):3162-3165.
[19] CHEN Y,YANG W,TAN H,et al. Image enhancement for LD based imaging in turbid water[J]. Optik,2016,127(2):517-521.
[20] DONG C,LOY C C,HE K,et al. Learning a deep convolutional network for image super-resolution[C]//Proceedings of the 2014 European Conference on Computer Vision,LNCS 8692. Cham:Springer,2014:184-199.
[21] KIM J,LEE J K,LEE K M. Accurate image super-resolution using very deep convolutional networks[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1646-1654.
[22] KIM J,LEE J K,LEE K M. Deeply-recursive convolutional network for image super-resolution[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:1637-1645.
[23] LIM B,SON S,KIM H,et al. Enhanced deep residual networks for single image super-resolution[C]//Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition Workshops. Piscataway:IEEE,2017:1132-1140.
[24] FU X,ZENG D,HUANG Y,et al. A weighted variational model for simultaneous reflectance and illumination estimation[C]//Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway:IEEE,2016:2782-2790.
[25] JOBSON D J,RAHMAN Z,WOODELL G A. A multiscale retinex for bridging the gap between color images and the human observation of scenes[J]. IEEE Transactions on Image Processing,1997,6(7):965-976.
[26] GUO X,LI Y,LING H. LIME:low-light image enhancement via illumination map estimation[J]. IEEE Transactions on Image Processing,2017,26(2):982-993.
[27] NGUYEN C T,HAVLICEK J P. Linear adaptive infrared image fusion[C]//Proceedings of the 2014 Southwest Symposium on Image Analysis and Interpretation. Piscataway:IEEE, 2014:117-120.

基于深度学习的双通道夜视图像复原方法

Dual-channel night vision image restoration method based on deep learning

PDF

可视化

摘要/Abstract

引用本文

使用本文

参考文献

相关文章 15

编辑推荐

Metrics

[1]	宋中山, 梁家锐, 郑禄, 刘振宇, 帖军. 基于双向门控尺度特征融合的遥感场景分类[J]. 计算机应用, 2021, 41(9): 2726-2735.
[2]	李康康, 张静. 基于注意力机制的多层次编码和解码的图像描述模型[J]. 计算机应用, 2021, 41(9): 2504-2509.
[3]	张永斌, 常文欣, 孙连山, 张航. 基于字典的域名生成算法生成域名的检测方法[J]. 计算机应用, 2021, 41(9): 2609-2614.
[4]	赵宏, 孔东一. 图像特征注意力与自适应注意力融合的图像内容中文描述[J]. 计算机应用, 2021, 41(9): 2496-2503.
[5]	徐江浪, 李林燕, 万新军, 胡伏原. 结合目标检测的室内场景识别方法[J]. 计算机应用, 2021, 41(9): 2720-2725.
[6]	牟长宁, 王海鹏, 周丕宇, 侯鑫行. 基于图卷积神经网络的串联质谱从头测序[J]. 计算机应用, 2021, 41(9): 2773-2779.
[7]	王贺兵, 张春梅. 基于非对称卷积-压缩激发-次代残差网络的人脸关键点检测[J]. 计算机应用, 2021, 41(9): 2741-2747.
[8]	曾祥银, 郑伯川, 刘丹. 基于深度卷积神经网络和聚类的左右轨道线检测[J]. 计算机应用, 2021, 41(8): 2324-2329.
[9]	曹玉红, 徐海, 刘荪傲, 王紫霄, 李宏亮. 基于深度学习的医学影像分割研究综述[J]. 计算机应用, 2021, 41(8): 2273-2287.
[10]	秦斌斌, 彭良康, 卢向明, 钱江波. 司机分心驾驶检测研究进展[J]. 计算机应用, 2021, 41(8): 2330-2337.
[11]	黄程程, 董霄霄, 李钊. 基于二维Winograd算法的深流水线5×5卷积方法[J]. 计算机应用, 2021, 41(8): 2258-2264.
[12]	高钦泉, 黄炳城, 刘文哲, 童同. 基于改进CenterNet的竹条表面缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1933-1938.
[13]	谭道强, 曾诚, 乔金霞, 张俊. 基于混合注意力模型的阴影检测方法[J]. 计算机应用, 2021, 41(7): 2076-2081.
[14]	武光利, 李雷霆, 郭振洲, 王成祥. 基于改进的双向长短期记忆网络的视频摘要生成模型[J]. 计算机应用, 2021, 41(7): 1908-1914.
[15]	吴则举, 焦翠娟, 陈亮. 基于改进Faster R-CNN的轮胎缺陷检测方法[J]. 计算机应用, 2021, 41(7): 1939-1946.