计算机应用 ›› 2021, Vol. 41 ›› Issue (6): 1775-1784.DOI: 10.11772/j.issn.1001-9081.2020091411

所属专题: 多媒体计算与计算机仿真

• 多媒体计算与计算机仿真 • 上一篇    下一篇

基于深度学习的双通道夜视图像复原方法

牛康力, 谌雨章, 沈君凤, 曾张帆, 潘永才, 王绎冲   

  1. 湖北大学 计算机与信息工程学院, 武汉 430062
  • 收稿日期:2020-09-11 修回日期:2020-11-11 出版日期:2021-06-10 发布日期:2020-11-26
  • 通讯作者: 曾张帆
  • 作者简介:牛康力(2000-),男,湖北宜昌人,主要研究方向:人工智能、深度学习;谌雨章(1984-),男,湖北武汉人,副教授,博士,主要研究方向:光电探测、图像处理;沈君凤(1977-),女,湖北随州人,副教授,博士,主要研究方向:光通信、信号处理;曾张帆(1983-),男,湖北武汉人,副教授,博士,主要研究方向:5G无线通信、信号处理;潘永才(1964-),男,湖北潜江人,教授,主要研究方向:光通信、信号处理;王绎冲(2000-),男,广东广州人,主要研究方向:图像处理、深度学习。
  • 基金资助:
    湖北省教育厅科学技术研究计划重点项目(D20181001)。

Dual-channel night vision image restoration method based on deep learning

NIU Kangli, CHEN Yuzhang, SHEN Junfeng, ZENG Zhangfan, PAN Yongcai, WANG Yichong   

  1. School of Computer Science and Information Engineering, Hubei University, Wuhan Hubei 430062, China
  • Received:2020-09-11 Revised:2020-11-11 Online:2021-06-10 Published:2020-11-26
  • Supported by:
    This work is partially supported by the Key Project of Science and Technology Research Plan of Education Department of Hubei Province (D20181001).

摘要: 针对夜间场景光线微弱、能见度低导致夜视图像信噪比低、成像质量差的问题,提出了基于深度学习的双通道夜视图像复原方法。首先,用两种基于全连接多尺度残差学习分块(FMRB)的卷积神经网络(CNN)分别对红外夜视图像和微光夜视图像进行多尺度特征提取和层次特征融合,从而得到重建的红外图像和增强的微光图像;然后,两种处理后的图像通过自适应加权平均算法进行融合,并根据场景的不同自适应地凸显两个图像中具有更高显著性的有效信息;最后,得到分辨率高且视觉效果好的夜视复原图像。使用基于FMRB的深度学习网络得到的红外夜视重建图像,相较于卷积神经网络超分辨率(SRCNN)重建算法得到的在峰值信噪比(PSNR)和结构相似性(SSIM)的平均值上分别提升了3.56 dB和0.091 2;相较于MSRCR,得到的微光夜视增强图像在PSNR和SSIM的平均值上分别提升了6.82 dB和0.132 1。实验结果表明,所提方法得到的重建图像的清晰度明显得到改善,获得的增强图像的亮度也明显得到提升,而且前二者的融合图像的视觉效果较好,可见所提方法能有效改善夜视图像的复原效果。

关键词: 卷积神经网络, 多尺度残差学习, 超分辨率重建, 图像增强, 图像融合, 夜视图像复原, 红外夜视图像, 微光夜视图像

Abstract: Due to the low light level and low visibility of night scene, there are many problems in night vision image, such as low signal to noise ratio and low imaging quality. To solve the problems, a dual-channel night vision image restoration method based on deep learning was proposed. Firstly, two Convolutional Neural Network (CNN) based on Fully connected Multi-scale Residual learning Block (FMRB) were used to extract multi-scale features and fuse hierarchical features of infrared night vision images and low-light-level night vision images respectively, so as to obtain the reconstructed infrared image and enhanced low-light-level image. Then, the two processed images were fused by the adaptive weighted averaging algorithm, and the effective information of the more salient one in the two images was highlighted adaptively according to the different scenes. Finally, the night vision restoration images with high resolution and good visual effect were obtained. The reconstructed infrared night vision image obtained by the FMRB based deep learning network had the average values of Peak Signal to Noise Ratio (PSNR) and Structural Similarity (SSIM) by 3.56 dB and 0.091 2 higher than the image obtained by Super-Resolution Convolutional Neural Network (SRCNN) reconstruction algorithm respectively, and the enhanced low-light-level night vision image obtained by the FMRB based deep learning network had the average values of PSNR and SSIM by 6.82dB and 0.132 1 higher than the image obtained by Multi-Scale Retinex with Color Restoration (MSRCR). Experimental results show that, by using the proposed method, the resolution of reconstructed image is improved obviously and the brightness of the enhanced image is also improved significantly, and the visual effect of the fusion image obtained by the above two images is better. It can be seen that the proposed algorithm can effectively restore the night vision images.

Key words: Convolution Neural Network (CNN), multi-scale residual learning, super-resolution reconstruction, image enhancement, image fusion, night vision image restoration, infrared night vision image, low-light-level night vision image

中图分类号: