Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (7): 2183-2191.DOI: 10.11772/j.issn.1001-9081.2023070976

• Multimedia computing and computer simulation • Previous Articles     Next Articles

Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention

Wenliang WEI1,2(), Yangping WANG1,2, Biao YUE1,2, Anzheng WANG1,2, Zhe ZHANG1,2   

  1. 1.School of Electronic and Information Engineering,Lanzhou Jiaotong University,Lanzhou Gansu 730070,China
    2.Gansu Artificial Intelligence and Graphics and Image Processing Engineering Research Center (Lanzhou Jiaotong University),Lanzhou Gansu 730070,China
  • Received:2023-07-19 Revised:2023-10-06 Accepted:2023-10-10 Online:2023-10-26 Published:2024-07-10
  • Contact: Wenliang WEI
  • About author:WANG Yangping, born in 1973, Ph. D., professor. Her research interests include digital image processing, computer vision.
    YUE Biao, born in 1994, Ph. D. candidate. His research interests include machine vision, intelligent information processing.
    WANG Anzheng, born in 1999, M. S. candidate. His research interests include intelligent information processing, pavement disease detection.
    ZHANG Zhe, born in 1996, M. S. candidate. His research interests include intelligent information processing.
    First author contact:WEI Wenliang, born in 1997, M. S. candidate. His research interests include intelligent information processing, image fusion.
  • Supported by:
    Central Guiding Local Science and Technology Development Fund(22ZY1QA002);Gansu Province Intellectual Property Program(21ZSCQ013);Gansu Province Key Research and Development Program(21YF5GA158);Gansu Province Education Technology Innovation Project(2021jyjbgs-05)

基于光照权重分配和注意力的红外与可见光图像融合深度学习模型

魏文亮1,2(), 王阳萍1,2, 岳彪1,2, 王安政1,2, 张哲1,2   

  1. 1.兰州交通大学 电子与信息工程学院, 兰州 730070
    2.甘肃省人工智能与图形图像处理工程研究中心(兰州交通大学), 兰州 730070
  • 通讯作者: 魏文亮
  • 作者简介:王阳萍(1973—),女,四川达州人,教授,博士,CCF会员,主要研究方向:数字图像处理、计算机视觉;
    岳彪(1994—),男,甘肃平凉人,博士研究生,主要研究方向:机器视觉、智能信息处理;
    王安政(1999—),男,山东邹平人,硕士研究生,主要研究方向:智能信息处理、路面病害检测;
    张哲(1996—),男,辽宁本溪人,硕士研究生,主要研究方向:智能信息处理。
    第一联系人:魏文亮(1997—),男,甘肃正宁人,硕士研究生,主要研究方向:智能信息处理、图像融合;
  • 基金资助:
    中央引导地方科技发展资金资助项目(22ZY1QA002);甘肃省知识产权计划项目(21ZSCQ013);甘肃省重点研发计划项目(21YF5GA158);甘肃省教育科技创新项目(2021jyjbgs-05)

Abstract:

Existing infrared and visible image fusion models ignore illumination factors in fusion process and use conventional fusion strategies, leading to the fusion results with the loss of detail information and inconspicuous salient information. To solve these problems, a deep learning model for infrared and visible image fusion based on illumination weight allocation and attention was proposed. Firstly, an Illumination Weight Allocation Network (IWA-Net) was designed to estimate the illumination distribution and calculate illumination weights. Secondly, a CM-L1-norm fusion strategy was introduced to enhance the dependency between pixels and achieve smooth processing of salient features. Finally, a decoding network composed of fully convolutional layers was employed to reconstruct fused images. The results of the fusion experiments on publicly available datasets show that the proposed model outperforms the contrastive models, with improvements observed in all six selected evaluation metrics; specifically, the Spatial Frequency (SF) and Mutual Information (MI) metrics increase by 45% and 41% in average, respectively. The proposed model effectively reduces edge blurring and enhances clarity and contrast of the fused images. The fusion results of the proposed model exhibits superior performance in both subjective and objective aspects compared to other contrastive models.

Key words: infrared and visible image, image fusion, illumination weight allocation, fusion strategy, attention mechanism

摘要:

针对现有红外与可见光图像融合模型在融合过程中忽略光照因素、使用常规的融合策略,导致融合结果存在细节信息丢失、显著信息不明显等问题,提出一种基于光照权重分配和注意力的红外与可见光图像融合深度学习模型。首先,设计光照权重分配网络(IWA-Net)来估计光照分布并计算光照权重;其次,引入CM-L1范式融合策略提高像素之间的依赖关系,完成对显著特征的平滑处理;最后,由全卷积层构成解码网络,完成对融合图像的重构。在公开数据集上的融合实验结果表明,所提模型相较于对比模型,所选六种评价指标均有所提高,其中空间频率(SF)和互信息(MI)指标分别平均提高了45%和41%,有效减少边缘模糊,使融合图像具有较高的清晰度和对比度。该模型的融合结果在主客观方面均优于其他对比模型。

关键词: 红外与可见光图像, 图像融合, 光照权重分配, 融合策略, 注意力机制

CLC Number: