Journal of Computer Applications ›› 2024, Vol. 44 ›› Issue (7): 2183-2191.DOI: 10.11772/j.issn.1001-9081.2023070976
• Multimedia computing and computer simulation • Previous Articles Next Articles
Wenliang WEI1,2(), Yangping WANG1,2, Biao YUE1,2, Anzheng WANG1,2, Zhe ZHANG1,2
Received:
2023-07-19
Revised:
2023-10-06
Accepted:
2023-10-10
Online:
2023-10-26
Published:
2024-07-10
Contact:
Wenliang WEI
About author:
WANG Yangping, born in 1973, Ph. D., professor. Her research interests include digital image processing, computer vision.Supported by:
魏文亮1,2(), 王阳萍1,2, 岳彪1,2, 王安政1,2, 张哲1,2
通讯作者:
魏文亮
作者简介:
王阳萍(1973—),女,四川达州人,教授,博士,CCF会员,主要研究方向:数字图像处理、计算机视觉;基金资助:
CLC Number:
Wenliang WEI, Yangping WANG, Biao YUE, Anzheng WANG, Zhe ZHANG. Deep learning model for infrared and visible image fusion based on illumination weight allocation and attention[J]. Journal of Computer Applications, 2024, 44(7): 2183-2191.
魏文亮, 王阳萍, 岳彪, 王安政, 张哲. 基于光照权重分配和注意力的红外与可见光图像融合深度学习模型[J]. 《计算机应用》唯一官方网站, 2024, 44(7): 2183-2191.
Add to citation manager EndNote|Ris|BibTeX
URL: https://www.joca.cn/EN/10.11772/j.issn.1001-9081.2023070976
模型 | EN | MI | VIF | SCD | SF | |
---|---|---|---|---|---|---|
CSF | 5.370 6 | 2.140 8 | 0.548 2 | 1.222 0 | 7.760 8 | 0.296 4 |
GTF | 5.405 2 | 1.442 8 | 0.524 4 | 0.823 6 | 11.075 6 | 0.428 4 |
IFCNN | 5.813 2 | 2.201 0 | 0.666 6 | 1.324 8 | 12.194 8 | 0.543 2 |
MDLatRR | 5.781 2 | 2.316 2 | 0.858 2 | 1.324 4 | 8.932 0 | 0.537 6 |
DenseFuse | 5.276 2 | 2.480 8 | 0.603 8 | 1.139 4 | 7.020 8 | 0.298 0 |
PIAFusion | 5.935 6 | 3.688 4 | 0.855 0 | 1.601 8 | 12.682 8 | 0.645 6 |
FusionGAN | 5.403 2 | 1.766 8 | 0.473 8 | 1.007 4 | 4.872 0 | 0.140 4 |
U2Fusion | 5.347 6 | 2.404 8 | 0.584 8 | 1.223 8 | 6.541 6 | 0.270 6 |
IA-Fusion | 6.284 2 | 3.981 8 | 0.887 6 | 1.714 0 | 14.589 8 | 0.634 2 |
Tab. 1 Mean evaluation metrics for group “07D”
模型 | EN | MI | VIF | SCD | SF | |
---|---|---|---|---|---|---|
CSF | 5.370 6 | 2.140 8 | 0.548 2 | 1.222 0 | 7.760 8 | 0.296 4 |
GTF | 5.405 2 | 1.442 8 | 0.524 4 | 0.823 6 | 11.075 6 | 0.428 4 |
IFCNN | 5.813 2 | 2.201 0 | 0.666 6 | 1.324 8 | 12.194 8 | 0.543 2 |
MDLatRR | 5.781 2 | 2.316 2 | 0.858 2 | 1.324 4 | 8.932 0 | 0.537 6 |
DenseFuse | 5.276 2 | 2.480 8 | 0.603 8 | 1.139 4 | 7.020 8 | 0.298 0 |
PIAFusion | 5.935 6 | 3.688 4 | 0.855 0 | 1.601 8 | 12.682 8 | 0.645 6 |
FusionGAN | 5.403 2 | 1.766 8 | 0.473 8 | 1.007 4 | 4.872 0 | 0.140 4 |
U2Fusion | 5.347 6 | 2.404 8 | 0.584 8 | 1.223 8 | 6.541 6 | 0.270 6 |
IA-Fusion | 6.284 2 | 3.981 8 | 0.887 6 | 1.714 0 | 14.589 8 | 0.634 2 |
模型 | EN | MI | VIF | SCD | SF | |
---|---|---|---|---|---|---|
CSF | 5.674 2 | 2.538 1 | 0.560 2 | 1.310 2 | 7.012 0 | 0.312 2 |
GTF | 5.472 9 | 2.042 9 | 0.500 1 | 0.803 1 | 9.201 3 | 0.410 2 |
IFCNN | 5.946 2 | 2.388 6 | 0.665 4 | 1.412 3 | 10.988 4 | 0.532 9 |
MDLatRR | 5.976 3 | 2.702 4 | 0.885 9 | 1.362 4 | 8.001 2 | 0.510 2 |
DenseFuse | 5.499 6 | 3.012 1 | 0.612 3 | 1.247 6 | 6.013 8 | 0.301 1 |
PIAFusion | 6.283 9 | 3.623 5 | 0.922 1 | 1.821 3 | 11.194 2 | 0.694 2 |
FusionGAN | 5.401 2 | 2.224 2 | 0.432 1 | 1.012 6 | 4.361 3 | 0.152 5 |
U2Fusion | 5.593 8 | 2.946 9 | 0.572 9 | 1.291 1 | 5.662 4 | 0.271 4 |
IA-Fusion | 6.561 2 | 4.574 5 | 0.979 2 | 1.912 9 | 14.216 1 | 0.634 6 |
Tab. 2 Analysis of metric means on overall MSRS test dataset
模型 | EN | MI | VIF | SCD | SF | |
---|---|---|---|---|---|---|
CSF | 5.674 2 | 2.538 1 | 0.560 2 | 1.310 2 | 7.012 0 | 0.312 2 |
GTF | 5.472 9 | 2.042 9 | 0.500 1 | 0.803 1 | 9.201 3 | 0.410 2 |
IFCNN | 5.946 2 | 2.388 6 | 0.665 4 | 1.412 3 | 10.988 4 | 0.532 9 |
MDLatRR | 5.976 3 | 2.702 4 | 0.885 9 | 1.362 4 | 8.001 2 | 0.510 2 |
DenseFuse | 5.499 6 | 3.012 1 | 0.612 3 | 1.247 6 | 6.013 8 | 0.301 1 |
PIAFusion | 6.283 9 | 3.623 5 | 0.922 1 | 1.821 3 | 11.194 2 | 0.694 2 |
FusionGAN | 5.401 2 | 2.224 2 | 0.432 1 | 1.012 6 | 4.361 3 | 0.152 5 |
U2Fusion | 5.593 8 | 2.946 9 | 0.572 9 | 1.291 1 | 5.662 4 | 0.271 4 |
IA-Fusion | 6.561 2 | 4.574 5 | 0.979 2 | 1.912 9 | 14.216 1 | 0.634 6 |
模块 | SF | SCD |
---|---|---|
FS-A | 10.521 4 | 1.584 2 |
CM-L1 | 12.544 6 | 1.751 3 |
FS-A+IWA-Net | 12.796 8 | 1.780 2 |
CM-L1+IWA-Net | 13.986 9 | 1.965 1 |
Tab. 3 Objective evaluation metrics for ablation experiments
模块 | SF | SCD |
---|---|---|
FS-A | 10.521 4 | 1.584 2 |
CM-L1 | 12.544 6 | 1.751 3 |
FS-A+IWA-Net | 12.796 8 | 1.780 2 |
CM-L1+IWA-Net | 13.986 9 | 1.965 1 |
1 | MA J, MA Y, LI C. Infrared and visible image fusion methods and applications: a survey [J]. Information Fusion, 2019, 45: 153-178. |
2 | P-H DINH. Combining Gabor energy with equilibrium optimizer algorithm for multi-modality medical image fusion [J]. Biomedical Signal Processing and Control, 2021, 68: 102696. |
3 | LI C, ZHU C, HUANG Y, et al. Cross-modal ranking with soft consistency and noisy labels for robust RGB-T tracking [C]// Proceedings of the 15th European Conference on Computer Vision. Cham: Springer, 2018: 831-847. |
4 | LU Y, WU Y, LIU B, et al. Cross-modality person re-identification with shared-specific feature transfer [C]// Proceedings of the 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2020: 13376-13386. |
5 | HA Q, WATANABE K, KARASAWA T, et al. MFNet: towards real-time semantic segmentation for autonomous vehicles with multi-spectral scenes [C]// Proceedings of the 2017 IEEE/RSJ International Conference on Intelligent Robots and Systems. Piscataway: IEEE, 2017: 5108-5115. |
6 | 朱浩然,刘云清,张文颖.基于对比度增强与多尺度边缘保持分解的红外与可见光图像融合[J].电子与信息学报, 2018, 40(6): 1294-1300. |
ZHU H R, LIU Y Q, ZHANG W Y. Infrared and visible image fusion based on contrast enhancement and multi-scale edge-preserving decomposition [J]. Journal of Electronics & Information Technology, 2018, 40(6): 1294-1300. | |
7 | ZHANG Q, LIU Y, BLUM R S, et al. Sparse representation based multi-sensor image fusion for multi-focus and multi-modality images: A review [J]. Information Fusion, 2018, 40: 57-75. |
8 | LI S, KANG X, FANG L, et al. Pixel-level image fusion: a survey of the state of the art [J]. Information Fusion, 2017, 33: 100-112. |
9 | PRABHAKAR K R, SRIKAR V S, BABU R V. DeepFuse: a deep unsupervised approach for exposure fusion with extreme exposure image pairs [C]// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway: IEEE, 2017: 4724-4732. |
10 | 陈永,张娇娇,王镇.多尺度密集连接注意力的红外与可见光图像融合[J].光学精密工程, 2022, 30(18): 2253-2266. |
CHEN Y, ZHANG J J, WANG Z. Infrared and visible image fusion based on multi-scale dense attention connection network [J]. Optics and Precision Engineering, 2022, 30(18): 2253-2266. | |
11 | LI H, WU X-J, DURRANI T. NestFuse: an infrared and visible image fusion architecture based on nest connection and spatial/channel attention models [J]. IEEE Transactions on Instrumentation and Measurement, 2020, 69(12): 9645-9656. |
12 | LI H, WU X-J, KITTLER J. RFN-Nest: an end-to-end residual fusion network for infrared and visible images [J]. Information Fusion, 2021, 73: 72-86. |
13 | ZHANG H, XU H, XIAO Y, et al. Rethinking the image fusion: a fast unified image fusion network based on proportional maintenance of gradient and intensity [J]. Proceedings of the AAAI Conference on Artificial Intelligence, 2020, 34(7): 12797-12804. |
14 | MA J, YU W, LIANG P, et al. FusionGAN: A generative adversarial network for infrared and visible image fusion [J]. Information Fusion, 2019, 48: 11-26. |
15 | HOU R, ZHOU D, NIE R, et al. VIF-Net: an unsupervised framework for infrared and visible image fusion [J]. IEEE Transactions on Computational Imaging, 2020, 6: 640-651. |
16 | LI C, SONG D, TONG R, et al. Illumination-aware faster R-CNN for robust multispectral pedestrian detection [J]. Pattern Recognition, 2019, 85: 161-171. |
17 | GUO X, MENG L, MEI L, et al. Multi-focus image fusion with Siamese self-attention network [J]. IET Image Processing, 2020, 14(7): 1339-1346. |
18 | 王莹,王晶,高岚,等.一种注意力机制优化方法及硬件加速设计[J/OL].电子学报: 1-9[2023-06-19]. . |
WANG Y, WANG J, GAO L, et al. An improved attention mechanism algorithm model and hardware acceleration design method [J/OL]. Acta Electronica Sinica: 1-9[2023-06-19]. . | |
19 | AN W-B, WANG H-M. Infrared and visible image fusion with supervised convolutional neural network [J]. Optik, 2020, 219: 165120. |
20 | XU H, ZHANG H, MA J. Classification saliency-based rule for visible and infrared image fusion [J]. IEEE Transactions on Computational Imaging, 2021, 7: 824-836. |
21 | MA J, CHEN C, LI C, et al. Infrared and visible image fusion via gradient transfer and total variation minimization [J]. Information Fusion, 2016, 31: 100-109. |
22 | TANG L, YUAN J, ZHANG H, et al. PIAFusion: a progressive infrared and visible image fusion network based on illumination aware [J]. Information Fusion, 2022, 83/84: 79-92. |
23 | ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network [J]. Information Fusion, 2020, 54: 99-118. |
24 | LI H, WU X-J, KITTLER J. MDLatLRR: a novel decomposition method for infrared and visible image fusion [J]. IEEE Transactions on Image Processing, 2020, 29: 4733-4746. |
25 | XU H, MA J, JIANG J, et al. U2Fusion: a unified unsupervised image fusion network [J]. IEEE Transactions on Pattern Analysis and Machine Intelligence, 2020, 44(1): 502-518. |
[1] | Zhiqiang ZHAO, Peihong MA, Xinhong HEI. Crowd counting method based on dual attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2886-2892. |
[2] | Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974. |
[3] | Liting LI, Bei HUA, Ruozhou HE, Kuang XU. Multivariate time series prediction model based on decoupled attention mechanism [J]. Journal of Computer Applications, 2024, 44(9): 2732-2738. |
[4] | Ying HUANG, Jiayu YANG, Jiahao JIN, Bangrui WAN. Siamese mixed information fusion algorithm for RGBT tracking [J]. Journal of Computer Applications, 2024, 44(9): 2878-2885. |
[5] | Kaipeng XUE, Tao XU, Chunjie LIAO. Multimodal sentiment analysis network with self-supervision and multi-layer cross attention [J]. Journal of Computer Applications, 2024, 44(8): 2387-2392. |
[6] | Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406. |
[7] | Zhonghua LI, Yunqi BAI, Xuejin WANG, Leilei HUANG, Chujun LIN, Shiyu LIAO. Low illumination face detection based on image enhancement [J]. Journal of Computer Applications, 2024, 44(8): 2588-2594. |
[8] | Shangbin MO, Wenjun WANG, Ling DONG, Shengxiang GAO, Zhengtao YU. Single-channel speech enhancement based on multi-channel information aggregation and collaborative decoding [J]. Journal of Computer Applications, 2024, 44(8): 2611-2617. |
[9] | Wu XIONG, Congjun CAO, Xuefang SONG, Yunlong SHAO, Xusheng WANG. Handwriting identification method based on multi-scale mixed domain attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2225-2232. |
[10] | Huanhuan LI, Tianqiang HUANG, Xuemei DING, Haifeng LUO, Liqing HUANG. Public traffic demand prediction based on multi-scale spatial-temporal graph convolutional network [J]. Journal of Computer Applications, 2024, 44(7): 2065-2072. |
[11] | Dianhui MAO, Xuebo LI, Junling LIU, Denghui ZHANG, Wenjing YAN. Chinese entity and relation extraction model based on parallel heterogeneous graph and sequential attention mechanism [J]. Journal of Computer Applications, 2024, 44(7): 2018-2025. |
[12] | Li LIU, Haijin HOU, Anhong WANG, Tao ZHANG. Generative data hiding algorithm based on multi-scale attention [J]. Journal of Computer Applications, 2024, 44(7): 2102-2109. |
[13] | Song XU, Wenbo ZHANG, Yifan WANG. Lightweight video salient object detection network based on spatiotemporal information [J]. Journal of Computer Applications, 2024, 44(7): 2192-2199. |
[14] | Dahai LI, Zhonghua WANG, Zhendong WANG. Dual-branch low-light image enhancement network combining spatial and frequency domain information [J]. Journal of Computer Applications, 2024, 44(7): 2175-2182. |
[15] | Mei WANG, Xuesong SU, Jia LIU, Ruonan YIN, Shan HUANG. Time series classification method based on multi-scale cross-attention fusion in time-frequency domain [J]. Journal of Computer Applications, 2024, 44(6): 1842-1847. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||