Journal of Computer Applications ›› 2025, Vol. 45 ›› Issue (1): 275-283.DOI: 10.11772/j.issn.1001-9081.2024010026
• Multimedia computing and computer simulation • Previous Articles Next Articles
Zongsheng ZHENG1, Jia DU1(), Yuhe CHENG1, Zecheng ZHAO1, Yuewei ZHANG2, Xulong WANG3
Received:
2024-01-15
Revised:
2024-03-26
Accepted:
2024-04-01
Online:
2024-05-09
Published:
2025-01-10
Contact:
Jia DU
About author:
ZHENG Zongsheng, born in 1979, Ph. D., associate professor. His research interests include deep learning, remote sensing image processing.Supported by:
郑宗生1, 杜嘉1(), 成雨荷1, 赵泽骋1, 张月维2, 王绪龙3
通讯作者:
杜嘉
作者简介:
郑宗生(1979—),男,河北唐山人,副教授,博士,主要研究方向:深度学习、遥感图像处理;基金资助:
CLC Number:
Zongsheng ZHENG, Jia DU, Yuhe CHENG, Zecheng ZHAO, Yuewei ZHANG, Xulong WANG. Cross-modal dual-stream alternating interactive network for infrared-visible image classification[J]. Journal of Computer Applications, 2025, 45(1): 275-283.
郑宗生, 杜嘉, 成雨荷, 赵泽骋, 张月维, 王绪龙. 用于红外-可见光图像分类的跨模态双流交替交互网络[J]. 《计算机应用》唯一官方网站, 2025, 45(1): 275-283.
Add to citation manager EndNote|Ris|BibTeX
URL: https://www.joca.cn/EN/10.11772/j.issn.1001-9081.2024010026
卷积层 | 卷积核大小 | 通道数 | 激活函数 | 池化操作 | 池化大小 |
---|---|---|---|---|---|
Conv1 | 5×5 | 16 | ReLU | MaxPooling | 2×2 |
Conv2 | 3×3 | 32 | ReLU | MaxPooling | 2×2 |
Conv3 | 3×3 | 64 | ReLU | MaxPooling | 2×2 |
Conv4 | 3×3 | 128 | ReLU | MaxPooling | 2×2 |
Tab. 1 Parameters of convolutional network architecture
卷积层 | 卷积核大小 | 通道数 | 激活函数 | 池化操作 | 池化大小 |
---|---|---|---|---|---|
Conv1 | 5×5 | 16 | ReLU | MaxPooling | 2×2 |
Conv2 | 3×3 | 32 | ReLU | MaxPooling | 2×2 |
Conv3 | 3×3 | 64 | ReLU | MaxPooling | 2×2 |
Conv4 | 3×3 | 128 | ReLU | MaxPooling | 2×2 |
编号 | 台风等级 | 风速/(m·s-1) | 训练集样本数 | 测试集样本数 |
---|---|---|---|---|
合计 | 4 906 | 548 | ||
1 | 热带低压 | <17 | 1 114 | 124 |
2 | 热带风暴 | ≥17~<25 | 1 015 | 114 |
3 | 强热带风暴 | ≥25~<33 | 1 024 | 114 |
4 | 台风 | ≥33~<42 | 1 035 | 116 |
5 | 强台风 | ≥42 | 718 | 80 |
Tab. 2 Numbers of training set samples and test set samples for each category in self-built multi-modal typhoon datasets
编号 | 台风等级 | 风速/(m·s-1) | 训练集样本数 | 测试集样本数 |
---|---|---|---|---|
合计 | 4 906 | 548 | ||
1 | 热带低压 | <17 | 1 114 | 124 |
2 | 热带风暴 | ≥17~<25 | 1 015 | 114 |
3 | 强热带风暴 | ≥25~<33 | 1 024 | 114 |
4 | 台风 | ≥33~<42 | 1 035 | 116 |
5 | 强台风 | ≥42 | 718 | 80 |
编号 | 类别 | 训练样本数 | 测试样本数 |
---|---|---|---|
合计 | 378 | 99 | |
1 | 乡村(country) | 41 | 11 |
2 | 田地(field) | 40 | 11 |
3 | 森林(forest) | 42 | 11 |
4 | 室内(indoor) | 45 | 11 |
5 | 山脉(mountain) | 44 | 11 |
6 | 旧建筑(old building) | 40 | 11 |
7 | 街道(street) | 39 | 11 |
8 | 都市(urban) | 47 | 11 |
9 | 海域(water) | 40 | 11 |
Tab. 3 Numbers of training set samples and test set samples for each category in RGB-NIR dataset
编号 | 类别 | 训练样本数 | 测试样本数 |
---|---|---|---|
合计 | 378 | 99 | |
1 | 乡村(country) | 41 | 11 |
2 | 田地(field) | 40 | 11 |
3 | 森林(forest) | 42 | 11 |
4 | 室内(indoor) | 45 | 11 |
5 | 山脉(mountain) | 44 | 11 |
6 | 旧建筑(old building) | 40 | 11 |
7 | 街道(street) | 39 | 11 |
8 | 都市(urban) | 47 | 11 |
9 | 海域(water) | 40 | 11 |
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
单个红外模态 | 0.798 4 | 0.657 9 | 0.649 1 | 0.758 6 | 0.612 5 | 0.703 5 | 0.789 7 |
简单级联结构 | 0.903 2 | 0.614 0 | 0.429 8 | 0.681 0 | 0.737 5 | 0.673 1 | 0.755 2 |
DAE+简单级联 | 0.830 6 | 0.596 5 | 0.605 3 | 0.793 1 | 0.787 5 | 0.721 4 | 0.798 9 |
DAE+CMFI | 0.790 3 | 0.675 4 | 0.684 2 | 0.744 1 | 0.787 5 | 0.739 2 | 0.817 6 |
Tab. 4 Test results with infrared data in test stage on self-built infrared-visible typhoon multi-modal dataset
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
单个红外模态 | 0.798 4 | 0.657 9 | 0.649 1 | 0.758 6 | 0.612 5 | 0.703 5 | 0.789 7 |
简单级联结构 | 0.903 2 | 0.614 0 | 0.429 8 | 0.681 0 | 0.737 5 | 0.673 1 | 0.755 2 |
DAE+简单级联 | 0.830 6 | 0.596 5 | 0.605 3 | 0.793 1 | 0.787 5 | 0.721 4 | 0.798 9 |
DAE+CMFI | 0.790 3 | 0.675 4 | 0.684 2 | 0.744 1 | 0.787 5 | 0.739 2 | 0.817 6 |
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
单个可见光模态 | 0.830 6 | 0.578 9 | 0.605 3 | 0.612 1 | 0.687 5 | 0.660 7 | 0.759 9 |
简单级联结构 | 0.879 0 | 0.473 7 | 0.394 7 | 0.620 7 | 0.612 5 | 0.603 5 | 0.717 7 |
DAE+简单级联 | 0.790 3 | 0.570 2 | 0.482 5 | 0.637 9 | 0.625 0 | 0.626 7 | 0.721 8 |
DAE+CMFI | 0.806 5 | 0.596 5 | 0.552 6 | 0.646 6 | 0.612 5 | 0.642 8 | 0.742 5 |
Tab. 5 Test results with visible light data in test stage on self-built infrared-visible typhoon multi-modal dataset
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
单个可见光模态 | 0.830 6 | 0.578 9 | 0.605 3 | 0.612 1 | 0.687 5 | 0.660 7 | 0.759 9 |
简单级联结构 | 0.879 0 | 0.473 7 | 0.394 7 | 0.620 7 | 0.612 5 | 0.603 5 | 0.717 7 |
DAE+简单级联 | 0.790 3 | 0.570 2 | 0.482 5 | 0.637 9 | 0.625 0 | 0.626 7 | 0.721 8 |
DAE+CMFI | 0.806 5 | 0.596 5 | 0.552 6 | 0.646 6 | 0.612 5 | 0.642 8 | 0.742 5 |
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
IFCNN | 0.629 0 | 0.622 8 | 0.596 5 | 0.706 9 | 0.637 5 | 0.641 0 | 0.746 1 |
DenseFuse | 0.677 4 | 0.710 5 | 0.517 5 | 0.715 5 | 0.812 5 | 0.679 0 | 0.779 0 |
DAINet | 0.790 3 | 0.675 4 | 0.684 2 | 0.744 1 | 0.787 5 | 0.739 2 | 0.817 6 |
Tab. 6 Results comparison of proposed method, IFCNN and DenseFuse methods with infrared data in test stage on self-built infrared-visible typhoon multi-modal dataset
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
IFCNN | 0.629 0 | 0.622 8 | 0.596 5 | 0.706 9 | 0.637 5 | 0.641 0 | 0.746 1 |
DenseFuse | 0.677 4 | 0.710 5 | 0.517 5 | 0.715 5 | 0.812 5 | 0.679 0 | 0.779 0 |
DAINet | 0.790 3 | 0.675 4 | 0.684 2 | 0.744 1 | 0.787 5 | 0.739 2 | 0.817 6 |
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
IFCNN | 0.411 3 | 0.535 1 | 0.535 1 | 0.387 9 | 0.487 5 | 0.469 0 | 0.627 5 |
DenseFuse | 0.596 8 | 0.622 8 | 0.561 4 | 0.629 3 | 0.762 5 | 0.626 0 | 0.713 0 |
DAINet | 0.806 5 | 0.596 5 | 0.552 6 | 0.646 6 | 0.612 5 | 0.642 8 | 0.742 5 |
Tab. 7 Results comparison of proposed method, IFCNN and DenseFuse methods with visible light data in test stage on self-built infrared-visible typhoon multi-modal dataset
方法 | 类别准确率 | 总体准确率 | G-mean | ||||
---|---|---|---|---|---|---|---|
热带低压 | 热带风暴 | 强热带风暴 | 台风 | 强台风 | |||
IFCNN | 0.411 3 | 0.535 1 | 0.535 1 | 0.387 9 | 0.487 5 | 0.469 0 | 0.627 5 |
DenseFuse | 0.596 8 | 0.622 8 | 0.561 4 | 0.629 3 | 0.762 5 | 0.626 0 | 0.713 0 |
DAINet | 0.806 5 | 0.596 5 | 0.552 6 | 0.646 6 | 0.612 5 | 0.642 8 | 0.742 5 |
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
单个红外模态 | 0.454 5 | 0.727 3 | 0.454 5 | 0.454 5 | 0.090 9 | 0.636 4 | 0.909 1 | 0.636 4 | 0.636 4 | 0.568 4 |
简单级联结构 | 0.181 8 | 0.454 5 | 0.636 4 | 0.545 5 | 0.363 6 | 0.363 6 | 0.909 1 | 0.363 6 | 0.727 3 | 0.505 1 |
DAE+简单级联 | 0.090 9 | 0.636 4 | 0.545 5 | 0.818 2 | 0.181 8 | 0.454 5 | 1.000 0 | 0.000 0 | 0.909 1 | 0.532 7 |
DAE +CMFI | 0.363 6 | 0.818 2 | 0.909 1 | 0.727 3 | 0.272 7 | 0.454 5 | 0.909 1 | 0.636 4 | 0.636 4 | 0.639 8 |
Tab. 8 Test results with infrared data in test stage on RGB-NIR dataset
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
单个红外模态 | 0.454 5 | 0.727 3 | 0.454 5 | 0.454 5 | 0.090 9 | 0.636 4 | 0.909 1 | 0.636 4 | 0.636 4 | 0.568 4 |
简单级联结构 | 0.181 8 | 0.454 5 | 0.636 4 | 0.545 5 | 0.363 6 | 0.363 6 | 0.909 1 | 0.363 6 | 0.727 3 | 0.505 1 |
DAE+简单级联 | 0.090 9 | 0.636 4 | 0.545 5 | 0.818 2 | 0.181 8 | 0.454 5 | 1.000 0 | 0.000 0 | 0.909 1 | 0.532 7 |
DAE +CMFI | 0.363 6 | 0.818 2 | 0.909 1 | 0.727 3 | 0.272 7 | 0.454 5 | 0.909 1 | 0.636 4 | 0.636 4 | 0.639 8 |
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
单个可见光模态 | 0.363 6 | 0.727 3 | 0.727 3 | 0.545 5 | 1.000 0 | 0.363 6 | 0.818 2 | 0.636 4 | 0.727 3 | 0.657 7 |
简单级联结构 | 0.363 6 | 0.363 6 | 0.727 3 | 0.181 8 | 0.727 3 | 0.363 6 | 0.818 2 | 0.272 7 | 0.636 4 | 0.494 9 |
DAE +简单级联 | 0.272 7 | 0.636 4 | 0.363 6 | 0.909 1 | 0.454 5 | 0.545 5 | 0.818 2 | 0.000 0 | 0.818 2 | 0.550 5 |
DAE +CMFI | 0.454 5 | 0.545 5 | 0.818 2 | 0.636 4 | 0.272 7 | 0.636 4 | 0.818 2 | 0.454 5 | 0.636 4 | 0.633 9 |
Tab. 9 Test results with visible light data in test stage on RGB-NIR dataset
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
单个可见光模态 | 0.363 6 | 0.727 3 | 0.727 3 | 0.545 5 | 1.000 0 | 0.363 6 | 0.818 2 | 0.636 4 | 0.727 3 | 0.657 7 |
简单级联结构 | 0.363 6 | 0.363 6 | 0.727 3 | 0.181 8 | 0.727 3 | 0.363 6 | 0.818 2 | 0.272 7 | 0.636 4 | 0.494 9 |
DAE +简单级联 | 0.272 7 | 0.636 4 | 0.363 6 | 0.909 1 | 0.454 5 | 0.545 5 | 0.818 2 | 0.000 0 | 0.818 2 | 0.550 5 |
DAE +CMFI | 0.454 5 | 0.545 5 | 0.818 2 | 0.636 4 | 0.272 7 | 0.636 4 | 0.818 2 | 0.454 5 | 0.636 4 | 0.633 9 |
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
IFCNN | 0.000 0 | 0.454 5 | 0.909 1 | 0.818 2 | 0.545 5 | 0.090 9 | 0.545 5 | 0.272 7 | 0.181 8 | 0.424 2 |
DenseFuse | 0.000 0 | 0.545 5 | 0.909 1 | 0.909 1 | 0.454 5 | 0.090 9 | 0.818 2 | 0.636 4 | 0.545 5 | 0.545 5 |
DAINet | 0.363 6 | 0.818 2 | 0.909 1 | 0.727 3 | 0.272 7 | 0.454 5 | 0.909 1 | 0.636 4 | 0.636 4 | 0.639 8 |
Tab. 10 Results comparison of proposed method, IFCNN and DenseFuse methods with infrared data in test stage on RGB-NIR dataset
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
IFCNN | 0.000 0 | 0.454 5 | 0.909 1 | 0.818 2 | 0.545 5 | 0.090 9 | 0.545 5 | 0.272 7 | 0.181 8 | 0.424 2 |
DenseFuse | 0.000 0 | 0.545 5 | 0.909 1 | 0.909 1 | 0.454 5 | 0.090 9 | 0.818 2 | 0.636 4 | 0.545 5 | 0.545 5 |
DAINet | 0.363 6 | 0.818 2 | 0.909 1 | 0.727 3 | 0.272 7 | 0.454 5 | 0.909 1 | 0.636 4 | 0.636 4 | 0.639 8 |
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
IFCNN | 0.000 0 | 0.727 3 | 0.636 4 | 0.545 5 | 0.727 3 | 0.363 6 | 0.818 2 | 0.454 5 | 0.727 3 | 0.555 6 |
DenseFuse | 0.454 5 | 0.636 4 | 0.454 5 | 0.454 5 | 0.636 4 | 0.363 6 | 0.818 2 | 0.636 4 | 0.727 3 | 0.575 8 |
DAINet | 0.454 5 | 0.545 5 | 0.818 2 | 0.636 4 | 0.272 7 | 0.636 4 | 0.818 2 | 0.454 5 | 0.454 5 | 0.633 9 |
Tab. 11 Results comparison of proposed method, IFCNN and DenseFuse methods with visible light data in test stage on RGB-NIR dataset
方法 | 类别准确率 | 总体准确率 | ||||||||
---|---|---|---|---|---|---|---|---|---|---|
乡村 | 田地 | 森林 | 室内 | 山脉 | 旧建筑 | 街道 | 都市 | 海域 | ||
IFCNN | 0.000 0 | 0.727 3 | 0.636 4 | 0.545 5 | 0.727 3 | 0.363 6 | 0.818 2 | 0.454 5 | 0.727 3 | 0.555 6 |
DenseFuse | 0.454 5 | 0.636 4 | 0.454 5 | 0.454 5 | 0.636 4 | 0.363 6 | 0.818 2 | 0.636 4 | 0.727 3 | 0.575 8 |
DAINet | 0.454 5 | 0.545 5 | 0.818 2 | 0.636 4 | 0.272 7 | 0.636 4 | 0.818 2 | 0.454 5 | 0.454 5 | 0.633 9 |
1 | 南轲,齐华,叶沅鑫.深度卷积特征表达的多模态遥感影像模板匹配方法[J].测绘学报, 2019, 48(6): 727-736. |
NAN K, QI H, YE Y X. A template matching method of multimodal remote sensing images based on deep convolutional feature representation [J]. Acta Geodaetica et Cartographica Sinica, 2019, 48(6): 727-736. | |
2 | SUN X, TIAN Y, LU W, et al. From single-to multi-modal remote sensing imagery interpretation: a survey and taxonomy [J]. SCIENCE CHINA Information Sciences, 2023, 66(4): No.140301. |
3 | 王佩瑾,闫志远,容雪娥,等.数据受限条件下的多模态处理技术综述[J].中国图象图形学报, 2022, 27(10): 2803-2834. |
WANG P J, YAN Z Y, RONG X E, et al. Review of multimodal data processing techniques with limited data [J]. Journal of Image and Graphics, 2022, 27(10): 2803-2834. | |
4 | HANG R, LI Z, GHAMISI P, et al. Classification of hyperspectral and LiDAR data using coupled CNNs [J]. IEEE Transactions on Geoscience and Remote Sensing, 2020, 58(7): 4939-4950. |
5 | MORCHHALE S, PAUCA V P, PLEMMONS R J, et al. Classification of pixel-level fused hyperspectral and LiDAR data using deep convolutional neural networks [C]// Proceedings of the 8th Workshop on Hyperspectral Image and Signal Processing: Evolution in Remote Sensing. Piscataway: IEEE, 2016: 1-5. |
6 | CHEN J, ZHANG A. HGMF: heterogeneous graph-based fusion for multimodal data with incompleteness [C]// Proceedings of the 26th ACM SIGKDD International Conference on Knowledge Discovery and Data Mining. New York: ACM, 2020: 1295-1305. |
7 | DELDARI S, XUE H, SAEED A, et al. COCOA: cross modality contrastive learning for sensor data [J]. Proceedings of the ACM on Interactive, Mobile, Wearable and Ubiquitous Technologies, 2022, 6(3): No.108. |
8 | HONG D, GAO L, YOKOYA N, et al. More diverse means better: multimodal deep learning meets remote-sensing imagery classification [J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(5): 4340-4354. |
9 | MAI S, ZENG Y, HU H. Multimodal information bottleneck: learning minimal sufficient unimodal and multimodal representations [J]. IEEE Transactions on Multimedia, 2023, 25: 4121-4134. |
10 | WANG Q, CHI Y, SHEN T, et al. Improving RGB-infrared object detection by reducing cross-modality redundancy [J]. Remote Sensing, 2022, 14(9): No.2020. |
11 | CUI S, CAO J, CONG X, et al. Enhancing multimodal entity and relation extraction with variational information bottleneck [J]. IEEE/ACM Transactions on Audio, Speech, and Language Processing, 2024, 32: 1274-1285. |
12 | 汪超,唐超,王文剑,等.基于多模态注意力网络的红外人体行为识别方法[J].计算机科学, 2024, 51(8): 232-241. |
WANG C, TANG C, WANG W J, et al. Infrared human action recognition method based on multimodal attention network [J]. Computer Science, 2024, 51(8): 232-241. | |
13 | CHEN H, LI Y. Three-stream attention-aware network for RGB-D salient object detection [J]. IEEE Transactions on Image Processing, 2019, 28(6): 2825-2835. |
14 | SUN X, ZHANG L, YANG H, et al. Enhancement of spectral resolution for remotely sensed multispectral image [J]. IEEE Journal of Selected Topics in Applied Earth Observations and Remote Sensing, 2015, 8(5): 2198-2211. |
15 | MALEC S, ROGGE D, HEIDEN U, et al. Capability of spaceborne hyperspectral EnMAP mission for mapping fractional cover for soil erosion modeling [J]. Remote Sensing, 2015, 7(9): 11776-11800. |
16 | YOKOYA N, YAIRI T, IWASAKI A. Coupled nonnegative matrix factorization unmixing for hyperspectral and multispectral data fusion [J]. IEEE Transactions on Geoscience and Remote Sensing, 2012, 50(2): 528-537. |
17 | MOU L, ZHU X X. RiFCN: recurrent network in fully convolutional network for semantic segmentation of high resolution remote sensing images [EB/OL]. [2023-11-12]. . |
18 | HONG D, YOKOYA N, CHANUSSOT J, et al. CoSpace: common subspace learning from hyperspectral-multispectral correspondences [J]. IEEE Transactions on Geoscience and Remote Sensing, 2019, 57(7): 4349-4359. |
19 | HONG D, YOKOYA N, GE N, et al. Learnable Manifold Alignment (LeMA): a semi-supervised cross-modality learning framework for land cover and land use classification [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2019, 147: 193-205. |
20 | LI G, LIU Z, CHEN M, et al. Hierarchical alternate interaction network for RGB-D salient object detection [J]. IEEE Transactions on Image Processing, 2021, 30: 3528-3542. |
21 | CHEN M, XING L, WANG Y, et al. Enhanced multimodal representation learning with cross-modal KD [C]// Proceedings of the 2023 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2023: 11766-11775. |
22 | HONG D, YOKOYA N, XIA G S, et al. X-ModalNet: a semi-supervised deep cross-modal network for classification of remote sensing data [J]. ISPRS Journal of Photogrammetry and Remote Sensing, 2020, 167: 12-23. |
23 | HONG D, YAO J, MENG D, et al. Multimodal GANs: toward crossmodal hyperspectral-multispectral image segmentation [J]. IEEE Transactions on Geoscience and Remote Sensing, 2021, 59(6): 5103-5113. |
24 | HUANG J, HUANG X, YANG J. Residual enhanced multi-hypergraph neural network [C]// Proceedings of the 2021 IEEE International Conference on Image Processing. Piscataway: IEEE, 2021: 3657-3661. |
25 | XU N, MAO W. A residual merged neutral network for multimodal sentiment analysis [C]// Proceedings of IEEE 2nd International Conference on Big Data Analysis. Piscataway: IEEE, 2017: 6-10. |
26 | SAXE A M, BANSAL Y, DAPELLO J, et al. On the information bottleneck theory of deep learning [J]. Journal of Statistical Mechanics: Theory and Experiment, 2019, 2019(12): No.124020. |
27 | WAN B, LV C, ZHOU X, et al. TMNet: triple-modal interaction encoder and multi-scale fusion decoder network for V-D-T salient object detection [J]. Pattern Recognition, 2024, 147: No.110074. |
28 | CHENG D, LI X, QI M, et al. Exploring cross-modality commonalities via dual-stream multi-branch network for infrared-visible person re-identification [J]. IEEE Access, 2020, 8: 12824-12834. |
29 | LI J, LIU J, ZHOU S, et al. Infrared and visible image fusion based on residual dense network and gradient loss [J]. Infrared Physics and Technology, 2023, 128: No.104486. |
30 | 武凌霄,康家银,姬云翔,等.基于多判别器双流生成对抗网络的红外与可见光图像融合[J].兵工学报, 2024, 45(6): 1799-1812. |
WU L X, KANG J Y, JI Y X, et al. Infrared and visible image fusion using dual-stream generative adversarial network with multiple discriminators [J]. Acta Armamentarii, 2024, 45(6): 1799-1812. | |
31 | 贾永乐,周李涌,刘月峰,等.基于改进双流ResNet网络的人体行为识别算法研究[J].内蒙古科技大学学报, 2023, 42(2): 145-148. |
JIA Y L, ZHOU L Y, LIU Y F, et al. Research on human behavior recognition algorithm based on improved dual-current ResNet network [J]. Journal of Inner Mongolia University of Science and Technology, 2023, 42(2): 145-148. | |
32 | 杨叶君,刘刚,肖刚,等.基于自适应特征增强和生成器路径交互的红外与可见光图像融合[J].激光与光电子学进展, 2023, 60(14): No.1410018. |
YANG Y J, LIU G, XIAO G, et al. Infrared and visible image fusion based on adaptive feature enhancement and generator path interaction [J]. Laser and Optoelectronics Progress, 2023, 60(14): No.1410018. | |
33 | 吴波,张荣芬,刘宇红.改进ViT的RGB-T多模态交互跟踪算法研究[J/OL].计算机工程与应用[2024-03-19]. . |
WU B, ZHANG R F, LIU Y H. Research on RGB-T multimodal interaction tracking algorithm with improved ViT [J/OL]. Computer Engineering and Applications [2024-03-19]. . | |
34 | 朱伟杰,陈莹.双流网络信息交互机制下的微表情识别[J].计算机辅助设计与图形学学报, 2021, 33(4): 545-552. |
ZHU W J, CHEN Y. Micro-expression recognition based on dual-stream networks information interaction [J]. Journal of Computer-Aided Design and Computer Graphics, 2021, 33(4): 545-552. | |
35 | ZHU X X, TUIA D, MOU L, et al. Deep learning in remote sensing: a comprehensive review and list of resources [J]. IEEE Geoscience and Remote Sensing Magazine, 2017, 5(4): 8-36. |
36 | ZHANG Y, LIU Y, SUN P, et al. IFCNN: a general image fusion framework based on convolutional neural network [J]. Information Fusion, 2020, 54: 99-118. |
37 | LI H, WU X J. DenseFuse: a fusion approach to infrared and visible images [J]. IEEE Transactions on Image Processing, 2018, 28(5): 2614-2623. |
38 | BROWN M, SÜSSTRUNK S. Multi-spectral SIFT for scene category recognition [C]// Proceedings of the 2011 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway: IEEE, 2011: 177-184. |
[1] | Xinran XU, Shaobing ZHANG, Miao CHENG, Yang ZHANG, Shang ZENG. Bearings fault diagnosis method based on multi-pathed hierarchical mixture-of-experts model [J]. Journal of Computer Applications, 2025, 45(1): 59-68. |
[2] | Jietao LIANG, Bing LUO, Lanhui FU, Qingling CHANG, Nannan LI, Ningbo YI, Qi FENG, Xin HE, Fuqin DENG. Point cloud registration method based on coordinate geometric sampling [J]. Journal of Computer Applications, 2025, 45(1): 214-222. |
[3] | Yan YAN, Xingying QIAN, Pengbin YAN, Jie YANG. Federated learning-based statistical prediction and differential privacy protection method for location big data [J]. Journal of Computer Applications, 2025, 45(1): 127-135. |
[4] | Siqi ZHANG, Jinjun ZHANG, Tianyi WANG, Xiaolin QIN. Deep temporal event detection algorithm based on signal temporal logic [J]. Journal of Computer Applications, 2025, 45(1): 90-97. |
[5] | Shunyong LI, Shiyi LI, Rui XU, Xingwang ZHAO. Incomplete multi-view clustering algorithm based on self-attention fusion [J]. Journal of Computer Applications, 2024, 44(9): 2696-2703. |
[6] | Yexin PAN, Zhe YANG. Optimization model for small object detection based on multi-level feature bidirectional fusion [J]. Journal of Computer Applications, 2024, 44(9): 2871-2877. |
[7] | Jing QIN, Zhiguang QIN, Fali LI, Yueheng PENG. Diagnosis of major depressive disorder based on probabilistic sparse self-attention neural network [J]. Journal of Computer Applications, 2024, 44(9): 2970-2974. |
[8] | Xiyuan WANG, Zhancheng ZHANG, Shaokang XU, Baocheng ZHANG, Xiaoqing LUO, Fuyuan HU. Unsupervised cross-domain transfer network for 3D/2D registration in surgical navigation [J]. Journal of Computer Applications, 2024, 44(9): 2911-2918. |
[9] | Yunchuan HUANG, Yongquan JIANG, Juntao HUANG, Yan YANG. Molecular toxicity prediction based on meta graph isomorphism network [J]. Journal of Computer Applications, 2024, 44(9): 2964-2969. |
[10] | Pengqi GAO, Heming HUANG, Yonghong FAN. Fusion of coordinate and multi-head attention mechanisms for interactive speech emotion recognition [J]. Journal of Computer Applications, 2024, 44(8): 2400-2406. |
[11] | Yuhan LIU, Genlin JI, Hongping ZHANG. Video pedestrian anomaly detection method based on skeleton graph and mixed attention [J]. Journal of Computer Applications, 2024, 44(8): 2551-2557. |
[12] | Yanjie GU, Yingjun ZHANG, Xiaoqian LIU, Wei ZHOU, Wei SUN. Traffic flow forecasting via spatial-temporal multi-graph fusion [J]. Journal of Computer Applications, 2024, 44(8): 2618-2625. |
[13] | Qianhong SHI, Yan YANG, Yongquan JIANG, Xiaocao OUYANG, Wubo FAN, Qiang CHEN, Tao JIANG, Yuan LI. Multi-granularity abrupt change fitting network for air quality prediction [J]. Journal of Computer Applications, 2024, 44(8): 2643-2650. |
[14] | Zheng WU, Zhiyou CHENG, Zhentian WANG, Chuanjian WANG, Sheng WANG, Hui XU. Deep learning-based classification of head movement amplitude during patient anaesthesia resuscitation [J]. Journal of Computer Applications, 2024, 44(7): 2258-2263. |
[15] | Dongwei WANG, Baichen LIU, Zhi HAN, Yanmei WANG, Yandong TANG. Deep network compression method based on low-rank decomposition and vector quantization [J]. Journal of Computer Applications, 2024, 44(7): 1987-1994. |
Viewed | ||||||
Full text |
|
|||||
Abstract |
|
|||||