面向视觉问答的跨模态交叉融合注意网络

doi:10.11772/j.issn.1001-9081.2021030470

《计算机应用》唯一官方网站 ›› 2022, Vol. 42 ›› Issue (3): 854-859.DOI: 10.11772/j.issn.1001-9081.2021030470

• 人工智能 • 上一篇

面向视觉问答的跨模态交叉融合注意网络

王茂(), 彭亚雄, 陆安江

贵州大学大数据与信息工程学院，贵阳 550025

收稿日期:2021-03-29 修回日期:2021-05-23 接受日期:2021-07-07 发布日期:2022-04-09 出版日期:2022-03-10
通讯作者: 王茂
作者简介:彭亚雄（1963—），男，贵州贵阳人，副教授，主要研究方向：数字通信、音视频处理
陆安江（1978—），男，贵州贵阳人，副教授，博士，主要研究方向：嵌入式系统与集成、物联网安全、微传感。
基金资助:
贵州省科技成果转化项目(［2017］4856)

Cross-modal chiastopic-fusion attention network for visual question answering

Mao WANG(), Yaxiong PENG, Anjiang LU

College of Big Data and Information Engineering，Guizhou University，Guiyang Guizhou 550025，China

Received:2021-03-29 Revised:2021-05-23 Accepted:2021-07-07 Online:2022-04-09 Published:2022-03-10
Contact: Mao WANG
About author:PENG Yaxiong， born in 1963， associate professor. His research interests include digital communication， audio and video processing.
LU Anjiang， born in 1978， Ph. D.， associate professor. His research interests include embedded system and integration， Internet of things security， micro-sensing.
Supported by:
Guizhou Province Science and Technology Achievement Transformation Project(［2017］4856)

摘要/Abstract

摘要：

为了提高视觉问答（VQA）模型回答复杂图像问题的准确率，提出了面向视觉问答的跨模态交叉融合注意网络（CCAN）。首先，提出了一种改进的残差通道自注意方法对图像进行注意，根据图像整体信息来寻找重要区域，从而引入一种新的联合注意机制，将单词注意和图像区域注意结合在一起；其次，提出一种“跨模态交叉融合”网络生成多个特征，将两个动态信息流整合到一起，每个模态内产生有效的注意流，其中对联合特征使用逐元素相乘的方法。此外，为了避免计算成本增加，网络之间共享参数。在VQA v1.0数据集上的实验结果表明，该模型的准确率达到67.57%，较MLAN模型提高了2.97个百分点，较CAQT模型提高了1.20个百分点。所提方法有效提高了视觉问答模型的准确率，具有有效性和鲁棒性。

关键词: 视觉问答, 联合注意, 交叉融合, 残差通道, 联合特征

Abstract:

In order to improve the accuracy of Visual Question Answering （VQA） model in answering complex image questions， a Cross-modal Chiastopic-fusion Attention Network （CCAN） for VQA was proposed. Firstly， an improved residual channel self-attention method was proposed to pay attention to the image， and to find important areas according to overall information of the image， thereby introduced a new joint attention mechanism that combined word attention and image area attention； secondly， a “cross-modal chiastopic-fusion” network was proposed to generate multiple features to integrate the two dynamic information flows together， and an effective attention flow was generated in each modal. Among them， element-wise multiplication method was used for joint features. In addition， in order to avoid an increase in computational cost， parameters were shared between networks. Experimental results on VQA v1.0 dataset show that the accuracy of the proposed model reaches 67.57%， which is 2.97 percentage points higher than that of MLAN （Multi-level Attention Network） model， 1.20 percentage points higher than that of CAQT （Co-Attention network with Question Type） model. The proposed method effectively improves the accuracy of visual question answering model. The effectiveness and robustness of the method are verified.

Key words: visual question answering, joint attention, chiastopic-fusion, residual channel, joint feature

中图分类号:

TP391.41

王茂, 彭亚雄, 陆安江. 面向视觉问答的跨模态交叉融合注意网络[J]. 计算机应用, 2022, 42(3): 854-859.

Mao WANG, Yaxiong PENG, Anjiang LU. Cross-modal chiastopic-fusion attention network for visual question answering[J]. Journal of Computer Applications, 2022, 42(3): 854-859.

图/表 8

参考文献 25

1	LU J S， YANG J W， BATRA D， et al. Hierarchical question-image co-attention for visual question answering［C］// Proceedings of the 30th International Conference on Neural Information Processing Systems. Piscataway： IEEE， 2016： 289-297.
2	BEN-YOUNES H， CADENE R， CORD M， et al. MUTAN： multimodal tucker fusion for visual question answering ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017： 2631-2639. 10.1109/iccv.2017.285
3	TENEY D， ANDERSON P， HE X D， et al. Tips and tricks for visual question answering： learnings from the 2017 challenge ［C］// Proceedings of 2018 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018： 4223-4232. 10.1109/cvpr.2018.00444
4	HE K M， ZHANG X Y， REN S Q， et al. Deep residual learning for image recognition ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 770-778. 10.1109/cvpr.2016.90
5	YU Z， YU J， FAN J P， et al. Multi-modal factorized bilinear pooling with co-attention learning for visual question answering ［C］// Proceedings of the 2017 IEEE International Conference on Computer Vision. Piscataway： IEEE， 2017：1839-1848. 10.1109/iccv.2017.202
6	FUKUI A， PARK D H， YANG D， et al. Multimodal compact bilinear pooling for visual question answering and visual grounding ［EB/OL］.［2016-09-24］. . 10.18653/v1/d16-1044
7	赵宏，孔东一.图像特征注意力与自适应注意力融合的图像内容中文描述［J］.计算机应用， 2021， 41（9）： 2496-2503. 10.11772/j.issn.1001-9081.2020111829
	ZHAO H， KONG D Y. Chinese description of image content based on image feature attention and adaptive attention［J］.Journal of Computer Applications， 2021， 41（9）： 2496-2503. 10.11772/j.issn.1001-9081.2020111829
8	陈龙杰，张钰，张玉梅，等.基于多注意力多尺度特征融合的图像描述生成算法［J］.计算机应用，2019，39（2）：354-359. 10.11772/j.issn.1001-9081.2018071464
	CHEN L J， ZHANG Y， ZHANG Y M， et al. Deep multi-attention and multi-scale neural network for image caption ［J］. Journal of Computer Applications， 2019， 39（2）：354-359. 10.11772/j.issn.1001-9081.2018071464
9	CHEN C， HAN D， WANG J， et al. Multimodal encoder-decoder attention networks for visual question answering ［J］. IEEE Access， 2020， 8： 35662-35671. 10.1109/access.2020.2975093
10	YANG Z， HE X， GAO J， et al. Stacked attention networks for image question answering ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 21-29. 10.1109/cvpr.2016.10
11	D-K NGUYEN， OKATANI T. Improved fusion of visual and language representations by dense symmetric co-attention for visual question answering ［C］// Proceedings of the 2018 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2018：6087-6096. 10.1109/cvpr.2018.00637
12	GAO P， JIANG Z K， YOU H X， et al. Dynamic fusion with intra- and inter-modality attention flow for visual question answering［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 6639-6648. 10.1109/cvpr.2019.00680
13	YU Z， YU J， CUI Y H， et al. Deep modular co-attention networks for visual question answering［C］// Proceedings of the 2019 IEEE/CVF Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2019： 6281-6290. 10.1109/cvpr.2019.00644
14	AGRAWAL A， LU J， ANTOL S， et al. VQA： visual question answering ［J］. International Journal of Computer Vision， 2017， 123（1）： 4-31. 10.1007/s11263-016-0966-6
15	NOH H， SEO P H， HAN B. Image question answering using convolutional neural network with dynamic parameter prediction［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 30-38. 10.1109/cvpr.2016.11
16	ILIEVSKI I， YAN S， FENG J. A focused dynamic attention model for visual question answering ［EB/OL］. ［2016-04-06］. .
17	XU H， Ask SAENKO K.， attend and answer： exploring question-guided spatial attention for visual question answering ［C］// Proceedings of 14th European Conference on Computer Vision . Cham： Springer， 2016： 451-466. 10.1007/978-3-319-46478-7_28
18	YANG Z， HE X， GAO J， et al. Stacked attention networks for image question answering ［C］// Proceedings of the 2016 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2016： 21-29. 10.1109/cvpr.2016.10
19	KIM J-H， K-W ON， LIM W， et al. Hadamard product for low-rank bilinear pooling ［EB/OL］. ［2017-03-26］. .
20	NAM H， HA J， KIM J. Dual attention networks for multimodal reasoning and matching ［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 2156-2164. 10.1109/cvpr.2017.232
21	YU D， FU J， MEI T， et al. Multi-level attention networks for visual question answering［C］// Proceedings of the 2017 IEEE Conference on Computer Vision and Pattern Recognition. Piscataway： IEEE， 2017： 4187-4195. 10.1109/cvpr.2017.446
22	YANG C， JIANG M Q， JIANG B， et al. Co-attention network with question type for visual question answering ［J］. IEEE Access， 2019， 7： 40771-40781. 10.1109/access.2019.2908035
23	LIU F， LIU J， LU H Q， et al. Language and visual relations encoding for visual question answering ［C］// Proceedings of the 2019 IEEE International Conference on Image Processing. Piscataway： IEEE， 2019：3307-3311. 10.1109/icip.2019.8803670
24	QIAO Y Y， YU Z， LIU J. VC-VQA： Visual calibration mechanism for visual question answering ［C］// Proceedings of the 2020 IEEE International Conference on Image Processing. Piscataway： IEEE， 2020： 1481-1485. 10.1109/icip40778.2020.9190828
25	YU J， ZHANG W F， LU Y H， et al. Reasoning on the relation： enhancing visual representation for visual question answering and cross-modal retrieval ［J］. IEEE Transactions on Multimedia. 2020， 22（12）：3196-3209. 10.1109/tmm.2020.2972830

模型	Test-dev				Test-std
模型	是/否	数字	其他	Overall	是/否	数字	其他	Overall
LSTM Q+I^［14］	78.94	35.24	36.42	53.74	78.85	35.80	43.41	57.17
DPPNet^［15］	80.71	37.24	41.69	57.22	80.79	38.94	52.16	62.48
FDA^［16］	81.14	36.16	45.77	59.24	81.50	39.00	54.72	64.01
SMem^［17］	80.87	37.32	43.12	57.99	80.80	37.53	43.48	58.24
SAN^［18］	79.30	36.60	46.10	58.70	—	—	—	58.90
MLB^［19］	85.57	39.32	57.36	67.03	84.39	38.70	58.20	66.96
HieCoAtt^［1］	79.70	38.70	51.70	61.80	79.95	38.22	51.95	62.06
DAN^［20］	83.00	39.10	53.90	64.30	82.80	38.10	54.00	64.20
MLAN^［21］	83.80	40.20	53.70	64.60	83.70	40.90	53.70	64.80
CAQT^［22］	82.63	42.02	57.98	66.37	82.88	41.15	58.05	66.53
LVRE^［23］	85.60	40.60	57.50	67.20	—	—	—	67.40
VC-VQA^［24］	85.00	42.60	57.70	67.30	—	—	—	—
v-VRANET^［25］	83.31	45.51	58.41	67.20	83.39	44.96	58.49	67.34
CCAN	84.31	42.47	59.22	67.57	83.29	42.43	59.32	67.54

模型	Test-dev				Test-std
模型	是/否	数字	其他	Overall	是/否	数字	其他	Overall
LSTM Q+I^［14］	78.94	35.24	36.42	53.74	78.85	35.80	43.41	57.17
DPPNet^［15］	80.71	37.24	41.69	57.22	80.79	38.94	52.16	62.48
FDA^［16］	81.14	36.16	45.77	59.24	81.50	39.00	54.72	64.01
SMem^［17］	80.87	37.32	43.12	57.99	80.80	37.53	43.48	58.24
SAN^［18］	79.30	36.60	46.10	58.70	—	—	—	58.90
MLB^［19］	85.57	39.32	57.36	67.03	84.39	38.70	58.20	66.96
HieCoAtt^［1］	79.70	38.70	51.70	61.80	79.95	38.22	51.95	62.06
DAN^［20］	83.00	39.10	53.90	64.30	82.80	38.10	54.00	64.20
MLAN^［21］	83.80	40.20	53.70	64.60	83.70	40.90	53.70	64.80
CAQT^［22］	82.63	42.02	57.98	66.37	82.88	41.15	58.05	66.53
LVRE^［23］	85.60	40.60	57.50	67.20	—	—	—	67.40
VC-VQA^［24］	85.00	42.60	57.70	67.30	—	—	—	—
v-VRANET^［25］	83.31	45.51	58.41	67.20	83.39	44.96	58.49	67.34
CCAN	84.31	42.47	59.22	67.57	83.29	42.43	59.32	67.54

模型	准确率	模型	准确率
模型一	59.95	模型四	61.92
模型二	60.70	模型五	62.07
模型三	61.54

模型	准确率	模型	准确率
模型一	59.95	模型四	61.92
模型二	60.70	模型五	62.07
模型三	61.54

损失函数	准确率	损失函数	准确率
KLD	62.06	交叉熵	61.09
BCE	61.24

面向视觉问答的跨模态交叉融合注意网络

Cross-modal chiastopic-fusion attention network for visual question answering

RichHTML

PDF

可视化

摘要/Abstract

引用本文

使用本文

图/表 8

参考文献 25

相关文章 2

编辑推荐

Metrics

[1]	陈力, 丁世飞, 于文家. 基于跨通道交叉融合和跨模块连接的轻量级卷积神经网络[J]. 计算机应用, 2020, 40(12): 3451-3457.
[2]	张力, 张洞明, 郑宏. 基于联合层特征的卷积神经网络在车标识别中的应用[J]. 计算机应用, 2016, 36(2): 444-448.