关键词: 网格特征, 图注意力网络, 卷积神经网络, 图像描述生成, 全局特征

Abstract: Since existing image caption approaches only focused on the grid spatial location features without considering grid feature interaction and image global features, an image caption method based on Graph ATtention network(GAT) with global context was proposed to generate higher-quality image captions. By building a grid feature interaction graph, a multi-layer convolutional neural network(CNN) was utilized for visual encoding, and the grid features and entire image features of a given image were retrieved. Then, using the graph attention network, which included a global node and many local nodes, the feature extraction problem was changed into a node classification problem, and the global and local features could be completely utilized after updating and optimization. Finally, the Transformer-based decoding module made use of the enhanced visual features to provide image captions. The Microsoft COCO dataset is used for experiments evaluation. The experimental results demonstrate that the image caption method based on graph attention network with global context successfully captures the global and local features of the image and achieves 133.1% CIDEr(Consensus-based Image Description Evaluation). The image caption method is effective in improving the accuracy of image captioning, which allows perform processing tasks such as classification, retrieval, and analysis of images.

Key words: grid feature, graph Graph attention ATttention network(GAT), Convolutional Neural Network (CNN), image captioning, global feature