关键词: Swin Transformer, 多尺度特征, 特征融合, 图像描述, 深度可分离卷积

Abstract: Image caption methods based on Transformer use multi-head attention to calculate attention weights over the entire input sequence, lacking hierarchical feature extraction capabilities. Additionally, two-stage image caption methods limit model performance. To address these issues, an image caption method based on Swin Transformer and multi-scale feature fusion (STMSF) was proposed. In the encoder, Agent Attention was used to maintain global context modeling capability while improving computational efficiency. In the decoder, Multi-Scale Cross Attention (MSCA) was introduced, combining cross-attention and depthwise separable convolution, which not only obtained multi-scale features but also better fused multi-modal features. On the MSCOCO dataset, compared to the SCD-Net (Semantic-Conditional Diffusion Networks) method, the BiLingual Evaluation Understudy with 4-grams (BLEU4) and Consensus-based Image Description Evaluation (CIDEr) metrics were improved by 1.1 percentage points and 5.3 percentage points, respectively. Comparison and ablation experiment results show that the proposed single-stage method, STMSF, can effectively improve model performance and generate high-quality image captions.

Key words: Swin Transformer, multi-scale feature, feature fusion, image caption, depthwise separable convolution