关键词: 多模态命名实体识别, 因果干预, 互信息, 数据偏差, 模态差距

Abstract: With the continuous advancement of artificial intelligence technology, multimodal named entity recognition has gradually become a research hotspot. This task aims to identify entities with specific meanings from the joint data of text and images. However, current methods still have shortcomings in dealing with the two problems of data bias and modality gap. Data bias can cause harmful biases to mislead the attention module to focus on false correlations in the training data, thereby damaging the generalization ability of the model. The modality gap will hinder the establishment of correct semantic alignment between text and image, affecting the performance of the model. To solve these two problems, this paper proposes a multimodal named entity recognition method under causal intervention (CMNER). This method uses causal intervention theory to use backdoor intervention in the text modality to deal with observable confounding factors, and uses frontdoor causal intervention in the image modality to deal with confounding factors that cannot be directly observed, so as to mitigate the harmful effects of data bias. At the same time, combined with the mutual information correlation theory, the semantic "distance" between text and image is shortened. The experimental results in this paper show that the method proposed in this paper can effectively alleviate data bias and modality gap and improve the performance of multimodal named entity recognition tasks.

Key words: multimodal named entity recognition, causal intervention, mutual information, data bias, modal gap