Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Multimodal named entity recognition under causal intervention
Jiana MENG, Chenhao BAI, Di ZHAO, Bolin WANG, Linlin GAO
Journal of Computer Applications    2025, 45 (12): 3796-3803.   DOI: 10.11772/j.issn.1001-9081.2024111681
Abstract40)   HTML1)    PDF (902KB)(10)       Save

Multimodal Named Entity Recognition (MNER) task aims to recognize entities with specific meanings from the joint data of text and images. However, current methods have shortcomings in dealing with the two problems of data bias and modality gap. The data bias can cause harmful biases to mislead the attention module to focus on false correlations in the training data, thereby damaging generalization ability of the model; and the modality gap will hinder the establishment of correct semantic alignment between text and image, thereby affecting performance of the model. A method of Multimodal Named Entity Recognition under Causal intervention (CMNER) was proposed to solve these two problems. In the method, causal intervention theory was utilized to use backdoor intervention in the text modality to deal with observable confounding factors, and use frontdoor causal intervention in the image modality to deal with confounding factors that cannot be observed directly, so as to mitigate the harmful effects of data bias. At the same time, the Mutual Information (MI) correlation theory was combined to shorten the semantic “distance” between text and image. The entity recognition performance of the proposed method was verified in the multimodal domain. Experimental results on the Twitter-2015 and Twitter-2017 datasets show that CMNER method has the F1-scores reached 76.00% and 88.60%, respectively. Compared with the sub-optimal method, they are improved by 0.58 and 0.53 percentage points, respectively, achieving the optimal level. It can be seen that CMNER method can alleviate data bias and reduce modality gap effectively, thereby enhancing the performance of MNER tasks.

Table and Figures | Reference | Related Articles | Metrics