Journals
  Publication Years
  Keywords
Search within results Open Search
Please wait a minute...
For Selected: Toggle Thumbnails
Multimodal knowledge graph link prediction method based on fusing image and textual information
Huilin GUI, Kun YUE, Liang DUAN
Journal of Computer Applications    2025, 45 (11): 3540-3546.   DOI: 10.11772/j.issn.1001-9081.2024111561
Abstract80)   HTML0)    PDF (995KB)(90)       Save

The introduction of multimodal information to enhance knowledge graph link prediction has become a recent hotspot. However, most existing methods typically rely on simple concatenation or attention mechanisms for multimodal feature fusion, ignoring the correlation and semantic inconsistency between different modalities, which may fail to preserve modality-specific information and inadequately exploit the complementary information between modalities. To address these issues, a multimodal knowledge graph link prediction model based on cross-modal attention mechanism and contrastive learning was proposed, namely FITILP(Fusing Image and Textual Information for Link Prediction). Firstly, pretrained models, such as BERT (Bidirectional Encoder Representation of Transformer) and ResNet (Residual Network), were used to extract textual and visual features of entities. Then, a Contrastive Learning (CL) approach was applied to reduce semantic inconsistencies across modalities. A cross-modal attention module was designed to refine text feature attention parameters using image features, thereby enhancing the cross-modal correlations between text and image features. And Translation models, such as TransE (Translating Embeddings) and TransH (Translation on Hyperplanes), were employed to generate graph structural, visual, and textual features. Finally, the three types of features were fused to perform link prediction between entities. Experimental results on the DB15K dataset show that the FITILP model improves Mean Reciprocal Rank (MRR) by 6.6 percentage points compared to single-modal baseline TransE, and achieves improvements of 3.95, 11.37, and 14.01 percentage points in Hits@1, Hits@10 and Hits@100, respectively. The results indicate that the proposed method outperforms comparative baseline methods, demonstrating its effectiveness in leveraging multimodal information to enhance prediction performance.

Table and Figures | Reference | Related Articles | Metrics
Rough-set based approach to solve the inference conflict in qualitative probabilistic network
Shuang-xian LIU Wei-yi LIU Yue-kun YUE
Journal of Computer Applications   
Abstract1877)      PDF (593KB)(1177)       Save
Qualitative Probabilistic Networks (QPNs) are the qualitative abstraction of Bayesian networks by substituting the conditional probabilistic parameters by qualitative influences on directed edges. Efficient algorithms have been developed for QPN reasoning. Due to the high abstraction, unresolved trade-offs (i.e., conflicts) during inferences with qualitative probabilistic networks may be produced. Motivated by avoiding the conflicts of QPN reasoning, a rough-set-theory based approach was proposed. The attribute association degrees between node peers were calculated based on the rough-set-theory while the QPNs were constructed. The association degrees were adopted as the weights to solve the conflicts during QPN inferences. Accordingly, the algorithm of QPN reasoning was improved by incorporating the attribute association degrees. By applying this method, the efficiency of QPNs inferences can be preserved, and the inference conflict can be well addressed at the same time.
Related Articles | Metrics