Journal of Computer Applications

    Next Articles

Review of multi-modal knowledge graph completion methods

WANG Xue, ZHANG Liping, YAN Sheng, LI Na, ZHANG Xuefei   

  1. College of Computer Science and Technology, Inner Mongolia Normal University
  • Received:2025-03-17 Revised:2025-05-22 Online:2025-06-12 Published:2025-06-12
  • Contact: ZHANG Liping
  • About author:WANG Xue, born in 2001, M.S. candidate. Her research interests include educational data mining. ZHANG Liping, born in 1974, M.S., professor. Her research interests include smart education, software engineering. YAN Sheng, born in 1984, M.S., lecturer. His research interests include computer education. LI Na, born in 1980, M.S., professor. Her research interests include education informatization, information technology education, digital learning resources design and development. ZHANG Xuefei, born in 1997, M.S. candidate. His research interests include computer education.
  • Supported by:
    Inner Mongolia Natural Science Foundation (2023LHMS06009, 2024MS06020); 2023 Project of the 14th Five-Year Plan for Educational Science Research of Inner Mongolia Autonomous Region (2023NGHZXZH119, NGJGH2023234); Basic Research Funds of Inner Mongolia Normal University (2022JBQN108,2022JBQN008).

多模态知识图谱补全方法综述

王雪,张丽萍,闫盛,李娜,张学飞   

  1. 内蒙古师范大学 计算机科学技术学院
  • 通讯作者: 张丽萍
  • 作者简介:王雪(2001—),女,内蒙古巴彦淖尔人,硕士研究生,主要研究方向:教育数据挖掘;张丽萍(1974—),女,内蒙古呼和浩特人,教授,硕士,主要研究方向:智慧教育、软件工程;闫盛(1984—),男,内蒙古包头人,讲师,硕士,CCF会员,主要研究方向:计算机教育;李娜(1980—),女,副教授,硕士,主要研究方向:教育信息化、信息技术教育、数字化学习资源设计与开发;张学飞(1997—),男,内蒙古乌兰察布人,硕士研究生,CCF学生会员,主要研究方向:计算机教育。
  • 基金资助:
    内蒙古自然科学基金资助项目(2023LHMS06009,2024MS06020);内蒙古自治区教育科学研究“十四五”规划2023年度课题(2023NGHZXZH119,NGJGH2023234);内蒙古师范大学基本科研业务费专项资金(2022JBQN108,2022JBQN008)

Abstract: Traditional knowledge graphs were employed to provide a unified and machine-interpretable representation of information on the web; however, their limitations in handling multimodal applications were increasingly recognized. To address these limitations, Multi-Modal Knowledge Graphs (MMKGs) were proposed as an effective solution. Nevertheless, the integration of multimodal data into knowledge graphs was often accompanied by challenges such as inadequate modality fusion and reasoning difficulties, which significantly constrained the application and further advancement of MMKGs. Multi-Modal Knowledge Graph Completion (MMKGC) techniques were introduced to facilitate the comprehensive integration of cross-modal information during the construction phase and to enable the prediction of missing links after construction, thereby mitigating issues in modality fusion and reasoning. A detailed overview of MMKGC was presented, including widely used benchmark datasets and evaluation metrics. Subsequently, existing approaches were classified into fusion tasks during the MMKG construction phase and reasoning tasks in the post-construction phase. The former focused on key techniques such as entity alignment and entity linking, while the latter encompassed relation inference, missing information completion, and multimodal expansion. Representative MMKGC methods in each category were thoroughly introduced and their characteristics systematically analyzed. Finally, the prevailing challenges faced by MMKGC were examined, and a comprehensive summary was provided.

Key words: multi-modal data, multi-modal knowledge graph, Multi-Modal Knowledge Graphs Completion (MMKGC), entity alignment, relation inference 

摘要: 传统知识图谱为网络中的信息提供一种统一的、机器可理解的表示方式,但在处理多模态应用时逐渐暴露出局限性,多模态知识图谱能够有效解决这些问题。然而,知识图谱引入多模态数据后广泛存在模态融合不充分、推理困难的问题,这制约了多模态知识图谱的应用和发展。多模态知识图谱补全(MMKGC)技术不仅能够在构建阶段充分融合跨模态信息,还能够在构建完成阶段预测其中缺失的链接,以解决在模态融合和推理时遇到的问题。首先,详尽阐述了MMKGC的基本概述以及常用的基准数据集和评价指标;其次,将现有方法分为针对多模态知识图谱构建阶段的融合任务和构建完成阶段的推理任务,前者聚焦于关键技术实体对齐和实体链接,后者则涵盖关系推理、信息缺失补全及多模态扩展三类技术,对各类MMKGC方法进行详细介绍并分析其特点;最后,分析MMKGC面临的问题与挑战并总结全文。

关键词: 多模态数据, 多模态知识图谱, 多模态知识图谱补全, 实体对齐, 关系推理

CLC Number: