Awesome

KG-MM-Survey

Task

🙌 This repository collects papers integrating Knowledge Graphs (KGs) and Multi-Modal Learning, focusing on research in two principal aspects: KG-driven Multi-Modal (KG4MM) learning, where KGs support multi-modal tasks, and Multi-Modal Knowledge Graph (MM4KG), which extends KG studies into the MMKG realm.

😎 Welcome to recommend missing papers through Adding Issues or Pull Requests.

<details> <summary>👈 🔎 Roadmap </summary>

Roadmap

</details>

🔔 News

2024-02 We preprint our Survey Knowledge Graphs Meet Multi-Modal Learning: A Comprehensive Survey [Repo].

Todo:

- Finish updating papers

📜 Content

KG-MM-Survey

🤖🌄 KG-driven Multi-modal Learning (KG4MM)

Understanding & Reasoning Tasks

<details> <summary>👈 🔎 Pipeline </summary>

KG4MMR

</details>

Visual Question Answering

<details> <summary>👈 🔎 Benchmarks </summary>

VQA

</details>

[arXiv 2024] Knowledge Condensation and Reasoning for Knowledge-based VQA.
[arXiv 2024] VCD: Knowledge Base Guided Visual Commonsense Discovery in Images.
[arXiv 2024] Cognitive Visual-Language Mapper: Advancing Multimodal Comprehension with Enhanced Visual Knowledge Alignment.
[ACL 2024] Modality-Aware Integration with Large Language Models for Knowledge-based Visual Question Answering.
[arXiv 2024] II-MMR: Identifying and Improving Multi-modal Multi-hop Reasoning in Visual Question Answering.
[arXiv 2024] Knowledge Generation for Zero-shot Knowledge-based VQA.
[arXiv 2024] GeReA: Question-Aware Prompt Captions for Knowledge-based Visual Question Answering.
[arXiv 2024] Advancing Large Multi-modal Models with Explicit Chain-of-Reasoning and Visual Question Generation.
[AAAI 2024] BOK-VQA: Bilingual outside Knowledge-Based Visual Question Answering via Graph Representation Pretraining.
[arXiv 2024] Cross-modal Retrieval for Knowledge-based Visual Question Answering.
[TMM 2024] Learning to Supervise Knowledge Retrieval over a Tree Structure for Visual Question Answering.
[MTA 2024] Hierarchical Attention Networks for Fact-based Visual Question Answering.
[KAIS 2024] Knowledge enhancement and scene understanding for knowledge-based visual question answering.
[arXiv 2023] Multi-Clue Reasoning with Memory Augmentation for Knowledge-based Visual Question Answering.
[arXiv 2023] Open-Set Knowledge-Based Visual Question Answering with Inference Paths.
[arXiv 2023] Prompting Vision Language Model with Knowledge from Large Language Model for Knowledge-Based VQA.
[EMNLP 2023] Language Guided Visual Question Answering: Elevate Your Multimodal Language Model Using Knowledge-Enriched Prompts.
[EMNLP 2023] A Simple Baseline for Knowledge-Based Visual Question Answering.
[EMNLP 2023] MM-Reasoner: A Multi-Modal Knowledge-Aware Framework for Knowledge-Based Visual Question Answering.
[NeurIPS 2023] LoRA: A Logical Reasoning Augmented Dataset for Visual Question Answering.
[CVPR 2023] Prompting Large Language Models with Answer Heuristics for Knowledge-Based Visual Question Answering.
[EACL 2023] FVQA 2.0: Introducing Adversarial Samples into Fact-based Visual Question Answering.
[WACV 2023] VLC-BERT: Visual Question Answering with Contextualized Commonsense Knowledge.
[ICASSP 2023] Outside Knowledge Visual Question Answering Version 2.0.
[ICME 2023] A Retriever-Reader Framework with Visual Entity Linking for Knowledge-Based Visual Question Answering.
[TIP 2023] Semantic-Aware Modular Capsule Routing for Visual Question Answering.
[ACM MM 2023] AI-VQA: Visual Question Answering based on Agent Interaction with Interpretability.
[SIGIR 2023] A Symmetric Dual Encoding Dense Retrieval Framework for Knowledge-Intensive Visual Question Answering.
[ICMR 2023] Explicit Knowledge Integration for Knowledge-Aware Visual Question Answering about Named Entities.
[TMM 2023] Resolving Zero-shot and Fact-based Visual Question Answering via Enhanced Fact Retrieval.
[ESA 2023] Image captioning for effective use of language models in knowledge-based visual question answering.
[EMNLP 2022] Retrieval Augmented Visual Question Answering with Outside Knowledge.
[EMNLP 2022] Entity-Focused Dense Passage Retrieval for Outside-Knowledge Visual Question Answering.
[IJCKG 2022] LaKo: Knowledge-driven Visual Question Answering via Late Knowledge-to-Text Injection.
[NeurIPS 2022] REVIVE: Regional Visual Representation Matters in Knowledge-Based Visual Question Answering.
[CVPR 2022] MuKEA: Multimodal Knowledge Extraction and Accumulation for Knowledge-based Visual Question Answering.
[CVPR 2022] Transform-Retrieve-Generate: Natural Language-Centric Outside-Knowledge Visual Question Answering.
[ECCV 2022] A-OKVQA: A Benchmark for Visual Question Answering Using World Knowledge.
[ICCV 2022] VQA-GNN: Reasoning with Multimodal Semantic Graph for Visual Question Answering.
[AAAI 2022] Dynamic Key-Value Memory Enhanced Multi-Step Graph Reasoning for Knowledge-Based Visual Question Answering.
[AAAI 2022] An Empirical Study of GPT-3 for Few-Shot Knowledge-Based VQA.
[ACM MM 2022] A Unified End-to-End Retriever-Reader Framework for Knowledge-based VQA.
[ACL 2022] Hypergraph Transformer: Weakly-Supervised Multi-hop Reasoning for Knowledge-based Visual Question Answering.
[WWW 2022] Improving and Diagnosing Knowledge-Based Visual Question Answering via Entity Enhanced Knowledge Injection.
[SITIS 2022] Multimodal Knowledge Reasoning for Enhanced Visual Question Answering.
[KBS 2022] Fact-based visual question answering via dual-process system.
[ISWC 2021] Zero-Shot Visual Question Answering Using Knowledge Graph.
[ISWC 2021] Graphhopper: Multi-hop Scene Graph Reasoning for Visual Question Answering.
[ACL 2021] In Factuality: Efficient Integration of Relevant Facts for Visual Question Answering.
[KDD 2021] Select, Substitute, Search: A New Benchmark for Knowledge-Augmented Visual Question Answering.
[CVPR 2021] KRISP: Integrating Implicit and Symbolic Knowledge for Open-Domain Knowledge-Based VQA.
[PR 2021] Knowledge base graph embedding module design for Visual question answering model.
[SIGIR 2021] Passage Retrieval for Outside-Knowledge Visual Question Answering.
[TNNLS 2021] Rich Visual Knowledge-Based Augmentation Network for Visual Question Answering.
[COLING 2020] Towards Knowledge-Augmented Visual Question Answering.
[arXiv 2020] Seeing is Knowing! Fact-based Visual Question Answering using Knowledge Graph Embeddings.
[ACM MM 2020] Boosting Visual Question Answering with Context-aware Knowledge Aggregation.
[EMNLP 2020] ConceptBert: Concept-Aware Representation for Visual Question Answering.
[PR 2020] Cross-modal knowledge reasoning for knowledge-based visual question answering.
[IJCAI 2020] Mucko: Multi-Layer Cross-Modal Knowledge Reasoning for Fact-based Visual Question Answering.
[AAAI 2020] KnowIT VQA: Answering Knowledge-Based Questions about Videos.
[AAAI 2019] KVQA: Knowledge-Aware Visual Question Answering.
[CVPR 2019] OK-VQA: Visual Question Answering Benchmark Requiring External Knowledge.
[NeurIPS 2018] Out of the Box: Reasoning with Graph Convolution Nets for Factual Visual Question Answering.
[ECCV 2018] Straight to the Facts: Learning Knowledge Base Retrieval for Factual Visual Question Answering.
[CVPR 2018] Learning Visual Knowledge Memory Networks for Visual Question Answering.
[KDD 2018] R-VQA: Learning Visual Relation Facts with Semantic Attention for Visual Question Answering.
[TPAMI 2018] FVQA: Fact-Based Visual Question Answering.
[IJCAI 2017] Explicit Knowledge-based Reasoning for Visual Question Answering.
[CVPR 2016] Ask Me Anything: Free-Form Visual Question Answering Based on Knowledge from External Sources.

Visual Question Generation

[arXiv 2024] ConVQG: Contrastive Visual Question Generation with Multimodal Guidance.
[ICTIR 2023] Pre-Training Multi-Modal Dense Retrievers for Outside-Knowledge Visual Question Answering.
[ACM MM 2023] Deconfounded Visual Question Generation with Causal Inference.
[WACV 2023] K-VQG: Knowledge-aware Visual Question Generation for Common-sense Acquisition.
[TCSVT 2022] Knowledge-Based Visual Question Generation.

Visual Dialog

[WACV 2024] VD-GR: Boosting Visual Dialog with Cascaded Spatial-Temporal Multi-Modal GRaphs.
[ICMR 2023] Knowledge-Aware Causal Inference Network for Visual Dialog.
[TCSVT 2023] Heterogeneous Knowledge Network for Visual Dialog.
[TMM 2023] Counterfactual Visual Dialog: Robust Commonsense Knowledge Learning from Unbiased Training.
[TPAMI 2022] Context-Aware Graph Inference With Knowledge Distillation for Visual Dialog.
[ACM MM 2022] Unified Multimodal Model with Unlikelihood Training for Visual Dialog.
[arXiv 2022] Multimodal Dialog Systems with Dual Knowledge-enhanced Generative Pretrained Language Model.
[CVPR 2022 Workshops] Reasoning with Multi-Structure Commonsense Knowledge in Visual Dialog.
[EMNLP 2021] Reasoning Visual Dialog with Sparse Graph Learning and Knowledge Transfer.
[ICME 2021] SKANet: Structured Knowledge-Aware Network for Visual Dialog.
[ACM MM 2020] KBGN: Knowledge-Bridge Graph Network for Adaptive Vision-Text Reasoning in Visual Dialogue.
[CVPR 2020] Iterative Context-Aware Graph Inference for Visual Dialog.

Classification Tasks

<details> <summary>👈 🔎 Comparison </summary>

IMGC

</details>

Image Classification

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/imgctab.jpg" width="45%" height="auto" /> </div> </details>

[arXiv 2023] Bringing Back the Context: Camera Trap Species Identification as Link Prediction on Multimodal Knowledge Graphs.
[AAAI 2023] DUET: Cross-Modal Semantic Grounding for Contrastive Zero-Shot Learning.
[KDD 2023] Recognizing Unseen Objects via Multimodal Intensive Knowledge Graph Propagation.
[JWS 2023] Benchmarking knowledge-driven zero-shot learning.
[ICTAI 2022] A Zero-shot Learning Method with a Multi-Modal Knowledge Graph.
[KDD 2022] Disentangled Ontology Embedding for Zero-shot Learning.
[Trans. Mach. Learn. Res. 2022] Zero-Shot Learning with Common Sense Knowledge Graphs.
[Cogn. Comput. 2022] Improving Zero-Shot Learning Baselines with Commonsense Knowledge.
[ICCV 2021 Workshop] Zero-Shot Learning via Contrastive Learning on Dual Knowledge Graphs.
[WWW 2021] OntoZSL: Ontology-enhanced Zero-shot Learning.
[AAAI 2020] Zero-Shot Ingredient Recognition by Multi-Relational Graph Convolutional Network.
[KR 2020] Ontology-guided Semantic Composition for Zero-shot Learning.
[AAAI 2019] I Know the Relationships: Zero-Shot Action Recognition via Two-Stream Graph Convolutional Networks and Knowledge Graphs.
[CVPR 2019] Rethinking Knowledge Graph Propagation for Zero-Shot Learning.
[CVPR 2018] Zero-Shot Recognition via Semantic Embeddings and Knowledge Graphs.

Fake News Detection

[AAAI 2024] Reinforced Adaptive Knowledge Learning for Multimodal Fake News Detection.
[ADMA 2023] EmoKnow: Emotion- and Knowledge-Oriented Model for COVID-19 Fake News Detection.
[TKDE 2023] Inconsistent Matters: A Knowledge-Guided Dual-Consistency Network for Multi-Modal Rumor Detection.
[TOMCCAP 2021] Knowledge-aware Multi-modal Adaptive Graph Convolutional Networks for Fake News Detection.
[ICMR 2020] Fake News Detection via Knowledge-driven Multimodal Graph Convolutional Networks.

Movie Genre Classification

[ACM MM 2023] Incorporating Domain Knowledge Graph into Multimodal Movie Genre Classification with Self-Supervised Attention and Contrastive Learning.

Content Generation Tasks

Image Captioning

[TMM 2024] Exploring Visual Relationships via Transformer-based Graphs for Enhanced Image Captioning.
[ISPP 2023] Image caption generation based on object detection and knowledge enhancement.
[arXiv 2022] Generating image captions with external encyclopedic knowledge.
[PR Lett. 2021] Image captioning with transformer and knowledge graph.
[arXiv 2021] Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph.
[arXiv 2020] Integrating Image Captioning with Rule-based Entity Masking.
[AAAI 2020] When Radiology Report Generation Meets Knowledge Graph.
[CIKM 2020] Image Captioning with Internal and External Knowledge.
[AAAI 2020] Joint Commonsense and Relation Reasoning for Image and Video Captioning.
[arXiv 2019] Relational Reasoning using Prior Knowledge for Visual Captioning.
[WACV 2019] Improving Image Captioning by Leveraging Knowledge Graphs.
[TMM 2019] Know More Say Less: Image Captioning Based on Scene Graphs.
[EMNLP 2018] Entity-aware Image Caption Generation.
[arXiv 2015] From Images to Sentences through Scene Description Graphs using Commonsense Reasoning and Knowledge.

Visual Storytelling

[arXiv 2024] SCO-VIST: Social Interaction Commonsense Knowledge-based Visual Storytelling.
[TPAMI 2023] Knowledge-Enriched Attention Network With Group-Wise Semantic for Visual Storytelling.
[AAAI 2021] Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning.
[ACL 2021] Plot and Rework: Modeling Storylines for Visual Storytelling.
[AAAI 2021] Imagine, Reason and Write: Visual Storytelling with Graph Knowledge and Relational Reasoning.
[AAAI 2021] Commonsense Knowledge Aware Concept Selection For Diverse and Informative Visual Storytelling.
[AAAI 2020] Knowledge-Enriched Visual Storytelling.
[IJCAI 2019] Knowledgeable Storyteller: A Commonsense-Driven Generative Model for Visual Storytelling.

Conditional Text-to-Image Generation

[arXiv 2023] T2TD: Text-3D Generation Model based on Prior Knowledge Guidance.
[Pattern Recognition Letters 2023] Prior knowledge guided text to image generation.
[TCSVT 2022] RiFeGAN2: Rich Feature Generation for Text-to-Image Synthesis From Constrained Prior Knowledge.
[CSAE 2021] Enriching Attributes from Knowledge Graph for Fine-grained Text-to-Image Synthesis.
[ISCID 2021] KnHiGAN: Knowledge-enhanced Hierarchical Generative Adversarial Network for Fine-grained Text-to-Image Synthesis.
[CVPR 2020] RiFeGAN: Rich Feature Generation for Text-to-Image Synthesis From Prior Knowledge.
[NeurIPS 2019] Learn, Imagine and Create: Text-to-Image Generation from Prior Knowledge.

Scene Graph Generation

[arXiv 2024] External Knowledge Enhanced 3D Scene Generation from Sketch.
[arXiv 2024] Towards lifelong scene graph generation with knowledge-ware in-context prompt learning.
[CVPR 2024] HiKER-SGG: Hierarchical Knowledge Enhanced Robust Scene Graph Generation.
[AAAI 2024] R3CD: Scene Graph to Image Generation with Relation-Aware Compositional Contrastive Control Diffusion.
[ACM MM 2023] Prior Knowledge-driven Dynamic Scene Graph Generation with Causal Inference.
[WACV 2023] More Knowledge, Less Bias: Unbiasing Scene Graph Generation with Explicit Ontological Adjustment.
[ICME 2022] Zero-Shot Scene Graph Generation with Knowledge Graph Completion.
[ESWC 2022] Expressive Scene Graph Generation Using Commonsense Knowledge Infusion for Visual Understanding and Reasoning.
[IJCAI 2020] Learning from the Scene and Borrowing from the Rich: Tackling the Long Tail in Scene Graph Generation.
[ECCV 2020] Learning Visual Commonsense for Robust Scene Graph Generation.
[ECCV 2020] Bridging Knowledge Graphs to Generate Scene Graphs.
[CVPR 2019] Scene Graph Generation With External Knowledge and Image Reconstruction.
[CVPR 2019] Knowledge-Embedded Routing Network for Scene Graph Generation.
[IJCAI 2018] Representation Learning for Scene Graph Completion via Jointly Structural and Visual Embedding.
[CVPR 2018] Neural Motifs: Scene Graph Parsing With Global Context.

Retrieval Tasks

Cross-Modal Retrieval

[AAAI 2024] Self-Supervised Multi-Modal Knowledge Graph Contrastive Hashing for Cross-Modal Search.
[WSDM 2023] AGREE: Aligning Cross-Modal Entities for Image-Text Retrieval Upon Vision-Language Pre-trained Models.
[ACM MM 2023] External Knowledge Dynamic Modeling for Image-text Retrieval.
[TOMCCAP 2023] MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text Retrieval.
[TPAMI 2023] Entity-Graph Enhanced Cross-Modal Pretraining for Instance-Level Product Retrieval.
[TMM 2023] Commonsense-Guided Semantic and Relational Consistencies for Image-Text Retrieval.
[NeurIPS 2022] MACK: Multimodal Aligned Conceptual Knowledge for Unpaired Image-text Matching.
[ECCV 2022] CODER: Coupled Diversity-Sensitive Momentum Contrastive Learning for Image-Text Retrieval.
[ECCV 2020] Consensus-Aware Visual-Semantic Embedding for Image-Text Matching.
[IJCAI 2019] Knowledge Aware Semantic Concept Expansion for Image-Text Matching.

Visual Referring Expressions & Grounding

[AAAI 2024] Augmented Commonsense Knowledge for Remote Object Grounding.
[CVPR 2023] Advancing Visual Grounding with Scene Knowledge: Benchmark and Method.
[ACL 2023] Segment-Level and Category-Oriented Network for Knowledge-Based Referring Expression Comprehension.
[EACL 2023] CK-Transformer: Commonsense Knowledge Enhanced Transformers for Referring Expression Comprehension.
[AAAI 2022] Improving Zero-Shot Phrase Grounding via Reasoning on External Knowledge and Spatial Relations.
[ACM MM 2020] Give Me Something to Eat: Referring Expression Comprehension with Commonsense Knowledge.
[CVPR 2018] Knowledge Aided Consistency for Weakly Supervised Phrase Grounding.

KG-aware Mutli-modal Pre-training

Structure Knowledge aware Pre-training

[AAAI 2024] Structure-CLIP: Enhance Multi-modal Language Representations with Structure Knowledge.
[ACM MM 2021] ROSITA: Enhancing Vision-and-Language Semantic Alignments via Cross- and Intra-modal Knowledge Integration.
[ACL 2021] KM-BART: Knowledge Enhanced Multimodal BART for Visual Commonsense Generation.
[AAAI 2021] ERNIE-ViL: Knowledge Enhanced Vision-Language Representations through Scene Graphs.

Knowledge Graph aware Pre-training

[WWW 2023] Structure Pretraining and Prompt Tuning for Knowledge Graph Transfer.
[CVPR 2023] Improving Commonsense in Vision-Language Models via Knowledge Graph Riddles.
[ACM MM 2022] Align, Reason and Learn: Enhancing Medical Vision-and-Language Pre-training with Knowledge.

🌄🤖 Multi-modal Knowledge Graph (MM4KG)

<details> <summary>👈 🔎 N-MMKG Ontology </summary>

MMKGOnto

</details> <details> <summary>👈 🔎 Taxonomy </summary> <div align="center"> <img src="figures/mmkgtask.jpg" width="90%" height="auto" /> </div> </details>

MMKG Resources

Public MMKGs

<details> <summary>👈 🔎 MMKG Overview </summary>

MMKG

</details>

[AAAI 2024] Beyond Entities: A Large-Scale Multi-Modal Knowledge Graph with Triplet Fact Grounding.
[arXiv 2023] M2ConceptBase: A Fine-grained Aligned Multi-modal Conceptual Knowledge Base.
[arXiv 2023] Scene-Driven Multimodal Knowledge Graph Construction for Embodied AI.
[EMNLP 2023] VISTA: Visual-Textual Knowledge Graph Representation Learning.
[ISWC 2023] MMpedia: A Large-Scale Multi-modal Knowledge Graph.
[ACM MM 2023] TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio.
[CIKM 2023] AspectMMKG: A Multi-modal Knowledge Graph with Aspect-aware Entities.
[ISWC 2023] Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment.
[arXiv 2023] UKnow: A Unified Knowledge Protocol for Common-Sense Reasoning and Vision-Language Pre-training.
[ICASSP 2023] Vision, Deduction and Alignment: An Empirical Study on Multi-Modal Knowledge Graph Alignment.
[ICLR 2023] Multimodal Analogical Reasoning over Knowledge Graphs.
[ACM MM 2022] Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion.
[NAACL 2021] RESIN: A Dockerized Schema-Guided Cross-document Cross-lingual Cross-media Information Extraction and Event Tracking System.
[AAAI 2021] Visual Pivoting for (Unsupervised) Entity Alignment.
[EMNLP 2021] VisualSem: a high-quality knowledge graph for vision and language.
[BDR 2020] Richpedia: A Large-Scale, Comprehensive Multi-Modal Knowledge Graph.
[ACL 2020] GAIA: A Fine-grained Multimedia Knowledge Extraction System.
[ESWC 2019] MMKG: Multi-modal Knowledge Graphs.
[ISWC 2017] IMGpedia: A Linked Dataset with Content-Based Analysis of Wikimedia Images.
[TIP 2017] Robust ImageGraph: Rank-Level Feature Fusion for Image Search.
[IJCV 2017] Visual Genome: Connecting Language and Vision Using Crowdsourced Dense Image Annotations.
[ICCV 2013] NEIL: Extracting Visual Knowledge from Web Data.

MMKG Construction Methods

[DASFAA 2022] Visualizable or Non-visualizable? Exploring the Visualizability of Concepts in Multi-modal Knowledge Graph.
[ACM MM 2020] Towards Using Semantic-Web Technologies for Multi-Modal Knowledge Graph Construction.
[AKBC 2019] Answering Visual-Relational Queries in Web-Extracted Knowledge Graphs.
[IJCAI 2017] Extracting Visual Knowledge from the Web with Multimodal Learning.
[IJCAI 2017] Image-embodied Knowledge Representation Learning.
[ECCV 2016] Visual Relationship Detection with Language Priors.
[NAACL 2016] Stating the Obvious: Extracting Visual Common Sense Knowledge.
[CVPR 2015] Image retrieval using scene graphs.
[CVPR 2014] Enriching Visual Knowledge Bases via Object Discovery and Segmentation.

MMKG Acquisition

Multi-modal Named Entity Recognition

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/mnertab.jpg" width="45%" height="auto" /> </div> </details>

[arXiv 2024] LLMs as Bridges: Reformulating Grounded Multimodal Named Entity Recognition.
[arXiv 2023] A Novel Framework for Multimodal Named Entity Recognition with Multi-level Alignments.
[ACM MM 2023] Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction.
[EMNLP 2023] Prompting ChatGPT in MNER: Enhanced Multimodal Named Entity Recognition with Auxiliary Refined Knowledge.
[CIKM 2023] MGICL: Multi-Grained Interaction Contrastive Learning for Multimodal Named Entity Recognition.
[EMNLP 2022] Named Entity and Relation Extraction with Multi-Modal Retrieval.
[ICME 2022] CAT-MNER: Multimodal Named Entity Recognition with Knowledge-Refined Cross-Modal Attention.
[DASFAA 2022] PromptMNER: Prompt-Based Entity-Related Visual Clue Extraction and Integration for Multimodal Named Entity Recognition.
[NAACL 2022] ITA: Image-Text Alignments for Multi-Modal Named Entity Recognition.
[TASLP 2023] Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck.
[TCSS 2023] Dynamic Graph Construction Framework for Multimodal Named Entity Recognition in Social Media.
[SIGIR 2022] Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion.
[WSDM 2023] Reducing the Bias of Visual Objects in Multimodal Named Entity Recognition.
[AAAI 2023] MNER-QG: An End-to-End MRC Framework for Multimodal Named Entity Recognition with Query Grounding.
[TAI 2023] Text-Image Scene Graph Fusion for Multi-Modal Named Entity Recognition.
[CIKM 2023] MPMRC-MNER: A Unified MRC framework for Multimodal Named Entity Recognition based Multimodal Prompt.
[ACM MM 2022] Learning from Different text-image Pairs: A Relation-enhanced Graph Convolutional Network for Multimodal NER.
[IJMLC 2023] ‘What’and ‘Where’both matter: dual cross-modal graph convolutional networks for multimodal named entity recognition.
[EMNLP 2022] Entity-level Interaction via Heterogeneous Graph for Multimodal Named Entity Recognition.
[ACM MM 2022] Query Prior Matters: A MRC Framework for Multimodal Named Entity Recognition.
[COLING 2022] Flat Multi-modal Interaction Transformer for Named Entity Recognition.
[NAACL 2022] Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction.
[EMNLP 2022] A Span-based Multimodal Variational Autoencoder for Semi-supervised Multimodal Named Entity Recognition.
[WSDM 2022] MAF: A General Matching and Alignment Framework for Multimodal Named Entity Recognition.
[Appl. Intell. 2022] UAMNer: uncertainty-aware multimodal named entity recognition in social media posts.
[IPM 2024] Multi-granularity cross-modal representation learning for named entity recognition on social media.
[IEICE Trans. Inf. Syst. 2023] Multimodal Named Entity Recognition with Bottleneck Fusion and Contrastive Learning.
[AAAI 2021] Multi-modal Graph Fusion for Named Entity Recognition with Targeted Visual Guidance.
[ACL 2020] Improving Multimodal Named Entity Recognition via Entity Span Detection with Unified Multimodal Transformer.
[COLING 2020] RIVA: A Pre-trained Tweet Multimodal Model Based on Text-image Relation for Multimodal NER.
[DASFAA 2021] Multimodal Named Entity Recognition with Image Attributes and Image Knowledge.
[ACM MM 2021] Multimodal Relation Extraction with Efficient Graph Alignment.
[AAAI 2021] RpBERT: A Text-image Relation Propagation-based BERT Model for Multimodal NER.
[ACM MM 2020] Multimodal Representation with Embedded Visual Guiding Objects for Named Entity Recognition in Social Media Posts.

Multi-modal Relation Extraction

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/mmretab.jpg" width="45%" height="auto" /> </div> </details>

[ACM MM 2023] Prompt Me Up: Unleashing the Power of Alignments for Multimodal Entity and Relation Extraction.
[EMNLP 2023] Towards Zero-shot Relation Extraction in Web Mining: A Multimodal Approach with Relative XML Path.
[ACL 2023] Multimodal Relation Extraction with Cross-Modal Retrieval and Synthesis.
[TASLP 2023] Enhancing Multimodal Entity and Relation Extraction with Variational Information Bottleneck.
[ACL 2023] Dual-Gated Fusion with Prefix-Tuning for Multi-Modal Relation Extraction.
[ACL 2023] Information Screening whilst Exploiting! Multimodal Relation Extraction with Feature Denoising and Multimodal Topic Modeling.
[AAAI 2023 (Student Abstract)] On Analyzing the Role of Image for Visual-Enhanced Relation Extraction.
[SIGIR 2022] Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion.
[NAACL 2022] Good Visual Guidance Make A Better Extractor: Hierarchical Visual Prefix for Multimodal Entity and Relation Extraction.
[EMNLP 2022] Named Entity and Relation Extraction with Multi-Modal Retrieval.
[ACM MM 2021] Multimodal Relation Extraction with Efficient Graph Alignment.
[ICME 2021] MNRE: A Challenge Multimodal Dataset for Neural Relation Extraction with Visual Evidence in Social Media Posts.

Multi-modal Event Extraction

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/mmeetab.jpg" width="45%" height="auto" /> </div> </details>

Image-Text:

[ACM MM 2023] Training Multimedia Event Extraction With Generated Images and Captions.
[ACM MM 2022] Multimedia Event Extraction From News With a Unified Contrastive Learning Framework.
[CVPR 2022] CLIP-Event: Connecting Text and Images with Event Structures.
[ACL 2020] Cross-media Structured Common Space for Multimedia Event Extraction.

Video-Text:

[AAAI 2024] Video Event Extraction with Multi-View Interaction Knowledge Distillation.
[AAAU 2024] Beyond Grounding: Extracting Fine-Grained Event Hierarchies across Modalities.
[FUSION 2023] Few-shot Domain-Adaptative Visually-fused Event Detection from Text.
[EMNLP 2023] Three Stream Based Multi-level Event Contrastive Learning for Text-Video Event Extraction.
[DASFAA 2023] Cross-Modal Contrastive Learning for Event Extraction.
[EMNLP 2021] Joint Multimedia Event Extraction from Video and Article.

MMKG Fusion

Multi-modal Entity Alignment

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/mmeatab.jpg" width="45%" height="auto" /> </div> </details>

[arXiv 2024] The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework.
[arXiv 2024] Pseudo-Label Calibration Semi-supervised Multi-Modal Entity Alignment.
[arXiv 2024] ASGEA: Exploiting Logic Rules from Align-Subgraphs for Entity Alignment.
[ICLR 2024] Revisit and Outstrip Entity Alignment: A Perspective of Generative Models.
[arXiv 2024] Towards Semantic Consistency: Dirichlet Energy Driven Robust Multi-Modal Entity Alignment.
[EMNLP 2023] Multi-Modal Knowledge Graph Transformer Framework for Multi-Modal Entity Alignment.
[arXiv 2023] Universal Multi-modal Entity Alignment via Iteratively Fusing Modality Similarity Paths.
[ACM MM 2023] MEAformer: Multi-modal Entity Alignment Transformer for Meta Modality Hybrid.
[ACM MM 2023] Cross-Modal Graph Attention Network for Entity Alignment.
[ACM MM 2023] PSNEA: Pseudo-Siamese Network for Entity Alignment between Multi-modal Knowledge Graphs.
[ISWC 2023] Rethinking Uncertainly Missing and Ambiguous Visual Modality in Multi-Modal Entity Alignment.
[WWW 2023] Attribute-Consistent Knowledge Graph Representation Learning for Multi-Modal Entity Alignment.
[ICASSP 2023] Vision, Deduction and Alignment: An Empirical Study on Multi-Modal Knowledge Graph Alignment.
[COLING 2023] Multi-modal Contrastive Representation Learning for Entity Alignment.
[DSE 2023] Probing the Impacts of Visual Context in Multimodal Entity Alignment.
[KSEM 2023] Enhanced Entity Interaction Modeling for Multi-Modal Entity Alignment.
[Inf. Fusion 2023] MMIEA: Multi-modal Interaction Entity Alignment model for knowledge graphs.
[ESWA 2023] Leveraging multimodal features for knowledge graph entity alignment based on dynamic self-attention networks.
[Neurocomput. 2022] MultiJAF: Multi-modal joint entity alignment framework for multi-modal knowledge graph.
[KDD 2022] Multi-modal Siamese Network for Entity Alignment.
[Neurocomput. 2021] Multi-modal entity alignment in hyperbolic space.
[AAAI 2021] Visual Pivoting for (Unsupervised) Entity Alignment.
[KSEM 2020] MMEA: Entity Alignment for Multi-modal Knowledge Graph.

Multi-modal Entity Linking & Disambiguation

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/mmeltab.jpg" width="45%" height="auto" /> </div> </details>

[AAAI 2024] A dual-way enhanced framework from text matching point of view for multimodal entity linking.
[arXiv 2023] A Dual-way Enhanced Framework from Text Matching Point of View for Multimodal Entity Linking.
[arXiv 2023] Generative Multimodal Entity Linking.
[ACL 2023] Benchmarking Diverse-Modal Entity Linking with Generative Models.
[arXiv 2022] Kuaipedia: a Large-scale Multi-modal Short-video Encyclopedia.
[KDD 2023] Multi-Grained Multimodal Interaction Network for Entity Linking.
[UAI 2023] MMEL: A Joint Learning Framework for Multi-Mention Entity Linking.
[CVPR 2022] An Empirical Study of Training End-to-End Vision-and-Language Transformers.
[EMNLP 2022] Multimodal Knowledge Learning for Named Entity Disambiguation.
[SIGIR 2022] Multimodal Entity Linking with Gated Hierarchical Fusion and Contrastive Training.
[ICLR 2021] Autoregressive Entity Retrieval.
[NeurIPS 2021] Align before Fuse: Vision and Language Representation Learning with Momentum Distillation.
[ECIR 2020] Multimodal Entity Linking for Tweets.
[EMNLP 2020] Scalable Zero-shot Entity Linking with Dense Entity Retrieval.
[ACL 2018] Multimodal Named Entity Disambiguation for Noisy Social Media Posts.

MMKG Inference

Multi-modal Knowledge Graph Completion

<details> <summary>👈 🔎 Benchmarks </summary> <div align="center"> <img src="figures/mkgctab.jpg" width="45%" height="auto" /> </div> </details>

[arXiv 2024] MyGO: Discrete Modality Information as Fine-Grained Tokens for Multi-modal Knowledge Graph Completion.
[SIGIR 2024] NativE: Multi-modal Knowledge Graph Completion in the Wild.
[arXiv 2024] The Power of Noise: Toward a Unified Multi-modal Knowledge Graph Representation Framework.
[AAAI 2024] LAFA: Multimodal Knowledge Graph Completion with Link Aware Fusion and Aggregation.
[COLING 2024] Unleashing the Power of Imbalanced Modality Information for Multi-modal Knowledge Graph Completion.
[EMNLP 2023] VISTA: Visual-Textual Knowledge Graph Representation Learning.
[ACM MM 2023] TIVA-KG: A Multimodal Knowledge Graph with Text, Image, Video and Audio.
[TOMCCAP 2023] Hyper-node Relational Graph Attention Network for Multi-modal Knowledge Graph Completion.
[arXiv 2023] Structure Guided Multi-modal Pre-trained Transformer for Knowledge Graph Reasoning.
[TKDE 2023] Contrastive Multi-Modal Knowledge Graph Representation Learning.
[ICDE 2023] Multimodal Biological Knowledge Graph Completion via Triple Co-Attention Mechanism.
[ECML/PKDD 2022] MULTIFORM: Few-Shot Knowledge Graph Completion via Multi-modal Contexts.
[WWW 2023] IMF: Interactive Multimodal Fusion Model for Link Prediction.
[SIGIR 2022] Hybrid Transformer with Multi-level Fusion for Multimodal Knowledge Graph Completion.
[ACM MM 2022] Relation-enhanced Negative Sampling for Multimodal Knowledge Graph Completion.
[IJCNN 2023] Modality-Aware Negative Sampling for Multi-modal Knowledge Graph Embedding.
[Appl. Intell. 2022] MMKRL: A robust embedding approach for multi-modal knowledge graph representation learning.
[EMNLP 2022] MoSE: Modality Split and Ensemble for Multimodal Knowledge Graph Completion.
[NeurIPS 2022] OTKGE: Multi-modal Knowledge Graph Embeddings via Optimal Transport.
[KDD 2022] Knowledge Graph Completion with Pre-trained Multimodal Transformer and Twins Negative Sampling.
[ACM MM 2021] Is Visual Context Really Helpful for Knowledge Graph? A Representation Learning Perspective.
[IJCNN 2019] Multimodal Data Enhanced Representation Learning for Knowledge Graphs.
[EMNLP 2018] Embedding Multimodal Relational Data for Knowledge Base Completion.
[NAACL 2018] A Multimodal Translation-Based Approach for Knowledge Graph Representation Learning.
[IJCAI 2017] Image-embodied Knowledge Representation Learning.

Multi-modal Knowledge Graphs Reasoning

[ICLR 2023] Multimodal Analogical Reasoning over Knowledge Graphs.
[arXiv 2023] Do as I can, not as I get: Topology-aware multi-hop reasoning on multi-modal knowledge graphs.
[ICDE 2023] MMKGR: Multi-hop Multi-modal Knowledge Graph Reasoning.

MMKG-driven Tasks

Retrieval

Image Retrieval:

[EMNLP 2021] VisualSem: a high-quality knowledge graph for vision and language.
[ISWC 2017] IMGpedia: A Linked Dataset with Content-Based Analysis of Wikimedia Images.
[TIP 2017] Robust ImageGraph: Rank-Level Feature Fusion for Image Search.

Cross-modal Retrieval:

[AAAI 2023] Multi-Modal Knowledge Hypergraph for Diverse Image Retrieval.
[TOMCCAP 2023] MKVSE: Multimodal Knowledge Enhanced Visual-semantic Embedding for Image-text Retrieval.
[arXiv 2022] Endowing Language Models with Multimodal Knowledge Graph Representations.
[DASFAA 2021] Multimodal Named Entity Recognition with Image Attributes and Image Knowledge.

Reasoning & Generation

[arXiv 2023] M2ConceptBase: A Fine-grained Aligned Multi-modal Conceptual Knowledge Base.
[JIIS 2023] Self-supervised opinion summarization with multi-modal knowledge graph.
[TMM 2023] Boosting Entity-aware Image Captioning with Multi-modal Knowledge Graph.

Pre-training

Triple-level:

[NeurIPS 2022] Contrastive Language-Image Pre-Training with Knowledge Graphs.

Graph-level:

[NeurIPS 2023] GraphAdapter: Tuning Vision-Language Models With Dual Knowledge Graph.
[arXiv 2023] UKnow: A Unified Knowledge Protocol for Common-Sense Reasoning and Vision-Language Pre-training.

AI for Science

[AAAI 2023 (Student Abstract)] Multi-Modal Protein Knowledge Graph Construction and Applications.
[ICDE 2023] Multimodal Biological Knowledge Graph Completion via Triple Co-Attention Mechanism.
[Nat. Mach. Intell. 2023] Knowledge graph-enhanced molecular contrastive learning with functional prompt.
[AAAI 2022] Molecular Contrastive Learning with Chemical Element Knowledge Graph.
[ICLR 2022] OntoProtein: Protein Pretraining With Gene Ontology Embedding.
[IJCAI 2020] KGNN: Knowledge Graph Neural Network for Drug-Drug Interaction Prediction.

Industry Application

[PRCV 2023] Enhancing Recommender System with Multi-modal Knowledge Graph.
[ACL 2023] FashionKLIP: Enhancing E-Commerce Image-Text Retrieval with Fashion Multi-Modal Conceptual Knowledge Graph.
[ACM MM 2022] Cross-modal Knowledge Graph Contrastive Learning for Machine Learning Method Recommendation.
[CIKM 2021] AliMe MKG: A Multi-modal Knowledge Graph for Live-streaming E-commerce.
[ACM MM 2021] Knowledge Perceived Multi-modal Pretraining in E-commerce.
[CIKM 2020] Multi-modal Knowledge Graphs for Recommender Systems.

Contribution

👥 Contributors

🎉 Contributing ( welcome ! )

✨ Add a new paper or update an existing KG4MM or MM4KG paper.
🧐 Use the same format as existing entries to describe the work.
😄 A very brief explanation why you think a paper should be added or updated is recommended (Not Neccessary) via Adding Issues or Pull Requests.

Don't worry if you put something wrong, they will be fixed for you. Just feel free to contribute and promote your awesome work here! 🤩 We'll get back to you in time ~ 😉

🔖 Contact

📫 zhuo.chen@zju.edu.cn

🤝 Cite:

If this Repo is helpful to you, please consider citing our paper. We would greatly appreciate it :)

@article{chen2024knowledge,
  author       = {Zhuo Chen and
                  Yichi Zhang and
                  Yin Fang and
                  Yuxia Geng and
                  Lingbing Guo and
                  Xiang Chen and
                  Qian Li and
                  Wen Zhang and
                  Jiaoyan Chen and
                  Yushan Zhu and
                  Jiaqi Li and
                  Xiaoze Liu and
                  Jeff Z. Pan and
                  Ningyu Zhang and
                  Huajun Chen},
  title        = {Knowledge Graphs Meet Multi-Modal Learning: {A} Comprehensive Survey},
  journal      = {CoRR},
  volume       = {abs/2402.05391},
  year         = {2024}
}