Awesome

Awesome-LLM4Graph-Papers

A collection of papers and resources about Large Language Models (LLM) for Graph Learning (Graph).

Graphs are an essential data structure utilized to represent relationships in real-world scenarios. Prior research has established that Graph Neural Networks (GNNs) deliver impressive outcomes in graph-centric tasks, such as link prediction and node classification. Despite these advancements, challenges like data sparsity and limited generalization capabilities continue to persist. Recently, Large Language Models (LLMs) have gained attention in natural language processing. They excel in language comprehension and summarization. Integrating LLMs with graph learning techniques has attracted interest as a way to enhance performance in graph learning tasks.

News

🤗 We're actively working on this project, and your interest is greatly appreciated! To keep up with the latest developments, please consider hit the STAR and WATCH for updates.

🚀 Our LLM4Graph Survey is accepted by KDD 2024, and we will also give a lecture-style tutorial there!
🔥 We gave a tutorial on LLM4Graph at TheWebConf (WWW) 2024!
Our survey paper: A Survey of Large Language Models for Graphs is now ready.

Overview

This repository serves as a collection of recent advancements in employing large language models (LLMs) for modeling graph-structured data. We categorize and summarize the approaches based on four primary paradigms and nine secondary-level categories. The four primary categories include: 1) GNNs as Prefix, 2) LLMs as Prefix, 3) LLMs-Graphs Intergration, and 4) LLMs-Only

GNNs as Prefix

LLMs as Prefix

LLMs-Graphs Intergration

LLMs-Only

We hope this repository proves valuable to your research or practice in the field of self-supervised learning for recommendation systems. If you find it helpful, please consider citing our work:

@inproceedings{ren2024survey,
  title={A survey of large language models for graphs},
  author={Ren, Xubin and Tang, Jiabin and Yin, Dawei and Chawla, Nitesh and Huang, Chao},
  booktitle={Proceedings of the 30th ACM SIGKDD Conference on Knowledge Discovery and Data Mining},
  pages={6616--6626},
  year={2024}
}

@inproceedings{huang2024large,
  title={Large Language Models for Graphs: Progresses and Directions},
  author={Huang, Chao and Ren, Xubin and Tang, Jiabin and Yin, Dawei and Chawla, Nitesh},
  booktitle={Companion Proceedings of the ACM on Web Conference 2024},
  pages={1284--1287},
  year={2024}
}

Awesome-LLM4Graph-Papers

Related Resources

Large language models on graphs: A comprehensive survey [paper]
A Survey of Graph Meets Large Language Model: Progress and Future Directions [paper]

🌐 GNNs as Prefix

Node-level Tokenization

(SIGIR'2024) GraphGPT: Graph instruction tuning for large language models [paper]
(arxiv'2024) HiGPT: Heterogeneous Graph Language Model [paper]
(WWW'2024) GraphTranslator: Aligning Graph Model to Large Language Model for Open-ended Tasks [paper]
(arxiv'2024) UniGraph: Learning a Cross-Domain Graph Foundation Model From Natural Language [paper]
(NeurIPS'2024) GIMLET:Aunifiedgraph-textmodelforinstruction-based molecule zero-shot learning [paper]
(arxiv'2024) XRec: Large Language Models for Explainable Recommendation [paper]

Graph-level

(arxiv'2023) GraphLLM: Boosting graph reasoning ability of large language model [paper]
(Computers in Biology and Medicine) GIT-Mol: A multi-modal large language model for molecular science with graph, image, and text [paper]
(EMNLP'2023) MolCA: Molecular graph-language modeling with cross- modal projector and uni-modal adapter [paper]
(arxiv'2023) InstructMol: Multi-modal integration for building a versatile and reliable molecular assistant in drug discovery [paper]
(arxiv'2024) G-Retriever: Retrieval-Augmented Generation for Textual Graph Understanding and Question Answering [paper]
(AAAI'2024) Graph neural prompting with large language models [paper]

🌐 LLMs as Prefix

Embs. from LLMs for GNNs

(arxiv'2023) Prompt-based node feature extractor for few-shot learning on text-attributed graphs [paper]
(arxiv'2023) SimTeG: A frustratingly simple approach improves textual graph learning [paper]
(KDD'2023) Graph-aware language model pre-training on a large graph corpus can help multiple graph applications [paper]
(ICLR'2024) One for all: Towards training one graph model for all classification tasks [paper]
(ICLR'2024) Harnessing explanations: Llm-to-lm interpreter for enhanced text-attributed graph representation learning [paper]
(WSDM'2024) LLMRec: Large language models with graph augmentation for recommendation [paper]

Labels from LLMs for GNNs

(arxiv'2024) OpenGraph: Towards Open Graph Foundation Models [paper]
(arxiv'2023) Label-free node classification on graphs with large language models (LLMs) [paper]
(arxiv'2024) GraphEdit: Large Language Models for Graph Structure Learning [paper]
(WWW'2024) Representation learning with large language models for recommendation [paper]

🌐 LLMs-Graphs Intergration

Alignment between GNNs and LLMs

(arxiv'2022) A molecular multimodal foundation model associating molecule graphs with natural language [paper]
(arxiv'2023) ConGraT: Self-supervised contrastive pretraining for joint graph and text embeddings [paper]
(arxiv'2023) Prompt tuning on graph-augmented low-resource text classification [paper]
(arxiv'2023) GRENADE: Graph-Centric Language Model for Self-Supervised Representation Learning on Text-Attributed Graphs [paper]
(Nature Machine Intelligence'2023) Multi-modal molecule structure–text model for text-based retrieval and editing [paper]
(arxiv'2023) Pretraining language models with text-attributed heterogeneous graphs [paper]
(arxiv'2022) Learning on large-scale text-attributed graphs via variational inference [paper]

Fusion Training of GNNs and LLMs

(ICLR'2022) GreaseLM: Graph reasoning enhanced language models for question answering [paper]
(arxiv'2023) Disentangled representation learning with large language models for text-attributed graphs [paper]
(arxiv'2024) Efficient Tuning and Inference for Large Language Models on Textual Graphs [paper]
(WWW'2024) Can GNN be Good Adapter for LLMs? [paper]

LLMs Agent for Graphs

(ACL'2023) Don't Generate, Discriminate: A Proposal for Grounding Language Models to Real-World Environments[paper]
(arxiv'2022) Graph Agent: Explicit Reasoning Agent for Graphs [paper]
(arxiv'2024) Middleware for LLMs: Tools Are Instrumental for Language Agents in Complex Environments [paper]
(arxiv'2023) Call Me When Necessary: LLMs can Efficiently and Faithfully Reason over Structured Environments [paper]
(ICLR'2024) Reasoning on graphs: Faithful and interpretable large language model reasoning [paper]

🌐 LLMs-Only

Tuning-free

(NeurIPS'2024) Can language models solve graph problems in natural language? [paper]
(arxiv'2023) GPT4Graph: Can large language models understand graph structured data? an empirical evaluation and benchmarking [paper]
(arxiv'2023) BeyondText:ADeepDiveinto Large Language Models’ Ability on Understanding Graph Data [paper]
(KDD'2024) Exploring the potential of large language models (llms) in learning on graphs [paper]
(arxiv'2023) Graphtext: Graph reasoning in text space [paper]
(arxiv'2023) Talk like a graph: Encoding graphs for large language models [paper]
(arxiv'2023) LLM4DyG:Can Large Language Models Solve Problems on Dynamic Graphs? [paper]
(arxiv'2023) Which Modality should I use–Text, Motif, or Image?: Understanding Graphs with Large Language Models [paper]
(arxiv'2023) When Graph Data Meets Multimodal: A New Paradigm for Graph Understanding and Reasoning [paper]

Tuning-required

(arxiv'2023) Natural language is all a graph needs [paper]
(NeurIPS'2024) Walklm:A uniform language model fine-tuning framework for attributed graph embedding [paper]
(arxiv'2024) LLaGA: Large Language and Graph Assistant [paper]
(arxiv'2024) InstructGraph: Boosting Large Language Models via Graph-centric Instruction Tuning and Preference Alignment [paper]
(arxiv'2024) ZeroG: Investigating Cross-dataset Zero-shot Transferability in Graphs [paper]
(arxiv'2024) GraphWiz: An Instruction-Following Language Model for Graph Problems [paper]
(arxiv'2024) GraphInstruct: Empowering Large Language Models with Graph Understanding and Reasoning Capability [paper]
(arxiv'2024) MuseGraph: Graph-oriented Instruction Tuning of Large Language Models for Generic Graph Mining [paper]

Contributing

If you have come across relevant resources, feel free to submit a pull request.

- (Journal/Confernce'20XX) **paper_name** [[paper](link)]

To add a paper to the survey, please consider providing more detailed information in the PR 😊

GNNs as Prefix
  - (Node-level Tokenization / Graph-level Tokenization)
LLMs as Prefix
  - (Embs. from LLMs for GNNs / Labels from LLMs for GNNs)
LLMs-Graphs Intergration
  - (Alignment between GNNs and LLMs / Fusion Training of GNNs and LLMs / LLMs Agent for Graphs)
LLMs-Only
  - (Tuning-free / Tuning-required)
Please also consider providing a brief introduction about the method to help us quickly add the paper to our survey :)

Acknowledgements

The design of our README.md is inspired by Awesome-LLM-KG and Awesome-LLMs-in-Graph-tasks, thanks to their works!