Awesome

LLM4IR-Survey

This is the collection of papers related to large language models for information retrieval. These papers are organized according to our survey paper Large Language Models for Information Retrieval: A Survey.

Feel free to contact us if you find a mistake or have any advice. Email: yutaozhu94@gmail.com and dou@ruc.edu.cn.

🌟 Citation

Please kindly cite our paper if helps your research:

@article{LLM4IRSurvey,
    author={Yutao Zhu and
            Huaying Yuan and
            Shuting Wang and
            Jiongnan Liu and
            Wenhan Liu and
            Chenlong Deng and
            Haonan Chen and
            Zhicheng Dou and
            Ji-Rong Wen},
    title={Large Language Models for Information Retrieval: A Survey},
    journal={CoRR},
    volume={abs/2308.07107},
    year={2023},
    url={https://arxiv.org/abs/2308.07107},
    eprinttype={arXiv},
    eprint={2308.07107}
}

🚀 Update Log

• Version 3 [2024-09-03]

  • We refine the background to pay more attention to IR.
  • Rewriter: We add a new section "Formats of Rewritten Queries" to provide a more clear classfication and incorporated up-to-date methods.
  • Retriever: We incorporated up-to-date methods that utilize LLM to enlarge the dataset used for training retrievers or to improve the overall structure and design of retriever systems.
  • Reranker: We have added some unsupervised rerankers, several studies focusing on training data augmentation, and discussions on the limitations of LLM rerankers.
  • Reader: We added the latest studies on readers, particularly enriching the works in the active reader section.
  • Search Agent: We added the latest studies on static and dynamic search agents, particularly enriching the works in benchmarking and self-planning.

• Version 2 [2024-01-19]

  • We added a new section to introduce search agents, which represent an innovative approach to integrating LLMs with IR systems.
  • Rewriter: We added recent works on LLM-based query rewriting, most of which focus on conversational search.
  • Retriever: We added the latest techniques that leverage LLMs to expand the training corpus for retrievers or to enhance retrievers' architectures.
  • Reranker: We added recent LLM-based ranking works to each of the three part: Utilizing LLMs as Supervised Rerankers, Utilizing LLMs as Unsupervised Rerankers, and Utilizing LLMs for Training Data Augmentation.
  • Reader: We added the latest studies in LLM-enhanced reader area, including a section introducing the reference compression technique, a section discussing the applications of LLM-enhanced readers, and a section analyzing the characteristics of LLM-enhanced readers.
  • Future Direction: We added a section about search agents and a section discussing the bias caused by leveraging LLMs into IR systems.

📋 Table of Content

• Query Rewriter
  • Prompting Methods
  • Fine-tuning Methods
  • Knowledge Distillation Methods
• Retriever
  • Leveraging LLMs to Generate Search Data
  • Employing LLMs to Enhance Model Architecture
• Re-ranker
  • Utilizing LLMs as Supervised Rerankers
  • Utilizing LLMs as Unsupervised Rerankers
  • Utilizing LLMs for Training Data Augmentation
• Reader
  • Passive Reader
  • Active Reader
  • Compressor
  • Analysis
  • Applications
• Search Agent
  • Static Agent
  • Dynamic Agent
• Other Resources

📄 Paper List

Query Rewriter

Prompting Methods

1. Query2doc: Query Expansion with Large Language Models, Wang et al., arXiv 2023. [Paper]
2. Generative and Pseudo-Relevant Feedback for Sparse, Dense and Learned Sparse Retrieval, Mackie et al., arXiv 2023. [Paper]
3. Generative Relevance Feedback with Large Language Models, Mackie et al., SIGIR 2023 (short paper). [Paper]
4. GRM: Generative Relevance Modeling Using Relevance-Aware Sample Estimation for Document Retrieval, Mackie et al., arXiv 2023. [Paper]
5. Large Language Models Know Your Contextual Search Intent: A Prompting Framework for Conversational Search, Mao et al., arXiv 2023. [Paper]
6. Precise Zero-Shot Dense Retrieval without Relevance Labels, Gao et al., ACL 2023. [Paper]
7. Query Expansion by Prompting Large Language Models, Jagerman et al., arXiv 2023. [Paper]
8. Large Language Models are Strong Zero-Shot Retriever, Shen et al., arXiv 2023. [Paper]
9. Enhancing Conversational Search: Large Language Model-Aided Informative Query Rewriting, Ye et al., EMNLP 2023 (Findings). [Paper]
10. Can generative llms create query variants for test collections? an exploratory study, M. Alaofi et al., SIGIR 2023 (short paper). [Paper]
11. Corpus-Steered Query Expansion with Large Language Models, Lei et al., EACL 2024 (Short Paper). [Paper]
12. Large language model based long-tail query rewriting in taobao search, Peng et al., WWW 2024. [Paper]
13. Can Query Expansion Improve Generalization of Strong Cross-Encoder Rankers?, Li et al., SIGIR 2024. [Paper]
14. Query Performance Prediction using Relevance Judgments Generated by Large Language Models, Meng et al., arXiv 2024. [Paper]
15. RaFe: Ranking Feedback Improves Query Rewriting for RAG, Mao et al., arXiv 2024. [Paper]
16. Crafting the Path: Robust Query Rewriting for Information Retrieval, Baek et al., arXiv 2024. [Paper]
17. Query Rewriting for Retrieval-Augmented Large Language Models, Ma et al., arXiv 2023. [Paper]

Fine-tuning Methods

1. QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation, Srinivasan et al., EMNLP 2022 (Industry). [Paper] (This paper explore fine-tuning methods in baseline experiments.)

Knowledge Distillation Methods

1. QUILL: Query Intent with Large Language Models using Retrieval Augmentation and Multi-stage Distillation, Srinivasan et al., EMNLP 2022 (Industry). [Paper]
2. Knowledge Refinement via Interaction Between Search Engines and Large Language Models, Feng et al., arXiv 2023. [Paper]
3. Query Rewriting for Retrieval-Augmented Large Language Models, Ma et al., arXiv 2023. [Paper]

Retriever

Leveraging LLMs to Generate Search Data

1. InPars: Data Augmentation for Information Retrieval using Large Language Models, Bonifacio et al., arXiv 2022. [Paper]
2. Pre-training with Large Language Model-based Document Expansion for Dense Passage Retrieval, Ma et al., arXiv 2023. [Paper]
3. InPars-v2: Large Language Models as Efficient Dataset Generators for Information Retrieval, Jeronymo et al., arXiv 2023. [Paper]
4. Promptagator: Few-shot Dense Retrieval From 8 Examples, Dai et al., ICLR 2023. [Paper]
5. AugTriever: Unsupervised Dense Retrieval by Scalable Data Augmentation, Meng et al., arXiv 2023. [Paper]
6. UDAPDR: Unsupervised Domain Adaptation via LLM Prompting and Distillation of Rerankers, Saad-Falco et al., arXiv 2023. [Paper]
7. Soft Prompt Tuning for Augmenting Dense Retrieval with Large Language Models, Peng et al., arXiv 2023. [Paper]
8. CONVERSER: Few-shot Conversational Dense Retrieval with Synthetic Data Generation, Huang et al., ACL 2023. [Paper]
9. Leveraging LLMs for Synthesizing Training Data Across Many Languages in Multilingual Dense Retrieval, Thakur et al., arXiv 2023. [Paper]
10. Questions Are All You Need to Train a Dense Passage Retriever, Sachan et al., ACL 2023. [Paper]
11. Beyond Factuality: A Comprehensive Evaluation of Large Language Models as Knowledge Generators, Chen et al., EMNLP 2023. [Paper]
12. Gecko: Versatile Text Embeddings Distilled from Large Language Models, Lee et al., arXiv 2024. [Paper]
13. Improving Text Embeddings with Large Language Models, Wang et al., ACL 2024. [Paper]

Employing LLMs to Enhance Model Architecture

1. Text and Code Embeddings by Contrastive Pre-Training, Neelakantan et al., arXiv 2022. [Paper]
2. Fine-Tuning LLaMA for Multi-Stage Text Retrieval, Ma et al., arXiv 2023. [Paper]
3. Large Dual Encoders Are Generalizable Retrievers, Ni et al., EMNLP 2022. [Paper]
4. Task-aware Retrieval with Instructions, Asai et al., ACL 2023 (Findings). [Paper]
5. Transformer memory as a differentiable search index, Tay et al., NeurIPS 2022. [Paper]
6. Large Language Models are Built-in Autoregressive Search Engines, Ziems et al., ACL 2023 (Findings). [Paper]
7. Chatretriever: Adapting large language models for generalized and robust conversational dense retrieval, Mao et al., arXiv. [Paper]
8. How does generative retrieval scale to millions of passages?, Pradeep et al., ACL 2023. [Paper]
9. CorpusLM: Towards a Unified Language Model on Corpus for Knowledge-Intensive Tasks, Li et al., SIGIR. [Paper]

Reranker

Utilizing LLMs as Supervised Rerankers

1. Multi-Stage Document Ranking with BERT, Nogueira et al., arXiv 2019. [Paper]
2. Document Ranking with a Pretrained Sequence-to-Sequence Model, Nogueira et al., EMNLP 2020 (Findings). [Paper]
3. Text-to-Text Multi-view Learning for Passage Re-ranking, Ju et al., SIGIR 2021 (Short Paper). [Paper]
4. The Expando-Mono-Duo Design Pattern for Text Ranking with Pretrained Sequence-to-Sequence Models, Pradeep et al., arXiv 2021. [Paper]
5. RankT5: Fine-Tuning T5 for Text Ranking with Ranking Losses, Zhuang et al., SIGIR 2023 (Short Paper). [Paper]
6. Fine-Tuning LLaMA for Multi-Stage Text Retrieval, Ma et al., arXiv 2023. [Paper]
7. A Two-Stage Adaptation of Large Language Models for Text Ranking, Zhang et al., ACL 2024 (Findings). [Paper]
8. Rank-without-GPT: Building GPT-Independent Listwise Rerankers on Open-Source Large Language Models, Zhang et al., arXiv 2023. [Paper]
9. ListT5: Listwise Reranking with Fusion-in-Decoder Improves Zero-shot Retrieval, Yoon et al., ACL 2024. [Paper]
10. Q-PEFT: Query-dependent Parameter Efficient Fine-tuning for Text Reranking with Large Language Models, Peng et al., arXiv 2024. [Paper]
11. Leveraging Passage Embeddings for Efficient Listwise Reranking with Large Language Models, Liu et al., arXiv 2024. [Paper]

Utilizing LLMs as Unsupervised Rerankers

1. Holistic Evaluation of Language Models, Liang et al., arXiv 2022. [Paper]
2. Improving Passage Retrieval with Zero-Shot Question Generation, Sachan et al., EMNLP 2022. [Paper]
3. Discrete Prompt Optimization via Constrained Generation for Zero-shot Re-ranker, Cho et al., ACL 2023 (Findings). [Paper]
4. Open-source Large Language Models are Strong Zero-shot Query Likelihood Models for Document Ranking, Zhuang et al., EMNLP 2023 (Findings). [Paper]
5. PaRaDe: Passage Ranking using Demonstrations with Large Language Models, Drozdov et al., EMNLP 2023 (Findings). [Paper]
6. Beyond Yes and No: Improving Zero-Shot LLM Rankers via Scoring Fine-Grained Relevance Labels, Zhuang et al., arXiv 2023. [Paper]
7. Is ChatGPT Good at Search? Investigating Large Language Models as Re-Ranking Agent, Sun et al., EMNLP 2023. [Paper]
8. Zero-Shot Listwise Document Reranking with a Large Language Model, Ma et al., arXiv 2023. [Paper]
9. Found in the Middle: Permutation Self-Consistency Improves Listwise Ranking in Large Language Models, Tang et al., arXiv 2023. [Paper]
10. Large Language Models are Effective Text Rankers with Pairwise Ranking Prompting, Qin et al., NAACL 2024 (Findings). [Paper]
11. A Setwise Approach for Effective and Highly Efficient Zero-shot Ranking with Large Language Models, Zhuang et al., SIGIR 2024. [Paper]
12. InstUPR: Instruction-based Unsupervised Passage Reranking with Large Language Models, Huang and Chen, arXiv 2024. [Paper]
13. Generating Diverse Criteria On-the-Fly to Improve Point-wise LLM Rankers, Guo et al., arXiv 2024. [Paper]
14. DemoRank: Selecting Effective Demonstrations for Large Language Models in Ranking Task, Liu et al., arXiv 2024. [Paper]
15. An Investigation of Prompt Variations for Zero-shot LLM-based Rankers, Sun et al., arXiv 2024. [Paper]
16. TourRank: Utilizing Large Language Models for Documents Ranking with a Tournament-Inspired Strategy, Chen et al., arXiv 2024. [Paper]
17. Top-Down Partitioning for Efficient List-Wise Ranking, Parry et al., arXiv 2024. [Paper]
18. PRP-Graph: Pairwise Ranking Prompting to LLMs with Graph Aggregation for Effective Text Re-ranking, Luo et al., ACL 2024. [Paper]
19. Consolidating Ranking and Relevance Predictions of Large Language Models through Post-Processing, Yan et al., arXiv 2024. [Paper]

Utilizing LLMs for Training Data Augmentation

1. ExaRanker: Explanation-Augmented Neural Ranker, Ferraretto et al., SIGIR 2023 (Short Paper). [Paper]
2. InPars-Light: Cost-Effective Unsupervised Training of Efficient Rankers, Boytsov et al., arXiv 2023. [Paper]
3. Generating Synthetic Documents for Cross-Encoder Re-Rankers, Askari et al., arXiv 2023. [Paper]
4. Instruction Distillation Makes Large Language Models Efficient Zero-shot Rankers, Sun et al., arXiv 2023. [Paper]
5. RankVicuna: Zero-Shot Listwise Document Reranking with Open-Source Large Language Models, Pradeep et al., arXiv 2023. [Paper]
6. RankZephyr: Effective and Robust Zero-Shot Listwise Reranking is a Breeze!, Pradeep et al., arXiv 2023. [Paper]
7. ExaRanker-Open: Synthetic Explanation for IR using Open-Source LLMs, Ferraretto et al., arXiv 2024. [Paper]
8. Expand, Highlight, Generate: RL-driven Document Generation for Passage Reranking, Askari et al., EMNLP 2023. [Paper]
9. FIRST: Faster Improved Listwise Reranking with Single Token Decoding, Reddy et al., arXiv 2024. [Paper]

Reader

Passive Reader

1. REALM: Retrieval-Augmented Language Model Pre-Training, Guu et al., ICML 2020. [Paper]
2. Retrieval-Augmented Generation for Knowledge-Intensive NLP Tasks, Lewis et al., NeurIPS 2020. [Paper]
3. REPLUG: Retrieval-Augmented Black-Box Language Models, Shi et al., arXiv 2023. [Paper]
4. Atlas: Few-shot Learning with Retrieval Augmented Language Models, Izacard et al., JMLR 2023. [Paper]
5. Internet-augmented Language Models through Few-shot Prompting for Open-domain Question Answering, Lazaridou et al., arXiv 2022. [Paper]
6. Rethinking with Retrieval: Faithful Large Language Model Inference, He et al., arXiv 2023. [Paper]
7. FreshLLMs: Refreshing Large Language Models with Search Engine Augmentation, Vu et al., arxiv 2023. [Paper]
8. Enabling Large Language Models to Generate Text with Citations, Gao et al., EMNLP 2023. [Paper]
9. Chain-of-Note: Enhancing Robustness in Retrieval-Augmented Language Models, Yu et al., arxiv 2023. [Paper]
10. Improving Retrieval-Augmented Large Language Models via Data Importance Learning, Lyu et al., arXiv 2023. [Paper]
11. Search Augmented Instruction Learning, Luo et al., EMNLP 2023 (Findings). [Paper]
12. RADIT: Retrieval-Augmented Dual Instruction Tuning, Lin et al., arXiv 2023. [Paper]
13. Improving Language Models by Retrieving from Trillions of Tokens, Borgeaud et al., ICML 2022. [Paper]
14. In-Context Retrieval-Augmented Language Models, Ram et al., arXiv 2023. [Paper]
15. Interleaving Retrieval with Chain-of-thought Reasoning for Knowledge-intensive Multi-step Questions, Trivedi et al., ACL 2023. [Paper]
16. Improving Language Models via Plug-and-Play Retrieval Feedback, Yu et al., arXiv 2023. [Paper]
17. Enhancing Retrieval-Augmented Large Language Models with Iterative Retrieval-Generation Synergy, Shao et al., EMNLP 2023 (Findings). [Paper]
18. Retrieval-Generation Synergy Augmented Large Language Models, Feng et al., arXiv 2023. [Paper]
19. Self-RAG: Learning to Retrieve, Generate, and Critique through Self-Reflection, Asai et al., arXiv 2023. [Paper]
20. Active Retrieval Augmented Generation, Jiang et al., EMNLP 2023. [Paper]

Active Reader

1. Measuring and Narrowing the Compositionality Gap in Language Models, Press et al., arXiv 2022. [Paper]
2. DEMONSTRATE–SEARCH–PREDICT: Composing Retrieval and Language Models for Knowledge-intensive NLP, Khattab et al., arXiv 2022. [Paper]
3. Answering Questions by Meta-Reasoning over Multiple Chains of Thought, Yoran et al., arXiv 2023. [Paper]
4. PlanRAG: A Plan-then-Retrieval Augmented Generation for Generative Large Language Models as Decision Makers, Lee ei al., arXiv 2024. [Paper]
5. Learning to Plan for Retrieval-Augmented Large Language Models from Knowledge Graphs, Wang et al., arXiv 2024. [Paper]

Compressor

1. LeanContext: Cost-Efficient Domain-Specific Question Answering Using LLMs, Arefeen et al., arXiv 2023. [Paper]
2. RECOMP: Improving Retrieval-Augmented LMs with Compression and Selective Augmentation, Xu et al., arXiv 2023. [Paper]
3. TCRA-LLM: Token Compression Retrieval Augmented Large Language Model for Inference Cost Reduction, Liu et al., EMNLP 2023 (Findings). [Paper]
4. Learning to Filter Context for Retrieval-Augmented Generation, Wang et al., arXiv 2023. [Paper]

Analysis

1. Lost in the Middle: How Language Models Use Long Contexts, Liu et al., arXiv 2023. [Paper]
2. Investigating the Factual Knowledge Boundary of Large Language Models with Retrieval Augmentation, Ren et al., arXiv 2023. [Paper]
3. Exploring the Integration Strategies of Retriever and Large Language Models, Liu et al., arXiv 2023. [Paper]
4. Characterizing Attribution and Fluency Tradeoffs for Retrieval-Augmented Large Language Models, Aksitov et al., arXiv 2023. [Paper]
5. When Not to Trust Language Models: Investigating Effectiveness of Parametric and Non-Parametric Memories, Mallen et al., ACL 2023. [Paper]

Applications

1. Augmenting Black-box LLMs with Medical Textbooks for Clinical Question Answering, Wang et al., arXiv 2023. [Paper]
2. ATLANTIC: Structure-Aware Retrieval-Augmented Language Model for Interdisciplinary Science, Munikoti et al., arXiv 2023. [Paper]
3. Crosslingual Retrieval Augmented In-context Learning for Bangla, Li et al., arXiv 2023. [Paper]
4. Clinfo.ai: An Open-Source Retrieval-Augmented Large Language Model System for Answering Medical Questions using Scientific Literature, Lozano et al., arXiv 2023. [Paper]
5. Enhancing Financial Sentiment Analysis via Retrieval Augmented Large Language Models, Zhang et al., ICAIF 2023. [Paper]
6. Interpretable Long-Form Legal Question Answering with Retrieval-Augmented Large Language Models, Louis et al., arXiv 2023. [Paper]
7. RETA-LLM: A Retrieval-Augmented Large Language Model Toolkit, Liu et al., arXiv 2023. [Paper]
8. Chameleon: a Heterogeneous and Disaggregated Accelerator System for Retrieval-Augmented Language Models, Jiang et al., arXiv 2023. [Paper]
9. RaLLe: A Framework for Developing and Evaluating Retrieval-Augmented Large Language Models, Hoshi et al., EMNLP 2023. [Paper]
10. Don't forget private retrieval: distributed private similarity search for large language models, Zyskind et al., arXiv 2023. [Paper]

Search Agent

Static Agent

1. LaMDA: Language Models for Dialog Applications, Thoppilan et al., arXiv 2022. [Paper]
2. Language Models that Seek for Knowledge: Modular Search & Generation for Dialogue and Prompt Completion, Shuster et al., EMNLP 2022 (Findings). [Paper]
3. Teaching language models to support answers with verified quotes, Menick et al., arXiv 2022. [Paper]
4. WebGLM: Towards An Efficient Web-Enhanced Question Answering System with Human Preferences, Liu et al., KDD 2023. [Paper]
5. A Real-World WebAgent with Planning, Long Context Understanding, and Program Synthesis, Gur et al., arXiv 2023. [Paper]
6. Know Where to Go: Make LLM a Relevant, Responsible, and Trustworthy Searcher, Shi et al., arXiv 2023. [Paper]
7. CoSearchAgent: A Lightweight Collaborative Search Agent with Large Language Models, Gong et al., SIGIR 2024. [Paper]
8. TRAD: Enhancing LLM Agents with Step-Wise Thought Retrieval and Aligned Decision, Zhou et al., SIGIR 2024. [Paper]

Dynamic Agent

1. WebGPT: Browser-assisted question-answering with human feedback, Nakano et al., arXiv 2021. [Paper]
2. WebShop: Towards Scalable Real-World Web Interaction with Grounded Language Agents, Yao et al., arXiv 2022. [Paper]
3. WebCPM: Interactive Web Search for Chinese Long-form Question Answering, Qin et al., ACL 2023. [Paper]
4. Mind2Web: Towards a Generalist Agent for the Web, Deng et al., arXiv 2023. [Paper]
5. WebArena: A Realistic Web Environment for Building Autonomous Agents, Zhou et al., arXiv 2023. [Paper]
6. Hierarchical Prompting Assists Large Language Model on Web Navigation, Sridhar et al., EMNLP 2023 (Findings). [Paper]
7. KwaiAgents: Generalized Information-seeking Agent System with Large Language Models, Pan et al., arXiv 2023. [Paper]
8. WebVoyager : Building an End-to-End Web Agent with Large Multimodal Models, He et al., arXiv 2024. [Paper]
9. AutoWebGLM: Bootstrap And Reinforce A Large Language Model-based Web Navigating Agent, Lai et al., KDD 2024. [Paper]
10. WebCanvas: Benchmarking Web Agents in Online Environments, Pan et al., arXiv 2024. [Paper]
11. Internet of Agents: Weaving a Web of Heterogeneous Agents for Collaborative Intelligence, Chen et al., arXiv 2024. [Paper]
12. Agent-E: From Autonomous Web Navigation to Foundational Design Principles in Agentic Systems, Abuelsaad et al., arXiv 2024. [Paper]
13. MindSearch: Mimicking Human Minds Elicits Deep AI Searcher, Chen et al., arXiv 2024. [Paper]

Other Resources

1. ACL 2023 Tutorial: Retrieval-based Language Models and Applications, Asai et al., ACL 2023. [Link]
2. A Survey of Large Language Models, Zhao et al., arXiv 2023. [Paper]
3. Information Retrieval Meets Large Language Models: A Strategic Report from Chinese IR Community, Ai et al., arXiv 2023. [Paper]