Awesome

ReasoningNLP

Paper list on reasoning in NLP. Here we mainly collect papers about datasets and methods using PLMs (under the progress).

See our survey on natural language reasoning:

• Natural Language Reasoning, A Survey

Here are anohter two related surveys on LLM prompting:

• Reasoning with Language Model Prompting: A Survey | resources
• Towards Reasoning in Large Language Models: A Survey | resources

• Methodology
  • Reasoning Paradigm
    • End-to-End Reasoning
    • Forward Reasoning
    • Backward Reasoning
  • Learning Paradigm
    • Finetuning
    • In-Context Learning
• Natural Language Reasoning
  • Classical Logical Reasoning
    • Datasets & Benchmarks
    • Related Works
  • Natural Language Inference
    • Datasets & Benchmarks
    • Related Works
  • Multi-hop Question Answering
    • Datasets & Benchmarks
    • Related Works
  • Commonsense Reasoning
    • Datasets & Benchmarks
    • Related Works
    • Knowledge Bases
  • Complex Reasoning
    • Datasets & Benchmarks
• Knowledge Graph Reasoning
• Mathematical Reasoning

1. Show Your Work: Scratchpads for Intermediate Computation with Language Models arXiv (2021)

   Maxwell Nye, Anders Johan Andreassen, Guy Gur-Ari, Henryk Michalewski, Jacob Austin, David Bieber, David Dohan, Aitor Lewkowycz, Maarten Bosma, David Luan, Charles Sutton, Augustus Odena [pdf]

2. Chain of Thought Prompting Elicits Reasoning in Large Language Models arXiv (2022)

   Jason Wei, Xuezhi Wang, Dale Schuurmans, Maarten Bosma, Brian Ichter, Fei Xia, Ed Chi, Quoc Le, Denny Zhou [pdf] [project]

3. Self-Consistency Improves Chain of Thought Reasoning in Language Models arXiv (2022)

   Xuezhi Wang, Jason Wei, Dale Schuurmans, Quoc Le, Ed Chi, Sharan Narang, Aakanksha Chowdhery, Denny Zhou [pdf]

4. STaR: Bootstrapping Reasoning With Reasoning arXiv (2022)

   Eric Zelikman, Yuhuai Wu, Jesse Mu, Noah D. Goodman [pdf]

5. Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning arXiv (2022)

   Antonia Creswell, Murray Shanahan, Irina Higgins [pdf]

6. Least-to-Most Prompting Enables Complex Reasoning in Large Language Models arXiv (2022)

   Denny Zhou, Nathanael Schärli, Le Hou, Jason Wei, Nathan Scales, Xuezhi Wang, Dale Schuurmans, Olivier Bousquet, Quoc Le, Ed Chi [pdf]

7. Large Language Models are Zero-Shot Reasoners arXiv (2022)

   Takeshi Kojima, Shixiang Shane Gu, Machel Reid, Yutaka Matsuo, Yusuke Iwasawa [pdf] [project]

8. Language models show human-like content effects on reasoning arXiv (2022)

   Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill [pdf]

9. Language Model Cascades arXiv (2022)

   David Dohan, Winnie Xu, Aitor Lewkowycz, Jacob Austin, David Bieber, Raphael Gontijo Lopes, Yuhuai Wu, Henryk Michalewski, Rif A. Saurous, Jascha Sohl-dickstein, Kevin Murphy, Charles Sutton [pdf] [project]

10. Faithful Reasoning Using Large Language Models arXiv (2022)

    Antonia Creswell, Murray Shanahan [pdf]

11. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought arXiv (2022)

    Abulhair Saparov, He He [pdf] [project]

12. ThinkSum: Probabilistic reasoning over sets using large language models arXiv (2022)

    Batu Ozturkler, Nikolay Malkin, Zhen Wang, Nebojsa Jojic [pdf]

13. Measuring and Narrowing the Compositionality Gap in Language Models arXiv (2022)

    Ofir Press, Muru Zhang, Sewon Min, Ludwig Schmidt, Noah A. Smith, Mike Lewis [pdf] [project]

Some datasets explicitly target at philosophical reasoning types, e.g. deduction, abduction and induction. Thus, we call them as ``classical logical reasoning tasks''. A key characteristic of this topic is that tasks are mostly artificial to study reasoning.

Papers on dataset artifacts:

1. On the Paradox of Learning to Reason from Data arXiv (2022)

   Honghua Zhang, Liunian Harold Li, Tao Meng, Kai-Wei Chang, Guy Van den Broeck [pdf] [project]

1. Transformers as Soft Reasoners over Language IJCAI (2020)

   Peter Clark, Oyvind Tafjord, Kyle Richardson [pdf] [project]

2. PRover: Proof Generation for Interpretable Reasoning over Rules EMNLP (2020)

   Swarnadeep Saha, Sayan Ghosh, Shashank Srivastava, Mohit Bansal [pdf] [project]

3. multiPRover: Generating Multiple Proofs for Improved Interpretability in Rule Reasoning NAACL (2021)

   Swarnadeep Saha, Prateek Yadav, Mohit Bansal [pdf] [project]

4. Critical Thinking for Language Models IWCS (2021)

   Gregor Betz, Christian Voigt, Kyle Richardson [pdf] [project]

5. Explainable Multi-hop Verbal Reasoning Through Internal Monologue NAACL (2021)

   Zhengzhong Liang, Steven Bethard, Mihai Surdeanu [pdf] [project]

6. ProofWriter: Generating Implications, Proofs, and Abductive Statements over Natural Language ACL findings (2021)

   Oyvind Tafjord, Bhavana Dalvi, Peter Clark [pdf] [project]

7. Flexible Generation of Natural Language Deductions EMNLP (2021)

   Kaj Bostrom, Xinyu Zhao, Swarat Chaudhuri, Greg Durrett [pdf] [project]

8. FaiRR: Faithful and Robust Deductive Reasoning over Natural Language ACL (2022)

   Soumya Sanyal, Harman Singh, Xiang Ren [pdf] [project]

9. Interpretable Proof Generation via Iterative Backward Reasoning NAACL (2022)

   Hanhao Qu, Yu Cao, Jun Gao, Liang Ding, Ruifeng Xu [pdf] [project]

10. Selection-Inference: Exploiting Large Language Models for Interpretable Logical Reasoning arXiv (2022)

    Antonia Creswell, Murray Shanahan, Irina Higgins [pdf]

11. Generating Natural Language Proofs with Verifier-Guided Search arXiv (2022)

    Kaiyu Yang, Jia Deng, Danqi Chen [pdf] [project]

12. ROBUSTLR: A Diagnostic Benchmark for Evaluating Logical Robustness of Deductive Reasoners arXiv (2022)

    Soumya Sanyal, Zeyi Liao, Xiang Ren [pdf] [project]

13. Language models show human-like content effects on reasoning arXiv (2022)

    Ishita Dasgupta, Andrew K. Lampinen, Stephanie C. Y. Chan, Antonia Creswell, Dharshan Kumaran, James L. McClelland, Felix Hill [pdf]

14. Faithful Reasoning Using Large Language Models arXiv (2022)

    Antonia Creswell, Murray Shanahan [pdf]

15. Language Models Are Greedy Reasoners: A Systematic Formal Analysis of Chain-of-Thought arXiv (2022)

    Abulhair Saparov, He He [pdf] [project]

16. LAMBADA: Backward Chaining for Automated Reasoning in Natural Language arXiv (2022)

    Seyed Mehran Kazemi, Najoung Kim, Deepti Bhatia, Xin Xu, Deepak Ramachandran [pdf]

Defeasible reasoning:

1. Learning to Rationalize for Nonmonotonic Reasoning with Distant Supervision AAAI (2020)

   Faeze Brahman, Vered Shwartz, Rachel Rudinger, Yejin Choi [pdf] [project]

2. Could you give me a hint ? Generating inference graphs for defeasible reasoning ACL findings (2021)

   Aman Madaan, Dheeraj Rajagopal, Niket Tandon, Yiming Yang, Eduard H. Hovy [pdf] [project]

3. Think about it! Improving defeasible reasoning by first modeling the question scenario EMNLP (2021)

   Aman Madaan, Niket Tandon, Dheeraj Rajagopal, Peter Clark, Yiming Yang, Eduard H. Hovy [pdf] [project]

4. Language Models as Inductive Reasoners arXiv (2022)

   Zonglin Yang, Li Dong, Xinya Du, Hao Cheng, Erik Cambria, Xiaodong Liu, Jianfeng Gao, Furu Wei [pdf]

Task: given a premise-hypothesis pair, classify it into three classes: entailment, contradiction and neutral.

There are mainly three types of premise-hypothesis pairs in NLI task: paraphrase, compound semantics understanding (CSU), and reasoning. Here we just consider the last one.

Recently, some datasets are proposed to model different subjective opinions (on classifying into which class) of crowdworkers.

Some datasets for other language:

Papers on dataset artifacts:

1. Performance Impact Caused by Hidden Bias of Training Data for Recognizing Textual Entailment LREC (2018)

   Masatoshi Tsuchiya [pdf]

2. Annotation Artifacts in Natural Language Inference Data NAACL (2018)

   Suchin Gururangan, Swabha Swayamdipta, Omer Levy, Roy Schwartz, Samuel R. Bowman, Noah A. Smith [pdf]

3. Hypothesis Only Baselines in Natural Language Inference SEM (2019)

   Adam Poliak, Jason Naradowsky, Aparajita Haldar, Rachel Rudinger, Benjamin Van Durme [pdf]

1. NILE : Natural Language Inference with Faithful Natural Language Explanations ACL (2020)

   Sawan Kumar, Partha P. Talukdar [pdf] [project]

2. Identifying inherent disagreement in natural language inference NAACL (2021)

   Xinliang Frederick Zhang, Marie-Catherine de Marneffe [pdf] [project]

3. KACE: Generating Knowledge Aware Contrastive Explanations for Natural Language Inference ACL (2021)

   Qianglong Chen, Feng Ji, Xiangji Zeng, Feng-Lin Li, Ji Zhang, Haiqing Chen, Yin Zhang [pdf] [project]

4. Investigating Transfer Learning in Multilingual Pre-trained Language Models through Chinese Natural Language Inference ACL findings (2021)

   Hai Hu, He Zhou, Zuoyu Tian, Yiwen Zhang, Yina Patterson, Yanting Li, Yixin Nie, Kyle Richardson [pdf] [project]

5. Enhancing Cross-lingual Natural Language Inference by Prompt-learning from Cross-lingual Templates ACL (2022)

   Kunxun Qi, Hai Wan, Jianfeng Du, Haolan Chen [pdf] [project]

6. Generating Intermediate Steps for NLI with Next-Step Supervision arXiv (2022)

   Deepanway Ghosal, Somak Aditya, Monojit Choudhury [pdf]

This topic studies answering the complex questions that require to reason over evidences scattered in different contexts. The term ``hop'' here indicates the number of contexts required to reason. There are two settings on the required contexts: (1) all provided along with some distractors (i.e. distractor), (2) need to be retrieved (i.e. retrieval).

Some datasets annotate the ground supporting evidences (paragraph-level, sentence-level, or triple-level), decomposed sub-questions (and the corresponding evidences), or reasoning paths.

Papers on dataset artifacts:

1. Understanding Dataset Design Choices for Multi-hop Reasoning NAACL (2019)

   Jifan Chen, Greg Durrett [pdf]

2. Avoiding Reasoning Shortcuts: Adversarial Evaluation, Training, and Model Development for Multi-Hop QA ACL (2019)

   Yichen Jiang, Mohit Bansal [pdf] [project]

3. Compositional Questions Do Not Necessitate Multi-hop Reasoning ACL (2019)

   Sewon Min, Eric Wallace, Sameer Singh, Matt Gardner, Hannaneh Hajishirzi, Luke Zettlemoyer [pdf] [project]

4. Is Multihop QA in DiRe Condition? Measuring and Reducing Disconnected Reasoning EMNLP (2020)

   Harsh Trivedi, Niranjan Balasubramanian, Tushar Khot, Ashish Sabharwal [pdf] [project]

1. Dynamically Fused Graph Network for Multi-hop Reasoning ACL (2019)

   Lin Qiu, Yunxuan Xiao, Yanru Qu, Hao Zhou, Lei Li, Weinan Zhang, Yong Yu [pdf] [project]

2. Multi-hop Reading Comprehension across Multiple Documents by Reasoning over Heterogeneous Graphs ACL (2019)

   Ming Tu, Guangtao Wang, Jing Huang, Yun Tang, Xiaodong He, Bowen Zhou [pdf]

3. Answering while Summarizing: Multi-task Learning for Multi-hop QA with Evidence Extraction ACL (2019)

   Kosuke Nishida, Kyosuke Nishida, Masaaki Nagata, Atsushi Otsuka, Itsumi Saito, Hisako Asano, Junji Tomita [pdf]

4. Multi-hop Reading Comprehension through Question Decomposition and Rescoring ACL (2019)

   Sewon Min, Victor Zhong, Luke Zettlemoyer, Hannaneh Hajishirzi [pdf] [project]

5. Differentiable Reasoning over a Virtual Knowledge Base ICLR (2020)

   Bhuwan Dhingra, Manzil Zaheer, Vidhisha Balachandran, Graham Neubig, Ruslan Salakhutdinov, William W. Cohen [pdf] [project]

6. Transformer-XH: Multi-Evidence Reasoning with eXtra Hop Attention ICLR Poster (2020)

   Chen Zhao, Chenyan Xiong, Corby Rosset, Xia Song, Paul N. Bennett, Saurabh Tiwary [pdf] [project]

7. Low-Resource Generation of Multi-hop Reasoning Questions ACL (2020)

   Jianxing Yu, Wei Liu, Shuang Qiu, Qinliang Su, Kai Wang, Xiaojun Quan, Jian Yin [pdf]

8. SRLGRN: Semantic Role Labeling Graph Reasoning Network EMNLP (2020)

   Chen Zheng, Parisa Kordjamshidi [pdf]

9. Learning to Retrieve Reasoning Paths over Wikipedia Graph for Question Answering ICLR (2020)

   Akari Asai, Kazuma Hashimoto, Hannaneh Hajishirzi, Richard Socher, Caiming Xiong [pdf] [project]

10. Robustifying Multi-hop QA through Pseudo-Evidentiality Training ACL (2021)

    Kyungjae Lee, Seung-won Hwang, Sang-eun Han, Dohyeon Lee [pdf]

11. Summarize-then-Answer: Generating Concise Explanations for Multi-hop Reading Comprehension EMNLP (2021)

    Naoya Inoue, Harsh Trivedi, Steven Sinha, Niranjan Balasubramanian, Kentaro Inui [pdf] [project]

12. Generative Context Pair Selection for Multi-hop Question Answering EMNLP (2021)

    Dheeru Dua, Cícero Nogueira dos Santos, Patrick Ng, Ben Athiwaratkun, Bing Xiang, Matt Gardner, Sameer Singh [pdf] [project]

13. Breadth First Reasoning Graph for Multi-hop Question Answering NAACL (2021)

    Yongjie Huang, Meng Yang [pdf]

14. Answering Complex Open-Domain Questions with Multi-Hop Dense Retrieval ICLR Poster (2021)

    Wenhan Xiong, Xiang Lorraine Li, Srini Iyer, Jingfei Du, Patrick S. H. Lewis, William Yang Wang, Yashar Mehdad, Scott Yih, Sebastian Riedel, Douwe Kiela, Barlas Oguz [pdf] [project]

15. Multi-Step Reasoning Over Unstructured Text with Beam Dense Retrieval NAACL (2021)

    Chen Zhao, Chenyan Xiong, Jordan L. Boyd-Graber, Hal Daumé III [pdf] [project]

16. Unsupervised Multi-hop Question Answering by Question Generation NAACL (2021)

    Liangming Pan, Wenhu Chen, Wenhan Xiong, Min-Yen Kan, William Yang Wang [pdf] [project]

17. If You Want to Go Far Go Together: Unsupervised Joint Candidate Evidence Retrieval for Multi-hop Question Answering NAACL (2021)

    Vikas Yadav, Steven Bethard, Mihai Surdeanu [pdf] [project]

18. Baleen: Robust Multi-Hop Reasoning at Scale via Condensed Retrieval NIPS (2021)

    Omar Khattab, Christopher Potts, Matei A. Zaharia [pdf] [project]

19. Modeling Multi-hop Question Answering as Single Sequence Prediction ACL (2022)

    Semih Yavuz, Kazuma Hashimoto, Yingbo Zhou, Nitish Shirish Keskar, Caiming Xiong [pdf]

20. CQG: A Simple and Effective Controlled Generation Framework for Multi-hop Question Generation ACL (2022)

    Zichu Fei, Qi Zhang, Tao Gui, Di Liang, Sirui Wang, Wei Wu, Xuanjing Huang [pdf] [project]

Commonsense reasoning deals with implicit commonsense knowledge, which may be non-trivial to machines since they are difficult to retrieve from the web due to reporting bias. While it is named with "reasoning", the common theme of this topic is commonsense knowledge rather than reasoning. Here, we only list reasoning datasets.

There are mainly three types of reasoning datasets towards "what" (i.e. assertions or events), "what if / why" (e.g. causal and temporal relations between events), and "how" (i.e. actions) respectively.

"What" commonsense reasoning require combining multiple pieces of knowledge that some are from external knowledge sources while others are commonsense knowledge.

"What if / Why" commonsense reasoning often reasons for causal and temporal relations between events. There are two causal relations: causes and effects, which can be seen as backward causal reasoning and forward causal reasoning respectively.

"How" commonsense reasoning is mainly about "how to do it". It is often related to problem-solving or decision-making.

Some datasets involve multiple types of reasoning.

1. Attention Is (not) All You Need for Commonsense Reasoning ACL (2019)

   Tassilo Klein, Moin Nabi [pdf]

2. COMET: Commonsense Transformers for Automatic Knowledge Graph Construction ACL (2019)

   Antoine Bosselut, Hannah Rashkin, Maarten Sap, Chaitanya Malaviya, Asli Celikyilmaz, Yejin Choi [pdf] [project]

3. Explain Yourself! Leveraging Language Models for Commonsense Reasoning ACL (2019)

   Nazneen Fatema Rajani, Bryan McCann, Caiming Xiong, Richard Socher [pdf] [project]

4. Commonsense Knowledge Mining from Pretrained Models EMNLP (2019)

   Joe Davison, Joshua Feldman, Alexander M. Rush [pdf]

5. How Reasonable are Common-Sense Reasoning Tasks: A Case-Study on the Winograd Schema Challenge and SWAG EMNLP (2019)

   Paul Trichelair, Ali Emami, Adam Trischler, Kaheer Suleman, Jackie Chi Kit Cheung [pdf] [project]

6. Guided Generation of Cause and Effect IJCAI (2020)

   Zhongyang Li, Xiao Ding, Ting Liu, J. Edward Hu, Benjamin Van Durme [pdf] [project]

7. Contrastive Self-Supervised Learning for Commonsense Reasoning ACL (2020)

   Tassilo Klein, Moin Nabi [pdf] [project]

8. Pre-training Is (Almost) All You Need: An Application to Commonsense Reasoning ACL (2020)

   Alexandre Tamborrino, Nicola Pellicanò, Baptiste Pannier, Pascal Voitot, Louise Naudin [pdf]

9. Evidence-Aware Inferential Text Generation with Vector Quantised Variational AutoEncoder ACL (2020)

   Daya Guo, Duyu Tang, Nan Duan, Jian Yin, Daxin Jiang, Ming Zhou [pdf] [project]

10. Scalable Multi-Hop Relational Reasoning for Knowledge-Aware Question Answering EMNLP (2020)

    Yanlin Feng, Xinyue Chen, Bill Yuchen Lin, Peifeng Wang, Jun Yan, Xiang Ren [pdf] [project]

11. Back to the Future: Unsupervised Backprop-based Decoding for Counterfactual and Abductive Commonsense Reasoning EMNLP (2020)

    Lianhui Qin, Vered Shwartz, Peter West, Chandra Bhagavatula, Jena D. Hwang, Ronan Le Bras, Antoine Bosselut, Yejin Choi [pdf] [project]

12. Self-Supervised Knowledge Triplet Learning for Zero-Shot Question Answering EMNLP (2020)

    Pratyay Banerjee, Chitta Baral [pdf]

13. Unsupervised Commonsense Question Answering with Self-Talk EMNLP (2020)

    Vered Shwartz, Peter West, Ronan Le Bras, Chandra Bhagavatula, Yejin Choi [pdf]

14. Paragraph-level Commonsense Transformers with Recurrent Memory AAAI (2021)

    Saadia Gabriel, Chandra Bhagavatula, Vered Shwartz, Ronan Le Bras, Maxwell Forbes, Yejin Choi [pdf] [project]

15. Knowledge-driven Data Construction for Zero-shot Evaluation in Commonsense Question Answering AAAI (2021)

    Kaixin Ma, Filip Ilievski, Jonathan Francis, Yonatan Bisk, Eric Nyberg, Alessandro Oltramari [pdf] [project]

16. Reflective Decoding: Beyond Unidirectional Generation with Off-the-Shelf Language Models ACL (2021)

    Peter West, Ximing Lu, Ari Holtzman, Chandra Bhagavatula, Jena D. Hwang, Yejin Choi [pdf] [project]

17. Doing Good or Doing Right? Exploring the Weakness of Commonsense Causal Reasoning Models ACL (2021)

    Mingyue Han, Yinglin Wang [pdf]

18. Learning Event Graph Knowledge for Abductive Reasoning ACL (2021)

    Li Du, Xiao Ding, Ting Liu, Bing Qin [pdf] [project]

19. ExCAR: Event Graph Knowledge Enhanced Explainable Causal Reasoning ACL (2021)

    Li Du, Xiao Ding, Kai Xiong, Ting Liu, Bing Qin [pdf] [project]

20. Differentiable Open-Ended Commonsense Reasoning NAACL (2021)

    Bill Yuchen Lin, Haitian Sun, Bhuwan Dhingra, Manzil Zaheer, Xiang Ren, William W. Cohen [pdf] [project]

21. QA-GNN: Reasoning with Language Models and Knowledge Graphs for Question Answering NAACL (2021)

    Michihiro Yasunaga, Hongyu Ren, Antoine Bosselut, Percy Liang, Jure Leskovec [pdf] [project]

22. Towards Zero-shot Commonsense Reasoning with Self-supervised Refinement of Language Models EMNLP (2021)

    Tassilo Klein, Moin Nabi [pdf] [project]

23. Exploring Strategies for Generalizable Commonsense Reasoning with Pre-trained Models EMNLP (2021)

    Kaixin Ma, Filip Ilievski, Jonathan Francis, Satoru Ozaki, Eric Nyberg, Alessandro Oltramari [pdf] [project]

24. Shortcutted Commonsense: Data Spuriousness in Deep Learning of Commonsense Reasoning EMNLP (2021)

    Ruben Branco, António Branco, João António Rodrigues, João Ricardo Silva [pdf] [project]

25. Improving Unsupervised Commonsense Reasoning Using Knowledge-Enabled Natural Language Inference EMNLP findings (2021)

    Canming Huang, Weinan He, Yongmei Liu [pdf] [project]

26. SalKG: Learning From Knowledge Graph Explanations for Commonsense Reasoning NIPS (2021)

    Aaron Chan, Jiashu Xu, Boyuan Long, Soumya Sanyal, Tanishq Gupta, Xiang Ren [pdf] [project]

27. GreaseLM: Graph REASoning Enhanced Language Models ICLR Spotlight (2022)

    Xikun Zhang, Antoine Bosselut, Michihiro Yasunaga, Hongyu Ren, Percy Liang, Christopher D. Manning, Jure Leskovec [pdf] [project]

28. Generated Knowledge Prompting for Commonsense Reasoning ACL (2022)

    Jiacheng Liu, Alisa Liu, Ximing Lu, Sean Welleck, Peter West, Ronan Le Bras, Yejin Choi, Hannaneh Hajishirzi [pdf] [project]

29. JointLK: Joint Reasoning with Language Models and Knowledge Graphs for Commonsense Question Answering NAACL (2022)

    Yueqing Sun, Qi Shi, Le Qi, Yu Zhang [pdf] [project]

30. Modularized Transfer Learning with Multiple Knowledge Graphs for Zero-shot Commonsense Reasoning NAACL (2022)

    Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, Jinyoung Yeo [pdf]

31. On Curriculum Learning for Commonsense Reasoning NAACL (2022)

    Yu Jin Kim, Beong-woo Kwak, Youngwook Kim, Reinald Kim Amplayo, Seung-won Hwang, Jinyoung Yeo [pdf] [project]

32. Embarrassingly Simple Performance Prediction for Abductive Natural Language Inference NAACL (2022)

    Emils Kadikis, Vaibhav Srivastav, Roman Klinger [pdf] [project]

33. Does Pre-training Induce Systematic Inference? How Masked Language Models Acquire Commonsense Knowledge NAACL (2022)

    Ian Porada, Alessandro Sordoni, Jackie Chi Kit Cheung [pdf]

34. ROCK: Causal Inference Principles for Reasoning about Commonsense Causality ICML (2022)

    Jiayao Zhang, Hongming Zhang, Weijie J. Su, Dan Roth [pdf] [project]

35. ALERT: Adapting Language Models to Reasoning Tasks arXiv (2022)

    Ping Yu, Tianlu Wang, Olga Golovneva, Badr AlKhamissy, Gargi Ghosh, Mona T. Diab, Asli Celikyilmaz [pdf]

36. Maieutic Prompting: Logically Consistent Reasoning with Recursive Explanations EMNLP (2022)

    Jaehun Jung, Lianhui Qin, Sean Welleck, Faeze Brahman, Chandra Bhagavatula, Ronan Le Bras, Yejin Choi [pdf] [project]

37. Using Commonsense Knowledge to Answer Why-Questions EMNLP (2022)

    Yash Kumar Lal, Niket Tandon, Tanvi Aggarwal, Horace Liu, Nathanael Chambers, Raymond J. Mooney, Niranjan Balasubramanian [pdf] [project]

There are some datasets collected from realistic examinations or tests or explicitly designed to challenge LLMs, which may require domain-specific knowledge and multiple types of reasoning skills.

Knowledge Graph Reasoning

Knowledge graph completion task aims to complete the graph, while multi-hop reasoning over KG is the task querying in incomplete graphs, both of which require reasoning over knowledge graphs. Temporal knowledge graph reasoning aims to predict links in future with the past quadruples.

Knowledge Graph Completion

1. Collaborative Policy Learning for Open Knowledge Graph Reasoning EMNLP (2019)

   Cong Fu, Tong Chen, Meng Qu, Woojeong Jin, Xiang Ren [pdf] [project]

2. DIVINE: A Generative Adversarial Imitation Learning Framework for Knowledge Graph Reasoning EMNLP (2019)

   Ruiping Li, Xiang Cheng [pdf] [project]

3. Learning Collaborative Agents with Rule Guidance for Knowledge Graph Reasoning EMNLP (2020)

   Deren Lei, Gangrong Jiang, Xiaotao Gu, Kexuan Sun, Yuning Mao, Xiang Ren [pdf] [project]

4. Incorporating Graph Attention Mechanism into Knowledge Graph Reasoning Based on Deep Reinforcement Learning EMNLP (2019)

   Heng Wang, Shuangyin Li, Rong Pan, Mingzhi Mao [pdf] [project]

5. Dynamically Pruned Message Passing Networks for Large-scale Knowledge Graph Reasoning ICLR Poster (2020)

   Xiaoran Xu, Wei Feng, Yunsheng Jiang, Xiaohui Xie, Zhiqing Sun, Zhi-Hong Deng [pdf] [project]

6. Inductive Relation Prediction by Subgraph Reasoning ICML (2020)

   Komal K. Teru, Etienne G. Denis, William L. Hamilton [pdf] [project]

7. Adapting Meta Knowledge Graph Information for Multi-Hop Reasoning over Few-Shot Relations EMNLP (2019)

   Xin Lv, Yuxian Gu, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu [pdf] [project]

8. Dynamic Anticipation and Completion for Multi-Hop Reasoning over Sparse Knowledge Graph EMNLP (2020)

   Xin Lv, Xu Han, Lei Hou, Juanzi Li, Zhiyuan Liu, Wei Zhang, Yichi Zhang, Hao Kong, Suhui Wu [pdf] [project]

9. UniKER: A Unified Framework for Combining Embedding and Definite Horn Rule Reasoning for Knowledge Graph Inference EMNLP (2021)

   Kewei Cheng, Ziqing Yang, Ming Zhang, Yizhou Sun [pdf]

10. Is Multi-Hop Reasoning Really Explainable? Towards Benchmarking Reasoning Interpretability EMNLP (2021)

    Xin Lv, Yixin Cao, Lei Hou, Juanzi Li, Zhiyuan Liu, Yichi Zhang, Zelin Dai [pdf] [project]

11. GMH: A General Multi-hop Reasoning Model for KG Completion EMNLP (2021)

    Yao Zhang, Hongru Liang, Adam Jatowt, Wenqiang Lei, Xin Wei, Ning Jiang, Zhenglu Yang [pdf]

12. Neural-Symbolic Commonsense Reasoner with Relation Predictors ACL (2021)

    Farhad Moghimifar, Lizhen Qu, Yue Zhuo, Gholamreza Haffari, Mahsa Baktashmotlagh [pdf] [project]

Multi-Hop Reasoning over KG

1. Query2box: Reasoning over Knowledge Graphs in Vector Space Using Box Embeddings ICLR Poster (2020)

   Hongyu Ren, Weihua Hu, Jure Leskovec [pdf] [project]

2. Beta Embeddings for Multi-Hop Logical Reasoning in Knowledge Graphs NIPS (2020)

   Hongyu Ren, Jure Leskovec [pdf] [project]

3. Probabilistic Entity Representation Model for Reasoning over Knowledge Graphs NIPS (2021)

   Nurendra Choudhary, Nikhil Rao, Sumeet Katariya, Karthik Subbian, Chandan K. Reddy [pdf] [project]

4. ConE: Cone Embeddings for Multi-Hop Reasoning over Knowledge Graphs NIPS (2021)

   Zhanqiu Zhang, Jie Wang, Jiajun Chen, Shuiwang Ji, Feng Wu [pdf] [project]

5. Complex Query Answering with Neural Link Predictors ICLR Oral (2021)

   Erik Arakelyan, Daniel Daza, Pasquale Minervini, Michael Cochez [pdf] [project]

Temporal Knowledge Graph Reasoning

1. Explainable Subgraph Reasoning for Forecasting on Temporal Knowledge Graphs ICLR Poster (2021)

   Zhen Han, Peng Chen, Yunpu Ma, Volker Tresp [pdf] [project]

2. Search from History and Reason for Future: Two-stage Reasoning on Temporal Knowledge Graphs ACL (2021)

   Zixuan Li, Xiaolong Jin, Saiping Guan, Wei Li, Jiafeng Guo, Yuanzhuo Wang, Xueqi Cheng [pdf]

3. Complex Evolutional Pattern Learning for Temporal Knowledge Graph Reasoning ACL (2022)

   Zixuan Li, Saiping Guan, Xiaolong Jin, Weihua Peng, Yajuan Lyu, Yong Zhu, Long Bai, Wei Li, Jiafeng Guo, Xueqi Cheng [pdf] [project]

Others

1. Quantum Embedding of Knowledge for Reasoning NIPS (2019)

   Dinesh Garg, Shajith Ikbal, Santosh K. Srivastava, Harit Vishwakarma, Hima P. Karanam, L. Venkata Subramaniam [pdf] [project]

2. Scalable Neural Methods for Reasoning With a Symbolic Knowledge Base ICLR Poster (2020)

   William W. Cohen, Haitian Sun, R. Alex Hofer, Matthew Siegler [pdf]

3. Probabilistic Logic Neural Networks for Reasoning NIPS (2019)

   Meng Qu, Jian Tang [pdf]

4. RNNLogic: Learning Logic Rules for Reasoning on Knowledge Graphs ICLR Poster (2021)

   Meng Qu, Junkun Chen, Louis-Pascal A. C. Xhonneux, Yoshua Bengio, Jian Tang [pdf] [project]

5. Efficient Probabilistic Logic Reasoning with Graph Neural Networks ICLR Poster (2020)

   Yuyu Zhang, Xinshi Chen, Yuan Yang, Arun Ramamurthy, Bo Li, Yuan Qi, Le Song [pdf]

6. Probabilistic Box Embeddings for Uncertain Knowledge Graph Reasoning NAACL (2021)

   Xuelu Chen, Michael Boratko, Muhao Chen, Shib Sankar Dasgupta, Xiang Lorraine Li, Andrew McCallum [pdf] [project]

7. Multimodal Analogical Reasoning over Knowledge Graphs ICLR (2023)

   Ningyu Zhang, Lei Li, Xiang Chen, Xiaozhuan Liang, Shumin Deng, Huajun Chen [pdf] [project]

Mathematical Reasoning

Benchmarks & Datasets

1. Analysing Mathematical Reasoning Abilities of Neural Models ICLR Poster (2019)

   David Saxton, Edward Grefenstette, Felix Hill, Pushmeet Kohli [pdf] [project]

2. HOList: An Environment for Machine Learning of Higher-Order Theorem Proving ICML (2019)

   Kshitij Bansal, Sarah M. Loos, Markus N. Rabe, Christian Szegedy, Stewart Wilcox [pdf] [project]

3. DROP: A Reading Comprehension Benchmark Requiring Discrete Reasoning Over Paragraphs EMNLP (2019)

   Dheeru Dua, Yizhong Wang, Pradeep Dasigi, Gabriel Stanovsky, Sameer Singh, Matt Gardner [pdf] [project]

4. IsarStep: a Benchmark for High-level Mathematical Reasoning ICLR Poster (2021)

   Wenda Li, Lei Yu, Yuhuai Wu, Lawrence C. Paulson [pdf] [project]

5. Towards Table-to-Text Generation with Numerical Reasoning ACL (2021)

   Lya Hulliyyatus Suadaa, Hidetaka Kamigaito, Kotaro Funakoshi, Manabu Okumura, Hiroya Takamura [pdf] [project]

6. Inter-GPS: Interpretable Geometry Problem Solving with Formal Language and Symbolic Reasoning ACL (2021)

   Pan Lu, Ran Gong, Shibiao Jiang, Liang Qiu, Siyuan Huang, Xiaodan Liang, Song-Chun Zhu [pdf] [project]

7. FINQA: A Dataset of Numerical Reasoning over Financial Data EMNLP (2021)

   Zhiyu Chen, Wenhu Chen, Charese Smiley, Sameena Shah, Iana Borova, Dylan Langdon, Reema Moussa, Matt Beane, Ting-Hao Huang, Bryan R. Routledge, William Yang Wang [pdf] [project]

8. SciGen: a Dataset for Reasoning-Aware Text Generation from Scientific Tables NIPS (2021)

   Nafise Sadat Moosavi, Andreas Rücklé, Dan Roth, Iryna Gurevych [pdf] [project]

9. MULTIHIERTT: Numerical Reasoning over Multi Hierarchical Tabular and Textual Data ACL (2022)

   Yilun Zhao, Yunxiang Li, Chenying Li, Rui Zhang [pdf] [project]

10. NUMGLUE: A Suite of Fundamental yet Challenging Mathematical Reasoning Tasks ACL (2022)

    Swaroop Mishra, Arindam Mitra, Neeraj Varshney, Bhavdeep Singh Sachdeva, Peter Clark, Chitta Baral, Ashwin Kalyan [pdf] [project]

Papers

1. Semantically-Aligned Equation Generation for Solving and Reasoning Math Word Problems NAACL (2019)

   Ting-Rui Chiang, Yun-Nung Chen [pdf] [project]

2. A Multi-Type Multi-Span Network for Reading Comprehension that Requires Discrete Reasoning EMNLP (2019)

   Minghao Hu, Yuxing Peng, Zhen Huang, Dongsheng Li [pdf] [project]

3. NumNet: Machine Reading Comprehension with Numerical Reasoning EMNLP (2019)

   Qiu Ran, Yankai Lin, Peng Li, Jie Zhou, Zhiyuan Liu [pdf] [project]

4. Mathematical Reasoning in Latent Space ICLR Oral (2020)

   Dennis Lee, Christian Szegedy, Markus N. Rabe, Sarah M. Loos, Kshitij Bansal [pdf]

5. Neural Module Networks for Reasoning over Text ICLR Poster (2020)

   Nitish Gupta, Kevin Lin, Dan Roth, Sameer Singh, Matt Gardner [pdf] [project]

6. Injecting Numerical Reasoning Skills into Language Models ACL (2020)

   Mor Geva, Ankit Gupta, Jonathan Berant [pdf] [project]

7. Question Directed Graph Attention Network for Numerical Reasoning over Text EMNLP (2020)

   Kunlong Chen, Weidi Xu, Xingyi Cheng, Zou Xiaochuan, Yuyu Zhang, Le Song, Taifeng Wang, Yuan Qi, Wei Chu [pdf]

8. Mathematical Reasoning via Self-supervised Skip-tree Training ICLR Spotlight (2021)

   Markus Norman Rabe, Dennis Lee, Kshitij Bansal, Christian Szegedy [pdf]

9. Incorporating External Knowledge to Enhance Tabular Reasoning NAACL (2021)

   J. Neeraja, Vivek Gupta, Vivek Srikumar [pdf] [project]

10. Measuring and Improving BERT's Mathematical Abilities by Predicting the Order of Reasoning ACL (2021)

    Piotr Piekos, Mateusz Malinowski, Henryk Michalewski [pdf]

11. GraphMR: Graph Neural Network for Mathematical Reasoning ACL (2021)

    Weijie Feng, Binbin Liu, Dongpeng Xu, Qilong Zheng, Yun Xu [pdf] [project]

12. LIME: Learning Inductive Bias for Primitives of Mathematical Reasoning ICML (2021)

    Yuhuai Wu, Markus N. Rabe, Wenda Li, Jimmy Ba, Roger B. Grosse, Christian Szegedy [pdf] [project]

13. Numerical reasoning in machine reading comprehension tasks: are we there yet? EMNLP (2021)

    Hadeel Al-Negheimish, Pranava Madhyastha, Alessandra Russo [pdf]

14. Learning to Reason Deductively: Math Word Problem Solving as Complex Relation Extraction ACL (2022)

    Zhanming Jie, Jierui Li, Wei Lu [pdf] [project]

15. FORTAP: Using Formulas for Numerical-Reasoning-Aware Table Pretraining ACL (2022)

    Zhoujun Cheng, Haoyu Dong, Ran Jia, Pengfei Wu, Shi Han, Fan Cheng, Dongmei Zhang [pdf] [project]

16. Right for the Right Reason: Evidence Extraction for Trustworthy Tabular Reasoning ACL (2022)

    Vivek Gupta, Shuo Zhang, Alakananda Vempala, Yujie He, Temma Choji, Vivek Srikumar [pdf] [project]

17. Turning Tables: Generating Examples from Semi-structured Tables for Endowing Language Models with Reasoning Skills ACL (2022)

    Ori Yoran, Alon Talmor, Jonathan Berant [pdf] [project]

18. OPERA: Operation-Pivoted Discrete Reasoning over Text NAACL (2022)

    Yongwei Zhou, Junwei Bao, Chaoqun Duan, Haipeng Sun, Jiahui Liang, Yifan Wang, Jing Zhao, Youzheng Wu, Xiaodong He, Tiejun Zhao [pdf] [project]

Contributor

Fei YU

Reference

@article{yu2023natural,
  title={Natural Language Reasoning, A Survey},
  author={Yu, Fei and Zhang, Hongbo and Wang, Benyou},
  journal={arXiv preprint arXiv:2303.14725},
  year={2023}
}