Home

Awesome

Awesome-Robotics-Foundation-Models

Awesome

alt text

This is the partner repository for the survey paper "Foundation Models in Robotics: Applications, Challenges, and the Future". The authors hope this repository can act as a quick reference for roboticists who wish to read the relevant papers and implement the associated methods. The organization of this readme follows Figure 1 in the paper (shown above) and is thus divided into foundation models that have been applied to robotics and those that are relevant to robotics in some way.

We welcome contributions to this repository to add more resources. Please submit a pull request if you want to contribute!

Table of Contents

Survey

This repository is largely based on the following paper:

Foundation Models in Robotics: Applications, Challenges, and the Future <br /> Roya Firoozi, Johnathan Tucker, Stephen Tian, Anirudha Majumdar, Jiankai Sun, Weiyu Liu, Yuke Zhu, Shuran Song, Ashish Kapoor, Karol Hausman, Brian Ichter, Danny Driess, Jiajun Wu, Cewu Lu, Mac Schwager <br />

If you find this repository helpful, please consider citing:

@article{firoozi2024foundation,
  title={Foundation Models in Robotics: Applications, Challenges, and the Future},
  author={Firoozi, Roya and Tucker, Johnathan and Tian, Stephen and Majumdar, Anirudha and Sun, Jiankai and Liu, Weiyu and Zhu, Yuke and Song, Shuran and Kapoor, Ashish and Hausman, Karol and others},
  journal={The International Journal of Robotics Research},
  year={2024}, 
  doi= {https://doi.org/10.1177/02783649241281508}
}
@article{firoozi2023foundation,
  title={Foundation Models in Robotics: Applications, Challenges, and the Future},
  author={Firoozi, Roya and Tucker, Johnathan and Tian, Stephen and Majumdar, Anirudha and Sun, Jiankai and Liu, Weiyu and Zhu, Yuke and Song, Shuran and Kapoor, Ashish and Hausman, Karol and others},
  journal={arXiv preprint arXiv:2312.07843},
  year={2023}
}

Robotics

Neural Scaling Laws

Robot Policy Learning for Decision-Making and Controls

Language-Conditioned Imitation Learning

Language-Assisted Reinforcement Learning

Language-Image Goal-Conditioned Value Learning

Robot Task Planning Using Large Language Models

LLM-Based Code Generation

Robot Transformers

In-context Learning for Decision-Making

Open-Vocabulary Robot Navigation and Manipulation

Relevant to Robotics (Perception)

Open-Vocabulary Object Detection and 3D Classification

Open-Vocabulary Semantic Segmentation

Open-Vocabulary 3D Scene Representations

Object Representations

Affordance Information

Predictive Models

Relevant to Robotics (Embodied AI)

Generalist AI

Simulators