Awesome

Awesome NVIDIA Isaac Gym 🤖

A curated collection of resources related to NVIDIA Isaac Gym, a high-performance GPU-based physics simulation environment for robot learning.

🎯 Quick Links

• Official Website

• Documentation

• Community Forum

• Latest Release Info

📋 Contents

• Latest Releases

• Getting Started

• Official Resources

• Learning Materials

  • Tutorials

  • Workshops

  • Video Guides

• Research Papers

  • Core Papers

  • Robot Manipulation

  • Locomotion & Control

  • Simulation & Learning

• Tools & Libraries

  • RL Frameworks

  • Community Projects

• Applications & Examples

• Community Resources

🚀 Latest Releases

• February 2024: Isaac Lab - A unified and modular framework for robot learning (Website)

• February 2024: PhysX 5 SDK release (GitHub)

• February 2022: Isaac Gym Preview 4 (1.3.0)

• October 2021: Isaac Gym Preview 3

• June 2021: NVIDIA Isaac Sim on Omniverse Open Beta

• March 23, 2022: GTC 2022 Session — Isaac Gym: The Next Generation — High-performance Reinforcement Learning in Omniverse.

• Isaac Gym Overview: Isaac Gym Session.

• GTC Spring 2021: Isaac Gym: End-to-End GPU-Accelerated Reinforcement Learning.

🎓 Getting Started

1. Installation & Setup

   • Official Isaac Gym Download

   • Quick Start Guide

   • Environment Setup

2. Basic Concepts

   • Introduction to Isaac Gym

   • transic: Official Implementation of "TRANSIC: Sim-to-Real Policy Transfer by Learning from Online Correction" CoRL 2024.

   • Robot Synesthesia: Official Implementation of "Robot Synesthesia: In-Hand Manipulation with Visuotactile Sensing" ICRA 2024.

   • RLAfford: Official Implementation of "RLAfford: End-to-end Affordance Learning with Reinforcement Learning" ICRA 2023.

   • Core Components Overview

   • Basic Tutorials

📚 Official Resources

Core Documentation

• Isaac SDK Documentation
• OmniIsaacGymEnvs Repository
• Official Blog Posts

Learning Resources

• Video Tutorials
• Developer Blog
• NVIDIA Omniverse Channel

📖 Learning Materials

Tutorials

Comprehensive tutorial series from RSS 2021 Workshop:

1. Introduction & Getting Started
2. Environments, Training & Tips
3. Academic Labs Series:
   • University of Toronto
   • IMLab
   • Stanford University
   • Soft-Body Simulation
   • ETH Zurich
4. New Frontiers in GPU Accelerated RL

Video Guides

• Robot Import Guide

• Simulator Basics

• Advanced Features

• Community Demonstrations

• From Point Clouds to Material Graphs: Explore the Latest in Omniverse Create 2021.3

• Robot Autonomy with the Digital Twin in Isaac Sim

• Can We Simulate a Real Robot? — A journey through finding a high-quality physics simulator for a robot quadruped.

• Teaching Robots to Walk with Reinforcement Learning — Robot simulation adventure, covering reinforcement learning with the Bittle robot.

• Robot Dog Learns to Walk — Bittle Reinforcement Learning Part 3 — Further progress in training robot quadrupeds to walk.

• Isaac Sim GTC 2021 — Sim-to-Real: Session on sim-to-real transfer using Isaac Sim.

• Isaac Sim Video Tutorials: Official video tutorials.

• Training Your JetBot in NVIDIA Isaac Sim: Guide on training JetBot using Isaac Sim.

• Training Your NVIDIA JetBot to Avoid Collisions Using NVIDIA Isaac Sim: Blog post on collision avoidance training.

• Introducing NVIDIA Isaac Gym: End-to-End Reinforcement Learning for Robotics: Introduction to Isaac Gym.

• Accelerating Robotics Simulation with NVIDIA Omniverse Isaac Sim: Blog post on using Omniverse with Isaac Sim.

• Developing Robotics Applications in Python with NVIDIA Isaac SDK: Guide on using Isaac SDK with Python.

• Building an Intelligent Robot Dog with the NVIDIA Isaac SDK: Tutorial on building a robot dog.

• NVIDIA Omniverse YouTube Channel: Official channel with various tutorials and demos.

• [ICRA2021] Sim-to-Real for Robotic Tactile Sensing via Physics-Based Simulation and Learned Latent Projections: paper

• [2021] DeformerNet: A Deep Learning Approach to 3D Deformable Object Manipulation: paper

• [RSS2021_VLRR] A Simple Method for Complex In-Hand Manipulation: paper, project

Locomotion

• [RSS2024] Agile But Safe: Learning Collision-Free High-Speed Legged Locomotion: paper, code

• [RSS2022] Rapid Locomotion via Reinforcement Learning: paper, openreview, code

• [CoRL2021] Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning: paper, openreview, code, project

• [CoRL2021] Learning to Walk in Minutes Using Massively Parallel Deep Reinforcement Learning: paper, openreview, code, project

• [ICRA2021] Dynamics Randomization Revisited:A Case Study for Quadrupedal Locomotion: project, paper, video

• [2021] GLiDE: Generalizable Quadrupedal Locomotion in Diverse Environments with a Centroidal Model: project, paper

• [CoRL2020] Learning a Contact-Adaptive Controller for Robust, Efficient Legged Locomotion: paper, video, project, blog

• [RAL2021] Learning a State Representation and Navigation in Cluttered and Dynamic Environments: paper

Blogs

• A Brief Introduction to NVIDIA Omniverse

📑 Research Papers

Core Papers

• Isaac Gym: High Performance GPU-Based Physics Simulation (NeurIPS 2021)
  • Project Page
  • OpenReview Discussion

Robot Manipulation

• RLAfford: Official implementation of "RLAfford: End-to-end Affordance Learning with Reinforcement Learning", ICRA 2023.
• Masked Visual Pre-training for Robotics (MVP): Repository for the MVP project.
• Factory: Fast Contact for Robotic Assembly: RSS 2022.
  • Paper
  • Code
• ASE: Adversarial Skill Embeddings: SIGGRAPH 2022.
  • Paper
  • Code
• Data-Driven Operational Space Control (OSCAR): Adaptive and robust robot manipulation.
  • Paper
  • Code
• DefGraspSim: Simulation-based grasping of deformable objects.
  • Paper
  • Video
  • Code
• In-Hand Object Pose Tracking: ICRA 2021.
  • Paper
• STORM: Fast Joint-Space MPC for Reactive Manipulation: CoRL 2021.
  • Paper
  • Code
• Transferring Dexterous Manipulation from GPU Simulation to Real-World TriFinger:
  • Paper
  • Code
• Causal Reasoning in Simulation for Robot Manipulation Policies: ICRA 2021.
  • Paper
• Reactive Long Horizon Task Execution: IROS 2021.
  • Paper

Localization & Control

• Learning to Walk in Minutes Using Massively Parallel Deep RL: CoRL 2021.
  • Paper
  • Code
• Dynamics Randomization Revisited: A case study for quadrupedal locomotion.
  • Paper
  • Video
• GLiDE: Generalizable Quadrupedal Locomotion:
  • Paper
• Learning a Contact-Adaptive Controller: For robust, efficient legged locomotion.
  • Paper
  • Video
  • Blog
• Learning a State Representation and Navigation: In cluttered and dynamic environments.

Others

• BayesSimIG: Scalable parameter inference for adaptive domain randomization with Isaac Gym.
  • Code
• Isaac Gym: High Performance GPU-Based Physics Simulation: NeurIPS 2021.
  • Paper
  • OpenReview
• Learning to Swim: Reinforcement learning for 6-DOF control of thruster-driven AUVs.
• MarineGym: Accelerated Training for Underwater Vehicles with High-Fidelity RL Simulation: Based on Issac Sim
• space_robotics_bench Space Robotics Bench
• Humanoid-Gym: Humanoid-Gym: Reinforcement Learning for Humanoid Robot with Zero-Shot Sim2Real Transfer

🛠 Tools & Libraries

RL Frameworks

• RL Games - Compatible RL algorithms

• ElegantRL

• skrl - Modular RL library

• Minimal Stable PPO

• skrl, paper

• RSL RL

Related GitHub Repos

• IsaacGymEnvs

• isaacgym_hammering

• isaacgym-utils: Developed by the CMU Intelligent Autonomous Manipulation Lab

• isaacgym_sandbox

• thormang3-gogoro-PPO: Steering-based control of a two-wheeled vehicle using RL-PPO and NVIDIA Isaac Gym

• dvrk_IssacGym, link

Community Projects

• IsaacGymEnvs: Official Isaac Gym RL environments.
• isaacgym_hammering: Hammering task implementation.
• isaacgym-utils: Utilities by CMU's Intelligent Autonomous Manipulation Lab.
• isaacgym_sandbox: Sandbox for Isaac Gym experiments.
• thormang3-gogoro-PPO: Two-wheeled vehicle control using PPO.
• Bez_IsaacGym: Environments for humanoid robot Bez.
• DexterousHands: Dual dexterous hand manipulation tasks.
• legged_gym_isaac: Legged robots in Isaac Gym.
• shifu: Environment builder for any robot.
• Rofunc: Python package for robot learning from demonstration.
• Dofbot Reacher: Sim2Real environment for Dofbot.
• UR10 Reacher: Sim2Real environment for UR10.
• TimeChamber: Massively parallel self-play framework.
• RL-MPC-Locomotion: Deep RL for quadruped locomotion.
• Isaac_Underwater: Water and underwater tests using NVIDIA Isaac Sim.
• VRKitchen2.0-IndoorKit: Omniverse IndoorKit Extension.
• agibot_x1_train: The reinforcement learning training code for AgiBot X1.

Conference Sessions and Talks

• Isaac Gym and Omniverse: High Performance Reinforcement Learning Evolved [A31118]
• Learning Challenging Tasks for Quadrupedal Robots: From Simulation to Reality [A31308]
• Sim-to-Real in Isaac Sim
• Isaac Gym: End-to-End GPU-Accelerated Reinforcement Learning
• Bridging Sim2Real Gap: Simulation Tuning for Training Deep Learning Robotic Perception Models
• Reinforcement Learning and Intralogistics
• Building Robotics Applications Using NVIDIA Isaac SDK
• NVIDIA Isaac Sim — Amazing Robot Models and Tasks
• Omniverse View 2021.2 — Application Tour
• ISAAC SIM Introduction and Live Demo
• NVIDIA On-Demand ISAAC SIM Sessions

🌟 Contributing

Contributions are welcome! Please read our contribution guidelines before submitting a pull request.

📄 License

This repository is licensed under the MIT License - see the LICENSE file for details.

🙏 Acknowledgments

Special thanks to all contributors and the NVIDIA Isaac team for making these resources available to the robotics community.