Awesome

NVIDIA Isaac Summary

A list of NVIDIA Isaac components. [link]

Component Descriptions

Overview

• Omniverse [link][docs]

  The platform for connecting and developing OpenUSD applications.

  -- NVIDIA Omniverse

  It is worth noting that Omniverse is a platform, and is not a standalone application. Therefore, the term Omniverse, should not be used to refer to a specific component, but to the entire platform. Technically speaking, there is no such thing as an Omniverse app/component in the entire software stack. For an example, Omniverse USD Composer (formerly Omniverse Create) and Omniverse USD Presenter (formerly Omniverse View) are apps built upon Omniverse Kit. They are software components that run on the Omniverse platform.

• Isaac [link]

  The NVIDIA Isaac robotics platform includes a full suite of GPU-accelerated innovations in AI perception, manipulation, simulation, and software.

  -- NVIDIA Isaac

  Isaac stands as NVIDIA's GPU-accelerated solution for robotics, which refers to the robotics platform instead of a specific software component. Alongside essential tools for general AI applications (such as TensorRT, TAO Toolkit, and Triton Inference Server), NVIDIA Isaac components can be categorized into two main branches: (1) Isaac Sim (including Isaac Gym, Isaac Orbit, etc.) for simulation, which currently requires x86 CPUs and RTX GPUs for acceleration. (2) Isaac ROS for deployment, which can run on both PCs (x86 CPUs) and on Jetson hardware (ARM CPUs).

(Omniverse) Isaac Sim

• (Omniverse) Isaac Sim [link][docs][ngc][dli][dli][youtube]
  a robotics simulation toolkit based on Omniverse.

  a scalable robotics simulation application and synthetic data-generation tool that powers photorealistic, physically accurate virtual environments.

  -- NVIDIA Isaac Sim

  Before starting, please make sure your hardware and software meet the system requirements.

  Technically, Isaac Sim is an app built upon Omniverse Kit, which is a SDK for building apps upon the Omniverse platform. The simulation is accelerated by PhysX, while the scene is rendered through RTX rendering.

  Isaac Sim can be downloaded through Omniverse Launcher here:

  • Linux
  • Windows

  The assets are accessed through Omniverse Nucleus, which requires setting up a (local) Nucleus account. In addition, installing Omniverse Cache can speed up the access to Nucleus. If no custom assets are used, the built-in assets can be accessed through the default remote Nucleus server.

  • Isaac Sim Unity3D [docs]
    Unity3D support has been deprecated (source). The term Isaac Sim now refer to the Omniverse-based version.

    allows you to use Unity3D as the simulation environment for Isaac robotics.

    -- NVIDIA Isaac SDK

  • ROS & ROS 2 Bridges [docs][docs]

    tools to facilitate integration with ROS systems.

    -- NVIDIA Isaac Sim

  • Isaac Cortex [docs]
    A behavior programming tool.

    enables programming task awareness and adaptive decision making into robots, and easily switching between simulation and reality.

    -- NVIDIA Isaac Cortex (slightly rephrased)

  • Isaac Core API [docs]
    A Python abstraction API (for Pixar USD API).

    a set of APIs that are designed to be used in robotics applications, APIs that abstract away the complexity of USD APIs and merge multiple steps into one for frequently performed tasks.

    -- NVIDIA Isaac Core API

  • Isaac Sim Assets [docs]
    A collection of USD assets including environments, robots, sensors, props, and other featured assets.
  • other features such as OmniGraph, Importers, etc.

• (Omniverse) Replicator [link][docs][blog]
  a synthetic data generation (SDG) toolkit based on Omniverse.

  an advanced, extensible SDK to generate physically accurate 3D synthetic data, and easily build custom synthetic data generation (SDG) tools to accelerate the training and accuracy of perception networks.

  -- NVIDIA Replicator

  • Isaac Sim Replicator [docs]

    a collection of extensions, python APIs, workflows, and tools such as Replicator Composer that enable a variety of synthetic data generation tasks.

    -- NVIDIA Isaac Sim

  • (Omniverse) Replicator Insight [link]

    an app that enables developers to quickly view, navigate, and inspect their synthetically generated renders.

    -- NVIDIA Replicator Insight

Isaac Lab

• Isaac Lab [link][docs][docs][github]
  an open-source platform based on Isaac Sim, currently the de facto framework for robot learning in Omniverse.
  Formerly Isaac Orbit.

  a unified and modular framework for robot learning that aims to simplify common workflows in robotics research (such as RL, learning from demonstrations, and motion planning). It is built upon NVIDIA Isaac Sim to leverage the latest simulation capabilities for photo-realistic scenes, and fast and efficient simulation.

  -- NVIDIA Isaac Lab

  For more details on the position of Isaac Lab in the Isaac ecosystem, see the Isaac Lab FAQ.

• Deprecated Frameworks

  Isaac Lab will be replacing previously released frameworks for robot learning and reinformcement learning, including IsaacGymEnvs for the Isaac Gym Preview Release, OmniIsaacGymEnvs for Isaac Sim, and Orbit for Isaac Sim.

  These frameworks are now deprecated in favor of continuing development in Isaac Lab. We encourage current users of these frameworks to migrate your work over to Isaac Lab. Migration guides are available to support the migration process:

  • Migrating from IsaacGymEnvs and Isaac Gym Preview Release: link

  • Migrating from OmniIsaacGymEnvs: link

  • Migrating from Orbit: link

  -- NVIDIA Isaac Lab

  • Isaac Orbit [docs][docs][arxiv][site][github]
    a general repository based on Isaac Sim that features a number of GPU-accelerated simulation environments, a variety of motion generators, integrations with several reinforcement learning libraries, utilities for imitation learning, etc.
    Released under the BSD 3-Clause License (source).

    a unified and modular framework for robot learning that aims to simplify common workflows in robotics research (such as RL, learning from demonstrations, and motion planning).

    -- NVIDIA Isaac Orbit

    Omniverse Isaac Gym is a light-weight framework focusing on reinforcement learning tasks, while Isaac Orbit is a more general and modular framework that focuses on robotics applications. (source, source)

    Isaac Orbit is now deprecated and will continue evolve as Isaac Lab. (source)

  • (Omniverse) Isaac Gym [docs][github]
    a light-weight repository based on Isaac Sim that provides a variety of GPU-accelerated reinforcement learning environments and algorithms.
    The repository is named as Omniverse Isaac Gym Environments (OIGE), and is released under the BSD 3-Clause License (source).

    an interface for performing reinforcement learning training and inferencing in Isaac Sim.

    -- NVIDIA Isaac Gym

    The latest release of Omniverse Isaac Gym is 4.0.0, and will not be further updated. (source)

  • Isaac Gym (Preview Release) [link][github][arxiv][site][youtube]
    the predecessor of (Omniverse) Isaac Gym that does not base on Isaac Sim (and Omniverse).
    The repository is named as Isaac Gym Environments (IGE), and is released under the BSD 3-Clause License (source). The documentation is provided in an offline form that can be accessed after download.

    NVIDIA’s physics simulation environment for reinforcement learning research.

    -- NVIDIA Isaac Gym

    The term Isaac Gym is ambiguous when viewed from a technical perspective. It's better to specify whether the mentioned Isaac Gym is based on Isaac Sim, or the preview version that does not base on Isaac Sim.

    Until Omniverse Isaac Gym functionality is feature complete, this standalone Isaac Gym Preview release will remain available.

    -- NVIDIA Isaac Gym

    The latest release of Isaac Gym (Preview Release) is Preview 4, and will not be further updated.

• Project GR00T (Generalist Robot 00 Technology) [link]

  a general-purpose foundation model that promises to transform humanoid robot learning in simulation and the real world. Trained in NVIDIA GPU-accelerated simulation, GR00T enables humanoid embodiments to learn from a handful of human demonstrations with imitation learning and NVIDIA Isaac Lab for reinforcement learning, as well as generating robot movements from video data. The GR00T model takes multimodal instructions and past interactions as input and produces the actions for the robot to execute.

  -- NVIDIA Project GR00T

  The GR00T model refers to a general-purpose foundation model for humanoid robots trained in Isaac Lab with imitation learning and reinforcement learning.

Isaac ROS

• Isaac Robot Operating System (ROS) [link][github][docs]
  a collection of GPU-accelerated ROS2 packages (i.e., Isaac GEMs) and pipelines.

  a collection of hardware-accelerated packages that make it easier for ROS developers to build high-performance solutions on NVIDIA hardware.

  -- Isaac ROS

  In legacy robotics deployment systems, ROS1 is commonly used but lacks support for key use cases such as multi-robot systems, real-time systems, and production-ready environments. Therefore, ROS2 was developed to address these issues. However, using ROS2 can still present challenges, such as reproducibility across different systems and efficient GPU communication between ROS nodes. Isaac ROS, a collection of GPU-accelerated ROS2 packages and pipelines, addresses these challenges by (1) adopting a workflow based on the NVIDIA Container Toolkit (previously NVIDIA Docker) to ensure reproducibility across systems, allowing near-identical deployment experiences across systems with x86 and ARM CPUs, and (2) enabling efficient GPU communication between ROS nodes by reducing memory copies between GPUs and CPUs through NITROS.

  Mainstream robotics applications can be roughly categorized into manipulation and navigation tasks. To address these tasks, Isaac Manipulator and Isaac Perceptor were developed as reference workflows. These workflows will be described in detail below.

  The term Isaac ROS refer to the packages for ROS 2, instead of Isaac SDK. Isaac ROS should not be confused with the ROS & ROS 2 Bridges in Isaac Sim, or the ROS Bridge in Isaac SDK.
  The packages (i.e., Isaac GEMs) are named as Isaac ROS <Package_Name>. Unfortunately, ambiguous terms such as Isaac Elbrus still exist (source). Since the Elbrus package exist in both Isaac ROS and Isaac SDK, Elbrus should be refered to as Isaac ROS Elbrus for preciseness.

  To get started quickly, follow these steps:

  1. Ensure your PC/Jetson hardware and operating system meet the system requirements.
  2. Set up your system environment.
  3. Configure your development environment.
  4. (Optional) To feel the power of Isaac ROS, simply follow the Nvblox tutorial for a quick introduction.

  Some of the Isaac ROS packages are listed below:

  • (Isaac ROS) Nvblox [github]

    processes depth and pose to reconstruct a 3D scene in real-time and outputs a 2D costmap for Nav2. The costmap is used in planning during navigation as a vision-based solution to avoid obstacles.

    -- Isaac ROS Nvblox

    This packages includes the use of the Isaac ROS Visual SLAM package, which uses cuVSLAM as the underlying algorithm.

  • (Isaac ROS) NVIDIA Isaac for Transport for ROS (NITROS) [github]

    the NVIDIA implementation of type adaption and negotiation for ROS2 that eliminates software/CPU overhead and improves performance of hardware acceleration.

    -- Isaac ROS (slightly rephrased)

  • etc.

• Isaac Manipulator [link][docs][github]
  a collection of Isaac ROS packages for manipulation tasks.

  a reference workflow of NVIDIA-accelerated libraries and AI models that enables developers to build AI-enabled robot arms, or manipulators, that can perceive, understand, and interact with their environments.

  -- NVIDIA Isaac Manipulator

  • Extended Robot Description Format (XRDF) [docs]

    designed to supplement the URDF by adding specification for: (1) Semantic labeling of configuration space (i.e., c-space) and tool frame(s) to control the robot, (2) Acceleration and jerk limits required to generate smooth motion, (3) Collision spheres to efficiently represent robot geometry, and (4) Masking to regulate self-collision avoidance.

    -- Extended Robot Description Format (XRDF)

    • Lula Robot Description and XRDF Editor [docs]

      UI tool to generate a configuration file that supplements the information available about a robot in its URDF.

      -- Lula Robot Description and XRDF Editor

  • (Isaac ROS) cuMotion [docs][github]

    provides CUDA-accelerated manipulation capabilities for robots in ROS 2. It provides two main capabilities: (1) Motion generation for robot arms via integration of cuMotion into MoveIt 2, and (2) Segmentation of robots from depth streams using cuMotion’s kinematics and geometry processing functions to accurately identify and filter out parts of the robot. This allows reconstruction of obstacles in the environment without spurious contributions from the robot itself.

    -- Isaac ROS cuMotion

    • cuRobo [link][docs][github]

      a CUDA accelerated library containing a suite of robotics algorithms that run significantly faster than existing implementations leveraging parallel compute.

      -- NVIDIA cuRobo

      cuRobo is shipped as Isaac ROS cuMotion for ROS2 integration. (source)

      • cuRobo with Isaac Sim [docs]

• Isaac Perceptor [link][docs][github]
  a collection of Isaac ROS packages for autonomous mobile robots (AMRs).
  Formerly Isaac for AMRs and Isaac AMR.

  a reference workflow of NVIDIA-accelerated libraries and AI models that helps you quickly build robust autonomous mobile robots (AMRs) to perceive, localize, and operate in unstructured environments like warehouses or factories.

  -- NVIDIA Isaac Perceptor

• Isaac AMR (before Isaac Perceptor) [link]

  Isaac AMR 2.0 features an autonomous navigation stack that includes lidar-based grid mapping, global and continuous localization, a global route planner, a mission client, a behavior planner, and wheel-IMU odometry, as well as a new path and trajectory planner and controller.

  This release also includes tools for data collection and a cloud-based map creation service. You can use the on-premises data center-based mission control for optimizing route planning with the NVIDIA cuOpt engine and delivering up-to-date maps to the robots.

  -- NVIDIA Isaac AMR

  Personally, I think Isaac AMR is an extension of Isaac SDK. This is based on the Isaac Sight UI screenshots in source, and I've noticed some Isaac SDK docs have been redirected/moved to Isaac AMR docs.

• Isaac SDK [docs]
  a toolkit for deploying GPU-accelerated algorithms on physical robots.

  a toolkit that includes building blocks and tools that accelerate robot developments that require the increased perception and navigation features enabled by AI.

  -- NVIDIA Isaac SDK

  Personally, I suggest using Isaac ROS instead of Isaac SDK for simplicity. Since researchers/engineers working on robotics tend to be more familiar with the Robot Operating System (ROS) than the bazel command used in Isaac SDK. The Isaac GEMs and Isaac Applications included in Isaac SDK are also available in Isaac ROS.

  Isaac includes Isaac GEMs for both NVIDIA’s Isaac SDK Engine and ROS2. Isaac ROS has been more recently updated to contribute hardware acceleration to the growing ROS ecosystem. You can choose whichever one is more suitable for your project.

  -- NVIDIA Forum Moderator (slightly rephrased)

  The latest release of Isaac SDK is 2021.1, since the future roadmap of Isaac SDK is still under development (source, source, source, source).

  The Isaac Sight GUI also belongs to Isaac SDK.

  • Isaac GEMs

    a collection of high-performance algorithms, also called GEMs, to accelerate the development of challenging robotics applications.

    -- NVIDIA Isaac SDK

  • Isaac Applications

    provides various sample applications, which highlight features of Isaac SDK Engine or focus on the functionality of a particular Isaac SDK GEM.

    -- NVIDIA Isaac SDK

  • Isaac (Robotics) Engine

    a feature-rich framework for building modular robotics applications.

    -- NVIDIA Isaac SDK

Mission Dispatch and Client

• Isaac Mission Dispatch [github]

  a cloud service that enables the communication between edge robots and other cloud services responsible for managing a fleet of robots. The communication between Mission Dispatch and robots is designed per VDA5050 protocol and uses MQTT, as MQTT is the industry standard for a highly efficient, scalable protocol for connecting devices over the internet. VDA 5050 is an open standard for communication between fleets of AGVs/AMRs and a central fleet service.

  -- Isaac Mission Dispatch

• Isaac ROS Mission Client [github]

  the ROS 2 packages for Mission Client, which communicates to a robot fleet management service. Mission Client receives tasks and actions from the fleet management service and updates its progress, state, and errors. Mission Client performs navigation actions with Nav2 and can be integrated with other ROS actions.

  -- Isaac ROS Mission Client

Miscellaneous

• Isaac Nova Orin [link]
  a reference architecture for AMRs based on NVIDIA Jetson AGX Orin.

  a state-of-the-art compute and sensor reference architecture to accelerate AMR development and deployment. It features up to two Jetson AGX Orin computers and a full sensor suite for next-gen AMRs that enable surround vision-based perception with lidar-based solutions.

  -- NVIDIA Isaac Nova Orin

• Nova Carter [link][spec]
  a reference design robot based on the Isaac Nova Orin architecture.

  a reference design robot that uses the Nova Orin compute and sensor architecture. It’s a complete robotics development platform that accelerates the development and deployment of next-generation Autonomous Mobile Robots (AMRs). You can learn more about it from our partner, Segway Robotics

  -- NVIDIA Isaac Nova Orin

• NVIDIA IGX Orin [link]

  an industrial-grade, edge AI platform that combines enterprise-level hardware, software, and support. It’s purpose-built for industrial and medical environments, delivering powerful AI compute, high-bandwidth sensor processing, enterprise security, and functional safety. The platform also comes with NVIDIA AI Enterprise and up to 10 years of support, so you can confidently deliver AI safely and securely to support human and machine collaboration.

  -- NVIDIA IGX Orin

• OSMO [link]

  a cloud-native workflow orchestration platform that lets you easily scale your workloads across distributed environments—from on-premises to private and public cloud. It provides a single pane of glass for scheduling complex multi-stage and multi-container heterogeneous computing workflows.

  -- NVIDIA OSMO

  OSMO refers to an orchestration platform, which can take a workflow spec and run the specified workloads on Omniverse Cloud or DGX Cloud.

• cuOpt [link][docs][ngc][dli][github]
  a GPU-accelerated solver for vehicle routing problem.
  Formerly ReOpt.

  a GPU-accelerated logistics solver that uses heuristics and optimizations to calculate complex vehicle routing problem variants with a wide range of constraints.

  -- NVIDIA cuOpt

  • cuOpt for Isaac Sim [docs][github]

    a reference for the use of NVIDIA cuOpt to solve routing optimization problems in simulation.

    -- NVIDIA Isaac Sim

• Omniverse Cloud [link]

  a platform of APIs, SDKs, and services available within a full-stack cloud environment for enterprise developers to easily integrate Universal Scene Description (OpenUSD) and RTX rendering technologies into their 3D industrial digitalization applications.

  -- NVIDIA Omniverse Cloud

• Omniverse Farm [docs]

  Omniverse Farm Queue and Omniverse Farm Agent allow you to run tasks in the background, and to run automated jobs defined by you or others.

  -- NVIDIA Omniverse Farm

  Omniverse Farm is actually flexible enough to run arbitrary tasks, not just rendering tasks. See the j3soon/omni-farm-isaac repository for more information.

Coding References

OpenUSD Coding References

• API Documentation
• USD Tutorials
• OpenUSD Developer Tutorials
• NVIDIA-Omniverse/OpenUSD-Code-Samples

Omniverse Kit Coding References

• Omniverse Kit Python Snippets
• Code Samples
• Python Scripting
• Set Up Your Development Environment
• Getting Started with Extensions
• NVIDIA-Omniverse/kit-extension-template
• Omniverse Commands Tool
• OmniCLI
• Omni.UI Documentation (in UI through Extensions)
• Physics Demos (in UI through Extensions)

(Omniverse) Isaac Sim Coding References

• Python Environment Installation
• Python Environment
• Debugging With Visual Studio Code
• Isaac Sim Workflows
• Extension Workflow for RL
• Isaac Sim: Extensions API
• Isaac Examples Menu (in UI through Extensions)
• Tips & Useful Links

Development Notes

• For VSCode Intellisense to work, you need to:
  • Correctly link the Omniverse app directories (ref, ref)
  • Correctly set the VSCode settings .vscode/settings.json (ref, ref)
  • Select the correct Python interpreter (e.g., the isaac-sim virtual environment if you are using Isaac Sim with Anaconda)
• Built-in Examples under ~/.local/share/ov/pkg/*
  • Search under ~/.local/share/ov/pkg/isaac-sim-4.0.0 for example API usages.

Epilogue

Please open an issue if you have spotted any errors or have questions regarding this document. For questions regarding the Isaac components, I recommend first going through the Known Issues, then considering asking in the NVIDIA Developer Forums under the Isaac topic.

I have documented some bug fixes and workarounds for Isaac in the j3soon/isaac-extended repository. I recommend also checking out that repository for reference.

Last updated on 2024/07/18.