Home

Awesome

Running Isaac Sim Workloads on Omniverse Farm

Support Matrix

The scripts below are currently based on Isaac Sim 4.1.0 and Isaac Lab 1.1.0. The scripts should work on other versions of Isaac Sim and Isaac Lab, but you may need to modify the scripts accordingly.

Isaac Sim

Docker ImageIsaac SimUbuntu
j3soon/omni-farm-isaac-sim:4.2.04.2.022.04.3 LTS
j3soon/omni-farm-isaac-sim:4.1.04.1.022.04.3 LTS
j3soon/omni-farm-isaac-sim:4.0.04.0.022.04.3 LTS
j3soon/omni-farm-isaac-sim:2023.1.12023.1.122.04.3 LTS

Isaac Lab

Docker ImageIsaac LabIsaac SimUbuntu
j3soon/omni-farm-isaac-lab:1.1.01.1.04.1.022.04.3 LTS

Installing Omniverse Farm

Skip this section if you already have Omniverse Farm installed.

Pre-Installation

Before proceeding with the installation, make sure you have modified the following values:

(Note that the pre-installation steps are not tested on a real machine yet...)

Installation

Follow the official installation guide to install Omniverse Farm.

After installation, you should have a installed Farm Queue, and one or more Farm Agent workers installed, which can be connected to the queue in subsequent steps. All Farm Agents should have access to the USD scenes that would be used in the submitted jobs through Nucleus.

Follow this example to test your Omniverse Farm installation. First, submit a rendering job through Movie Capture. Next, connect a Farm Agent to the Farm Queue, and make sure the job finished successfully by checking the output files. Please skip the Blender decimation example in the documentation, as it is not relevant to this repository.

This repo is tested on Omniverse Farm 105.1.0 with Kubernetes set up. The scripts are tested within a environment consists of multiple OVX server nodes with L40 GPUs, a CPU-only head node, along with a large NVMe storage server. These servers are interconnected via a high-speed network utilizing the BlueField-3 DPU and ConnectX-7 NIC. See this post and this post for more information. However, the scripts in this repository should work on any Omniverse Farm setup, even on a single machine.

Post-Installation

If you forgot to perform the pre-installation steps, you can still perform them after installation:

Setup

Clone this repository:

git clone https://github.com/j3soon/omni-farm-isaac.git
cd omni-farm-isaac

Install jq for JSON parsing. For example if you are using Ubuntu:

sudo apt-get update
sudo apt-get install -y jq

Fill in the Omniverse Farm server information in secrets/env.sh, for example:

export FARM_API_KEY="s3cr3t"
export FARM_URL="http://localhost:8222"
export FARM_USER="j3soon"
export NUCLEUS_HOSTNAME="localhost"

Then, for each shell session, make sure to source the environment variables by running the following command in the root directory of this repository:

source secrets/env.sh

In some examples below, we will upload files to Nucleus through omnicli, you can use the GUI to upload files to Nucleus instead.

All following commands assume you are in the root directory of this repository (omni-farm-isaac) and have sourced the environment variables file (secrets/env.sh).

Setup VPN

Skip this section if accessing your Omniverse Farm doesn't require a VPN.

There doesn't seem to be a way to use the OpenVPN Connect v3 GUI on Linux as in Windows or MacOS. Instead, use the command line to install OpenVPN 3 Client by following the official guide.

Then, copy your .ovpn client config file to secrets/client.ovpn and install the config, and connect to the VPN with:

scripts/vpn/install_config.sh client.ovpn
scripts/vpn/connect.sh

To disconnect from the VPN, and uninstall the VPN config, run:

scripts/vpn/disconnect.sh
scripts/vpn/uninstall_config.sh

These 4 scripts are just wrappers for the openvpn3 command line tool. See the official documentation for more details.

If a previous config is already installed, you must uninstall it before installing a new one. Otherwise, the scripts will create two VPN profiles with the same name, which can only be fixed by using the openvpn3 command line tool directly. Specifically, use the following commands:

openvpn3 sessions-list
openvpn3 session-manage -D --session-path "/net/openvpn/v3/sessions/<SESSION_ID>"
openvpn3 configs-list --verbose
openvpn3 config-remove --path "/net/openvpn/v3/configuration/<CONFIG_ID>"

Running Shell Commands

Save the job definition file and verify it:

scripts/save_job.sh echo-example
scripts/load_job.sh

Then, submit the job:

scripts/submit_task.sh echo-example "hello world" "Echo hello world"

You can remove the job definition file after the job has finished:

scripts/remove_job.sh echo-example

This demo allows running arbitrary shell commands on Omniverse Farm.

Running Isaac Sim Tasks

Built-in Tasks

Save the job definition file and verify it:

scripts/save_job.sh isaac-sim-dummy-example
scripts/load_job.sh

Then, submit the job:

scripts/submit_task.sh isaac-sim-dummy-example "./standalone_examples/api/omni.isaac.core/time_stepping.py" "Isaac Sim Time Stepping"
# or
scripts/submit_task.sh isaac-sim-dummy-example "./standalone_examples/api/omni.isaac.core/simulation_callbacks.py" "Isaac Sim Simulation Callbacks"

You can remove the job definition file after the job has finished:

scripts/remove_job.sh isaac-sim-dummy-example

This demo allows running arbitrary built-in Isaac Sim scripts on Omniverse Farm.

Custom Tasks

This script assumes that the Nucleus server has username admin and password admin. The commands below will fail if the Nucleus server has a different username and password. In this case, refer to the next section on how to setup Nucleus credentials.

Use omnicli to upload the script to Nucleus:

cd thirdparty/omnicli
./omnicli copy "../../tasks/isaac-sim-simulation-example.py" "omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Scripts/isaac-sim-simulation-example.py"
cd ../..

Save the job definition file and verify it:

scripts/save_job.sh isaac-sim-basic-example
scripts/load_job.sh

Then, submit the job:

scripts/submit_task.sh isaac-sim-basic-example \
"/run.sh \
  --download-src 'omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Scripts/isaac-sim-simulation-example.py' \
  --download-dest '/src/isaac-sim-simulation-example.py' \
  --upload-src '/results/isaac-sim-simulation-example.txt' \
  --upload-dest 'omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Results/isaac-sim-simulation-example.txt' \
  './python.sh -u /src/isaac-sim-simulation-example.py 10'" \
  "Isaac Sim Cube Fall"

You can remove the job definition file after the job has finished:

scripts/remove_job.sh isaac-sim-basic-example

This demo allows running arbitrary Isaac Sim scripts on Omniverse Farm by downloading the necessary files, executing the specified command, and then uploading the output files to Nucleus.

Setting Nucleus Credentials

If your Nucleus server have a non-default username and password. Use ./omnicli auth [username] [password] to enter your credentials for uploading files. Alternatively, you can use Omniverse Launcher to perform authentication through a GUI. In addition, use the isaac-sim-nucleus-example.json job description instead to include your username and password. The job description assumes nucleus-secret has been added to the K8s secrets by the admin, including OMNI_USER and OMNI_PASS. Alternatively, if security is not a concern, you may include the username and password directly through the env entry in the job descriptions.

Use omnicli to upload the script to Nucleus:

cd thirdparty/omnicli
./omnicli copy "../../tasks/isaac-sim-simulation-example.py" "omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Scripts/isaac-sim-simulation-example.py"
cd ../..

Save the job definition file and verify it:

scripts/save_job.sh isaac-sim-nucleus-example
scripts/load_job.sh

Then, submit the job:

scripts/submit_task.sh isaac-sim-nucleus-example \
"/run.sh \
  --download-src 'omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Scripts/isaac-sim-simulation-example.py' \
  --download-dest '/src/isaac-sim-simulation-example.py' \
  --upload-src '/results/isaac-sim-simulation-example.txt' \
  --upload-dest 'omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Results/isaac-sim-simulation-example.txt' \
  './python.sh -u /src/isaac-sim-simulation-example.py 10'" \
  "Isaac Sim Cube Fall"

You can remove the job definition file after the job has finished:

scripts/remove_job.sh isaac-sim-nucleus-example

Setting Persistent Volumes

The aforementioned methods only upload the results after the specified command runs successfully, potentially resulting in loss of results if the command fails. To prevent this, you can mount a persistent volume to the container. The isaac-sim-volume-example.json job description assumes that nfs-pv connecting to a storage server through NFS has been added to K8s persistent volume (PV), along with a corresponding nfs-pvc persistent volume claim (PVC) by the admin. This method allows you to keep the partial results even if the command fails.

This NFS setup is preferable for multiple nodes over using volumeMounts.mountPath. The latter mounts the volume to the node where the pod is running, which can become challenging to manage in clusters with multiple nodes.

Use omnicli to upload the script to Nucleus:

cd thirdparty/omnicli
./omnicli copy "../../tasks/isaac-sim-simulation-example.py" "omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Scripts/isaac-sim-simulation-example.py"
cd ../..

Save the job definition file and verify it:

scripts/save_job.sh isaac-sim-volume-example
scripts/load_job.sh

Then, submit the job:

scripts/submit_task.sh isaac-sim-volume-example \
"/run.sh \
  --download-src 'omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Scripts/isaac-sim-simulation-example.py' \
  --download-dest '/src/isaac-sim-simulation-example.py' \
  'ls /mnt/nfs' \
  'mkdir -p /mnt/nfs/results' \
  './python.sh -u /src/isaac-sim-simulation-example.py 10' \
  'cp /results/isaac-sim-simulation-example.txt /mnt/nfs/results/isaac-sim-simulation-example.txt'" \
  "Isaac Sim Cube Fall"

You can remove the job definition file after the job has finished:

scripts/remove_job.sh isaac-sim-volume-example

Note that you can remove the --download-src and --download-dest options if the script is stored in the persistent volume. In addition, the cp command here is only for demonstration purposes, the best practice is to directly write the results in the persistent volume. This can be achieved by making the script accept an additional argument for the output directory.

Running Isaac Lab Tasks

Make sure to follow the Running Isaac Sim Tasks section before moving on to this section.

The demo tasks here assume the aforementioned nuclues-secret and nfs-pvc setup. You can modify the job definition files to include your own credentials and persistent volume claim.

In this section, we only uses the j3soon/omni-farm-isaaclab docker image for simplicity. You can build your own docker image with the necessary dependencies and scripts for your tasks. This will require you to write a custom job definition and optionally copy omnicli when building your docker image.

Built-in Tasks

Save the job definition file and verify it:

scripts/save_job.sh isaac-lab-volume-example
scripts/load_job.sh

Then, submit the job:

scripts/submit_task.sh isaac-lab-volume-example \
"/run.sh \
  --upload-src '/root/IsaacLab/logs' \
  --upload-dest 'omniverse://$NUCLEUS_HOSTNAME/Projects/J3soon/Isaac/4.1/Results/IsaacLab/logs' \
  'ls /mnt/nfs' \
  'mkdir -p /mnt/nfs/results/IsaacLab/logs' \
  'ln -s /mnt/nfs/results/IsaacLab/logs logs' \
  '. /opt/conda/etc/profile.d/conda.sh' \
  'conda activate isaaclab' \
  './isaaclab.sh -p source/standalone/workflows/rl_games/train.py --task Isaac-Ant-v0 --headless'" \
  "Isaac Lab RL-Games Isaac-Ant-v0"

You can remove the job definition file after the job has finished:

scripts/remove_job.sh isaac-lab-volume-example

This demo allows running arbitrary built-in Isaac Lab scripts on Omniverse Farm.

Running Isaac Sim Jobs Locally During Development

For headless tasks, simply follow the official guide.

If your task requires a GUI during development, see this guide.

Refer to scripts/docker for potential useful scripts for running Isaac Sim tasks locally.

Developer Notes

References