Home

Awesome

LILAC – Online Language Corrections for Robotic Manipulation via Shared Autonomy

Open-Source Code Release for "No, to the Right – Online Language Corrections for Robotic Manipulation via Shared Autonomy"

Repository containing scripts for kinesthetic demonstration collection, model definitions (with baselines) in PyTorch, model training code (via PyTorch-Lightning), as well as code for deploying and running models on actual robots (e.g., for a demo or user study).

Uses Anaconda for maintaining python dependencies & reproducibility, and sane quality defaults (black, isort, flake8, precommit). Robot control stack for the Franka Emika Panda arm is built using Polymetis.

Quickstart

Clones lilac to the working directory, then walks through dependency setup using the pinned versions in environments/requirements.txt. If contributing to this repository, please make sure to run pre-commit install before pushing a commit.

We have two sets of installation instructions, mainly for setting up GPU-accelerated PyTorch for training models (this is by no means necessary; models train within 30 minutes on modern CPUs), as well as for setting up a CPU-only version for inference (e.g., what runs on our lightweight robot control computer).

Note: we have written the core of lilac as a Python module, with the setup.py and pyproject.toml providing the minimal information to fully replicate the Python environment used for the original work. For further reproducibility, we define environments/requirements.txt with more explicit pinned versions of the remaining dependencies.

Installation - Linux w/ GPU & CUDA 11.3 - Training

Note: the instructions below assume that conda (Miniconda/Anaconda are both fine) is installed and on your path. However, feel free to use the environment manager of your choosing!

git clone https://<anonymized>/lilac
cd lilac

conda create --name lilac python=3.8
conda activate lilac

# Install PyTorch == 1.11.0, Torchvision == 0.12.0, TorchAudio == 0.11.0 w/ CUDA Toolkit 11.3
#   > Any more recent versions will work as well; just edit the pinned versions in `requirements.txt`
pip install torch==1.11.0+cu113 torchvision==0.12.0+cu113 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/cu113

# Install remaining dependencies via an editable install (make sure you're in the root of this repository!)
#   > Note: if contributing to this repository: run `pip install -e ".[dev]"` for black / flake8 / isort / pre-commit
pip install -e .

Installation - Linux CPU (Intel NUC) - Robot Control Computer

To use Polymetis for the robot control stack, it's highly recommended to follow the conda-based installation instructions here. This will ensure the entire PyTorch ecosystem is installed as well.

Then, assuming a conda environment named lilac:

git clone https://<anonymized>/lilac
cd lilac

# Assumes Polymetis + PyTorch Ecosystem has already been installed...
conda activate lilac

# Install remaining dependencies...
#   > Note: if contributing to this repository: run `pip install -e ".[dev]"` for black / flake8 / isort / pre-commit
pip install -e .

Usage & Entry Points

This repository contains all the steps necessary to collect demonstration data for training LILAC models, as well as the LILA and Language-Conditioned Imitation Learning Baselines, with further code for running online evaluations.

Following the general structure of the method outlined in our paper, we have defined the following four "top-level" scripts that are meant to be run in order:

  1. python scripts/demonstrate.py <task_id> - This script walks through collecting kinesthetic demonstrations for a given task (e.g., water-plant, clean-trash, or towards-shelf from the paper). Note the format that the demonstrations are saved in.

    • After this, make sure to add language annotations following the directions in data/language

  2. python scripts/alphas.py - this script takes the language annotations collected in the prior step, and uses the OpenAI GPT-3 API to perform Alpha annotation, as per Section 4.4 of our paper. Note that this requires a (paid) OpenAI GPT-3 API key.

  3. python scripts/train.py --model <lilac | lila | imitation> ... - this standalone script reads the demonstrations collected in Step 1 and trains the various models compared in this work (you will need to pass a list of tasks you wish to train on). The default hyperparameters are fairly generalizable.

  4. python scripts/evaluate.py --run_directory <run-directory> - once you've trained a model, this script enables online evaluation of the learned policy (if Imitation Learning) or latent actions controller (if LILA/LILAC) on an actual robot.

These 4 steps comprise the full pipeline we use in our work. The user study is conducted as per the description in the paper, using the evaluate.py script.

Note: To run any of the real-robot scripts (e.g., demonstrate.py and evaluate.py), you'll need to run the Robot and Gripper servers on the Franka Control Computer; default launch scripts for reference can be found in scripts/bin/.

Repository Structure

High-level overview of the repository -- besides the entry points described above, the important parts of the codebase are in models/ (architecture, optimizer, and training definitions), and in robot/ (all robot control code).

Questions & Support

We are committed to maintaining this repository for the foreseeable future, across multiple PyTorch and Polymetis versions. If a specific platform/library release is not supported, please post an issue to the Github (or feel free to fork and PR).

For more sensitive queries, please email skaramcheti@cs.stanford.edu.