Home

Awesome

Equivariant Descriptor Fields with ROS MoveIt and PyBullet

Example codes for Equivariant Descriptor Fields with and ROS MoveIt in PyBullet Simulation Environment.
Please visit https://github.com/tomato1mule/edf for more information on Equivariant Descriptor Fields.

Installation

Step 1. Clone Github repository. It is important to clone RECURSIVELY.

git clone --recursive https://github.com/tomato1mule/edf_pybullet_ros_experiment

Step 2. Setup Conda/Mamba environment. We recommend using Mamba for faster installation.

# if you don't have mamba yet, install it first (not needed when using mambaforge):
conda install mamba -c conda-forge

# now create a new environment
mamba create -n edf_exp python=3.8
conda activate edf_exp

Step 3. Install EDF.

# go to edf folder
cd edf


# install edf and dependencies
CUDA=cu113
pip install torch==1.11.0 torchvision==0.12.0 torchaudio==0.11.0 --extra-index-url https://download.pytorch.org/whl/${CUDA}
pip install torch-cluster==1.6.0 -f https://data.pyg.org/whl/torch-1.11.0+${CUDA}.html
pip install torch-scatter==2.0.9 -f https://data.pyg.org/whl/torch-1.11.0+${CUDA}.html
pip install iopath fvcore
pip install --no-index --no-cache-dir pytorch3d==0.7.2 -f https://dl.fbaipublicfiles.com/pytorch3d/packaging/wheels/py38_${CUDA}_pyt1110/download.html
pip install -e .


# come back to root folder
cd ..

Step 4. Install ROS (Noetic) on conda via robostack

# Visit these links if you have trouble installing robostack.
# 1) https://robostack.github.io/GettingStarted.html
# 2) https://github.com/RoboStack/ros-noetic


# this adds the conda-forge channel to the new created environment configuration and the robostack channels
conda config --env --add channels conda-forge
conda config --env --add channels robostack
conda config --env --add channels robostack-experimental
conda config --env --add channels robostack-humble

# There is a bug with cryptography==39.0.0, so please downgrade it if you face OpenSSL related issues.
# https://stackoverflow.com/questions/74981558/error-updating-python3-pip-attributeerror-module-lib-has-no-attribute-openss
pip install cryptography==38.0.4

# Install the version of ROS you are interested in:
mamba install ros-noetic-desktop

# optionally, install some compiler packages if you want to e.g. build packages in a colcon_ws:
mamba install compilers cmake pkg-config make ninja

# on linux and osx (but not Windows) you might want to:
mamba install catkin_tools

# only on linux, if you are having issues finding GL/OpenGL, also do:
mamba install mesa-libgl-devel-cos7-x86_64 mesa-dri-drivers-cos7-x86_64 libselinux-cos7-x86_64 libxdamage-cos7-x86_64 libxxf86vm-cos7-x86_64 libxext-cos7-x86_64 xorg-libxfixes



# Reboot Conda env
conda deactivate
conda activate edf_exp

# if you want to use rosdep, also do:
mamba install rosdep
rosdep init  # note: do not use sudo!
rosdep update

Step 5. Install Moveit and other ROS packages

mamba install ros-noetic-moveit=1.1.0 ros-noetic-moveit-ros-perception=1.1.0 ros-noetic-ros-controllers=0.18.1 ros-noetic-ros-control=0.19.4 ros-noetic-ros-numpy=0.0.4

Step 6. Configure Catkin workspace for ros_edf_interface package

# Go to ros_edf_interface folder to setup ros workspace environment
cd ros_edf_interface
catkin_make

# This script will make ros workspace setups to be auto-loaded when conda env is activated
bash conda_env_setup.sh

# Come back to root folder
cd ..

# Reboot Conda env
conda deactivate
conda activate edf_exp

Step 7. Install PyBullet Simulation Environment

# Move to install edf_env
cd edf_env
pip install -e .

# Come back to root folder
cd ..

# Reboot Conda env
conda deactivate
conda activate edf_exp

Step 8. (Optional) Set .env file for Jupyter notebook

bash create_jupyter_dotenv.sh
pip install python-dotenv

Usage

A) Collect Human Demo

1. First, run Pybullet environment

bash bringup_edf_env_ros.sh

2. In another terminal, run demo-collection web server

# flag --dont-save if you don't want to save demonstrations.
# flag -n=<number> to set number of demonstrations to collect.
python collect_human_demo.py -n=10 --save-dir="demo/mug_demo"

3. Go to http://127.0.0.1:8050/ in your web browser and collect demonstrations.

Troubleshooter

4. Once all the demonstrations are collected, the 'Robot State' will inform that the demonstrations are saved to certain directions.

B) Train

python pick_train.py
python place_train.py

C) Evaluation

0 (Optional). If you want to use already trained weights, just rename 'checkpoint_example' folder to 'checkpoint'

1. First, run Pybullet environment in an evaluation setup (Unseen pose/instance/distractors)

bash bringup_edf_env_ros_eval.sh

2. In another terminal, run the inference code

python edf_pick_and_place.py

Note Our motion planning pipeline is currently very unstable. Therefore, even if EDFs sampled correct pick-and-place poses, these poses may get rejected as no motion plans can be found by the planner. Plus, we did not fine-tune hyperparameters for this new environments (The robot and perception pipelines are different from the original experiments).

As a result, the success rate in this implementation is lower than the rate reported in the paper. Still, the success rate should be reasonably high (about 7~80%). If the success rate is lower than this rate, something must have gone wrong.

To reproduce the original experimental results in the paper, please check the following branch: https://github.com/tomato1mule/edf/tree/iclr2023_rebuttal_ver