Home

Awesome

MultiGripperGrasp Toolkit

Simulation based grasp filter. This repository contains a grasp filter developed using Isaac Sim, it provides the usage and visualization tools for the MultiGripperGrasp dataset.

MultiGripperGrasp: A Dataset for Robotic Grasping from Parallel Jaw Grippers to Dexterous Hands

Luis Felipe Casas, Ninad Khargonkar, Balakrishnan Prabhakaran, Yu Xiang

[paper] [video] [arXiv] [project site] [dataset folder]

Citing MulitGripperGrasp Toolkit

Please cite MulitGripperGrasp Toolkit if it helps your research:

@misc{casas2024multigrippergraspdatasetroboticgrasping,
  title={MultiGripperGrasp: A Dataset for Robotic Grasping from Parallel Jaw Grippers to Dexterous Hands}, 
  author={Luis Felipe Casas and Ninad Khargonkar and Balakrishnan Prabhakaran and Yu Xiang},
  year={2024},
  eprint={2403.09841},
  archivePrefix={arXiv},
  primaryClass={cs.RO},
  url={https://arxiv.org/abs/2403.09841}, 
 }

License

MultiGripperGrasp Toolkit is released under the GNU General Public License v3.0.

Contents

Prerequisites

This code was tested with Isaac Sim 2023.1.0 on Linux.

Installation

  1. Clone the repo:

    git clone git@github.com:IRVLUTD/isaac_sim_grasping.git

  2. Install the Isaac Sim 2023.1.0 and cd to its python directory: Follow the instructions in the Isaac Sim Website for proper installation. Normally the python.sh is on the following directory:

    cd ~/.local/share/ov/pkg/isaac_sim-2023.1.0

  3. Install the required libraries using the Isaac Sim python environment:

     ./python.sh -m pip install tqdm

    Note: Only tqdm must be installed, all other libraries should come with the Isaac Sim version.

  4. Download the MultiGripperGrasp dataset and the object .usd models from the dataset folder. Links can also be found at the end of this document.

Visualizing Dataset Samples

To visualize the dataset a visualization standalone is included in the repository. The script must be ran with the Isaac Sim python.sh. The script requires the following parameters:

Note: To run the simulation without warnings add the following parameters to the command:

--/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Example command to run the visualization script:

./python.sh (repo directory)/visualize_grasps.py --json_dir=(dataset .json folder) --gripper_dir=(repo directory)/grippers --objects_dir=(object .usd folder) --num_w=10 --ub=3 --lb=0 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error
<p align="center"> <img src='media/visualize.gif'> </p>

Alternate Visualization Tool

Please refer to the scripts under the viz_optas folder for an alternative way to visualize the grasps without the IsaacSim dependency!

Running the Simulation

The simulation can use any gripper and object provided they are prepared correctly (See the documentation to add grippers and objects). It loads the grasp information from the files specified and creates multiple "workstations" to test all the grasps. Then, it tries to perform the grasps with the specified control routines. When the object falls or the testing time is up, the time they took to fall is recorded and then saved to the output file. If any grasps is unable to touch the object, it will be recorded as a negative "fall time" value. Once the grasp test is completed, the workstation will reset and load a new grasp for testing. When all the grasps are finished, the output file is saved, the complete simulation will reset and a new file loaded.

<p align="center"> <img src='media/transfer_far.gif'> </p>

Important Files/Folders description

  1. standalone.py: Graspit grasps standalone executable
  2. transfer_st.py: Transfered grasps standalone executable
  3. visualize_grasps.py: Standalone executable to visualize filtered grasps
  4. views.py: Simulation's behavioral code
  5. managers.py: Contains grasp information and the reporting of results for all the standalones
  6. controllers.py: Developed gripper controllers
  7. utils.py: General utility functions
  8. grippers: Gripper .usd files

Parameters and Inputs

A standalone executable for the graspit grasps (standalone.py) and for transferred grasps (transfer_st.py) are included within the repository. A sample command to run the graspit grasps is shown below along with the available options. Note: for Isaac Sim standalone executables, the commands must be run from the isaac sim python.sh directory.

The standalone.py takes as input:

Note: To run the simulation without warnings add the following parameters to the command:

 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Sample run command:

./python.sh (repo directory)/standalone.py --json_dir=(.json files folder) --gripper_dir=(repo directory)/grippers --objects_dir=(object .usd folder) --output_dir=(output directory) --num_w=100 --test_time=3 --controller=position --headless --print_results --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error
<p align="center"> <img src='https://github.com/IRVLUTD/isaac_sim_grasping/blob/main/media/robotiq_Clock.gif'> </p>

Transferred Grasps

The grasp filter is able to evaluate the object fall-off time for a large amount of generated grasps. The successful grasps of one gripper can represent successful grasps in others and increase the overall amount of successful grasps in the dataset. To test this hypothesis, we implemented the grasp transfer of successful grasps from one gripper to others and evaluated the transferred grasps using antoher Isaac Sim standalone (transfer_st.py).

We utilize an alignment between grippers to transfer grasps, which used a common notion of gripper pose (translation and orientation). The translation refers to the palm center of the gripper, and the orientation is with respect to a canonical pose of the gripper palm pointing in a fixed direction. Thus, any grasp pose from a gripper was transferred to another by using this pose alignment. We first transform a source gripper pose to its aligned pose, and then transform the aligned pose to the target gripper.

<p align="center"> <img src='media/transfer_grasp.png' width='1000'> </p>

The transfer_st.py takes as input:

Note: To run the simulation without warnings add the following parameters to the command:

 --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error

Sample run command:

./python.sh (repo directory)/transfer_st.py --json_dir=(.json files folder) --gripper_dir=(repo directory)/grippers --objects_dir=(object .usd folder) --output_dir=(output directory) --num_w=100 --test_time=3 --headless --print_results --/log/level=error --/log/fileLogLevel=error --/log/outputStreamLevel=error
<p align="center"> <img src='media/transfer_close.gif' width='1000'> </p>

Reproducing Paper Results

As of this version, the simulation uses a specific .json structure. The managers.py file is the responsible for reading and transforming the relevant grasp information to the format used by the program. The information is:

Additionally within the gripper_isaac_info.json you can find the configuration parameters used for each gripper. In order to reproduce the paper results, one must run the simulation with the same values. A detailed description of each field can be found within the Adding a new Gripper page. All the files provided within this repository and in the [dataset folder] are set up for use out of the box. To reproduce the results you will need to run the simulation on the dataset and save to a different folder (output_dir). This will override the fall_time values and save the latest results on the new file with the relevant information: grasp pose information, grasp dof information, test type, total test time, fall time.

Note: To reproduce the transferred grasp results the same must be done with the transfer parameters in gripper_isaac_info.json.

More Documentation

Helpful Links

Notes