Home

Awesome

Tacchi

This repository contains the code for the paper Tacchi: A Pluggable and Fast Gel Deformation Simulator for Optical Tactile Sensors. The code can be used for connection simulation with MuJoCo and Gazebo, Tacchi image generation, image quality comparison, and Sim2Real tasks. All the datasets generated by this repository can be found in Tacchi_dataset. The presentation video is available in Tacchi_video.

<img src="https://github.com/zixichen007115/Tacchi/blob/main/gif/Tacchi.gif" height="250px"> <img src="https://github.com/zixichen007115/Tacchi/blob/main/gif/MuJoCo.gif" height="250px"> <img src="https://github.com/zixichen007115/Tacchi/blob/main/gif/images.gif" height="250px">

Dependencies

0_stl2npy

This package generates object files for Tacchi. To generate objects with different particles,

python stl2ply.py

python ply2pcd.py --particle [particle number]

python pcd2npy.py

1_connection_simulation

This package connects MuJoCo and Gazebo with Tacchi for simulation. The objects generated by 0_stl2npy should be put in this package.

2_Tacchi_image_generation

This package generates images with Tacchi. The objects generated by 0_stl2npy should be put in this package.

python cmd.py --particle [particle number]

3_align_and_comparison

This package is from the repository Gelsight Simulation. This package can align images and evaluate simulation images. The images form 2_Tacchi_image_generation should be put in this package. To align images,

python -m experiments.preparation.align_per_object

python -m experiments.preparation.generate_sim

To evaluate simulation images,

python -m experiments.preparation.loss_eval

4_AA21

This package applied depth map smoothing method applied in [18] to generate tactile images. To generate images,

python data_loader_AA21.py

5_Sim2Real

This package applied ResNet18 to classifier images and evaluate images with Sim2Real tasks. The images form real experiments, Gelsight Simulation, and 2_Tacchi_image_generation should be put in this package. To train a ResNet,

python main.py --mode train --img_kind [image kind] --model_name [model name]

To test a classifier with real images,

python main.py --mode test --img_kind real --model_name [model name]

Citation

We ask that any publications which use Tacchi cite as following:

@article{chen2023tacchi,
  title={Tacchi: A pluggable and low computational cost elastomer deformation simulator for optical tactile sensors},
  author={Chen, Zixi and Zhang, Shixin and Luo, Shan and Sun, Fuchun and Fang, Bin},
  journal={IEEE Robotics and Automation Letters},
  volume={8},
  number={3},
  pages={1239--1246},
  year={2023},
  publisher={IEEE}
}

Thank Daniel Fernandes Gomes for Gelsight rendering method, alignment method and image evaluation code.