Awesome

📖 Table of Contents

• 📖 Table of Contents
• 📍 Overview
• :key: Key Features
  • Allsight Sim
  • SightGAN
• 📂 Contents
• 🚀 Getting Started
  • 🔧 Installation
  • 🤖 Running allsight_sim2real
• 📄 License
• 👏 Acknowledgments
• :clipboard: BibTeX

📍 Overview

This repository contains the code and resources related to the paper "Augmenting Tactile Simulators with Real-like and Zero-Shot Capabilities" (under review). The paper introduces SightGAN, a cutting-edge solution for enhancing tactile perception in high-resolution tactile sensors such as allsight sensor.

:key: Key Features

Allsight Sim

The simulation data was obtained using tha allsight_sim package that utilizing TACTO a physics-engine simulator for optical-based tactile sensors.

For more information about the simulation package please see the link provided above.

• Example for data collection session

SightGAN

A bi-directional Generative Adversarial Network which built upon CycleGAN and designed to bridge the reality gap between simulated and real tactile data, particularly for high-resolution tactile sensors.

SightGAN introduces contact-specific consistency losses:

• Spatial Contact Consistency loss
• Pixel-wise Contact Region Consistency loss

The GAN diagram and training process can be represented as follow:

• Example of SightGAN integrated in the simulation

📂 Contents

This repository contains several directories with the details as follow:

All the rest of the files were forked from CycleGAN and helps for building and managing the models during training and test procedures.

A Python package for training the spatial contact estimator and inherit the models.

Python files forked from CycleGAN repo

🚀 Getting Started

Dependencies

Please ensure you have the following dependencies installed on your system.

Project was tested on:

• Ubuntu 18/20
• python >= 3.8

🔧 Installation

1. Clone the allsight_sim2real repository:

git clone https://github.com/RobLab-Allsight/allsight_sim2real

2. Change to the project directory:

cd allsight_sim2real

3. Install the dependencies:

pip install -r requirements.txt

🤖 Running allsight_sim2real

NOTE: Be aware for the pathes adjusts and adaptation for your custom dataset.

NOTE: Please Be aware for the arguments adjusts needed for every script.

Assuming you have a dataset folder and updated package with the relevant pathes to your dataset:

Filter the real data:

python3 data_process/filter_real_images.py 

Merge sim data:

python3 data_process/merge_json_sim.py 

Add diff frame:

python3 data_process/add_diff_frame.py

Transfer images to gan folders:

python3 data_process/transfer_images.py 

Train CycleGAN:

python3 train.py <arguments> 

📄 License

This repository is licensed under the MIT License. Feel free to use, modify, and distribute the code as per the terms of this license.

👏 Acknowledgments

• Our code is inspired by CyclGAN.
• Simulation was implemented using TACTO.

:clipboard: BibTeX

@misc{azulay2023augmenting,
      title={Augmenting Tactile Simulators with Real-like and Zero-Shot Capabilities}, 
      author={Osher Azulay and Alon Mizrahi and Nimrod Curtis and Avishai Sintov},
      year={2023},
      eprint={2309.10409},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

↑ Return