Home

Awesome

9DTact

Table of contents

  1. Overview
  2. Installation
  3. 3D Shape Reconstruction
    1. Camera Calibration
    2. Sensor Calibration
    3. Shape Reconstruction
  4. 6D Force Estimation
    1. BOTA Driver
    2. Data Collection
    3. Data Processing
    4. Model Training
    5. Force Estimation
  5. Run in ROS
    1. Shape Reconstruction in ROS
    2. Force Estimation in ROS
    3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS

Overview <a name="overview"></a>

This repository contains the code and the hardware source files for the paper:

<b>9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation</b> <br> Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu <br> RAL, 2023 <br> Website / Arxiv Paper / Video Tutorial / Bom (CN)

Installation <a name="installation"></a>

Create a conda environment:

conda create -n 9dtact python=3.8

Install pytorch (choose the version that is compatible with your computer):

conda activate 9dtact
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia

In this repository, install the other requirements:

pip install -e .

3D Shape Reconstruction <a name="reconstruction"></a>

For all the terminals used in this section, they are located in the shape_reconstruction directory and based on the 9dtact conda environment:

cd shape_reconstruction
conda activate 9dtact

1. Camera Calibration <a name="camera_calibration"></a>

Before you start, 3d print the calibration board.<br> Run:

python _1_Camera_Calibration.py

Just follow the printed instructions.

2. Sensor Calibration <a name="sensor_calibration"></a>

Firstly, prepare a ball with a radius of 4.0 mm. (The radius of the ball depends on the thickness of the sensor surface. 4.0 mm is just a recommendation.)<br> Then, run:

python _2_Sensor_Calibration.py

Just follow the printed instructions.

3. Shape Reconstruction <a name="shape_reconstruction"></a>

python _3_Shape_Reconstruction.py

6D Force Estimation <a name="estimation"></a>

1. BOTA Driver <a name="bota"></a>

If you want to collect force data with a BOTA MiniONE Pro sensor, you need to:<br> Create a directory named 'bota_ws' as the ROS workspace, and install the bota driver package.

2. Data Collection <a name="collection"></a>

At the first terminal, open the BOTA sensor:

cd ~/xxx/bota_ws # Modify 'xxx' to enter the workspace directory
source devel/setup.bash
roslaunch rokubimini_serial rokubimini_serial.launch

At the second terminal, run:

source ~/xxx/bota_ws/devel/setup.bash
cd data_collection
conda activate 9dtact
python collect_data.py

At the third terminal, open the 9DTact sensor:

cd shape-force-ros
conda activate 9dtact
python _1_Sensor_ros.py

3. Data Processing <a name="processing"></a>

Open a terminal, normalize the wrench:

cd data_collection
conda activate 9dtact
python wrench_normalization.py  # remember to modify the object_num

At the same terminal, split the data by running:

python split_train_test.py

and also:

python split_train_test(objects).py

4. Model Training <a name="training"></a>

To train the model on the stadard training dataset, run:

cd force_estimation
python train.py --model_name="Densenet" --model_layer=169 --optimizer="ADAM" --lrs=False --image_type="RGB" --cuda_index=6 --resize_img=False --train_mode=True --test_object=False --mixed_image=True --pretrained=False --batch_size=64 --num_epoch=200 --learning_rate=5.0e-4 --weight_decay=0.0

You may also choose to use Weights and Bias (wandb) by setting use_wandb as True, which helps to track the training performance.

5. Force Estimation <a name="inference"></a>

You need to specify a model saved in the 'saved_models' directory as an estimator, by modifying the 'weights' parameters in the force_config.yaml.<br> After that, run:

cd force_estimation
python _1_Force_Estimation.py

Run in ROS <a name="ros"></a>

1. Shape Reconstruction in ROS <a name="shape_ros"></a>

At the first terminal, open the 9DTact sensor:

cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python _2_Shape_Reconstruction_ros.py

2. Force Estimation in ROS <a name="force_ros"></a>

At the first terminal, open the 9DTact sensor:

cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py

(Optional for visualization) At the third terminal, open the visualization window:

cd force_estimation
conda activate 9dtact
python force_visualizer.py

3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS <a name="shape_force"></a>

At the first terminal, open the force estimator:

cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py

At the second terminal, run:

cd shape-force_ros
conda activate 9dtact
python _4_Shape_Force_ros.py