Awesome
9DTact
Table of contents
Overview <a name="overview"></a>
This repository contains the code and the hardware source files for the paper:
<b>9DTact: A Compact Vision-Based Tactile Sensor for Accurate 3D Shape Reconstruction and Generalizable 6D Force Estimation</b> <br> Changyi Lin, Han Zhang, Jikai Xu, Lei Wu, and Huazhe Xu <br> RAL, 2023 <br> Website / Arxiv Paper / Video Tutorial / Bom (CN)
Installation <a name="installation"></a>
Create a conda environment:
conda create -n 9dtact python=3.8
Install pytorch (choose the version that is compatible with your computer):
conda activate 9dtact
conda install pytorch torchvision torchaudio pytorch-cuda=11.8 -c pytorch -c nvidia
In this repository, install the other requirements:
pip install -e .
3D Shape Reconstruction <a name="reconstruction"></a>
For all the terminals used in this section, they are located in the shape_reconstruction directory and based on the 9dtact conda environment:
cd shape_reconstruction
conda activate 9dtact
1. Camera Calibration <a name="camera_calibration"></a>
Before you start, 3d print the calibration board.<br> Run:
python _1_Camera_Calibration.py
Just follow the printed instructions.
2. Sensor Calibration <a name="sensor_calibration"></a>
Firstly, prepare a ball with a radius of 4.0 mm. (The radius of the ball depends on the thickness of the sensor surface. 4.0 mm is just a recommendation.)<br> Then, run:
python _2_Sensor_Calibration.py
Just follow the printed instructions.
3. Shape Reconstruction <a name="shape_reconstruction"></a>
python _3_Shape_Reconstruction.py
6D Force Estimation <a name="estimation"></a>
1. BOTA Driver <a name="bota"></a>
If you want to collect force data with a BOTA MiniONE Pro sensor, you need to:<br> Create a directory named 'bota_ws' as the ROS workspace, and install the bota driver package.
2. Data Collection <a name="collection"></a>
At the first terminal, open the BOTA sensor:
cd ~/xxx/bota_ws # Modify 'xxx' to enter the workspace directory
source devel/setup.bash
roslaunch rokubimini_serial rokubimini_serial.launch
At the second terminal, run:
source ~/xxx/bota_ws/devel/setup.bash
cd data_collection
conda activate 9dtact
python collect_data.py
At the third terminal, open the 9DTact sensor:
cd shape-force-ros
conda activate 9dtact
python _1_Sensor_ros.py
3. Data Processing <a name="processing"></a>
Open a terminal, normalize the wrench:
cd data_collection
conda activate 9dtact
python wrench_normalization.py # remember to modify the object_num
At the same terminal, split the data by running:
python split_train_test.py
and also:
python split_train_test(objects).py
4. Model Training <a name="training"></a>
To train the model on the stadard training dataset, run:
cd force_estimation
python train.py --model_name="Densenet" --model_layer=169 --optimizer="ADAM" --lrs=False --image_type="RGB" --cuda_index=6 --resize_img=False --train_mode=True --test_object=False --mixed_image=True --pretrained=False --batch_size=64 --num_epoch=200 --learning_rate=5.0e-4 --weight_decay=0.0
You may also choose to use Weights and Bias (wandb) by setting use_wandb as True, which helps to track the training performance.
5. Force Estimation <a name="inference"></a>
You need to specify a model saved in the 'saved_models' directory as an estimator, by modifying the 'weights' parameters in the force_config.yaml.<br> After that, run:
cd force_estimation
python _1_Force_Estimation.py
Run in ROS <a name="ros"></a>
1. Shape Reconstruction in ROS <a name="shape_ros"></a>
At the first terminal, open the 9DTact sensor:
cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py
At the second terminal, run:
cd shape-force_ros
conda activate 9dtact
python _2_Shape_Reconstruction_ros.py
2. Force Estimation in ROS <a name="force_ros"></a>
At the first terminal, open the 9DTact sensor:
cd shape-force_ros
conda activate 9dtact
python _1_Sensor_ros.py
At the second terminal, run:
cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py
(Optional for visualization) At the third terminal, open the visualization window:
cd force_estimation
conda activate 9dtact
python force_visualizer.py
3. Simultaneous Shape Reconstruction and Force Estimation (SSAF) in ROS <a name="shape_force"></a>
At the first terminal, open the force estimator:
cd shape-force_ros
conda activate 9dtact
python _3_Force_Estimation_ros.py
At the second terminal, run:
cd shape-force_ros
conda activate 9dtact
python _4_Shape_Force_ros.py