Home

Awesome

YCB-Sight: A visuo-tactile dataset for object understanding

CC BY-SA 4.0   License: MIT        <img height="20" src="media/robotouch.png" alt="Robotouch-logo" />    <img height="20" src="media/rpl.png" alt="RPL-logo" />

cover

YCB-Sight is a visuo-tactile dataset including the simulated and real data from a GelSight tactile sensor and Kinect Azure RGB-D camera on the YCB dataset.

Dataset

You can find the whole dataset here, or download partial data below

YCBSight-Sim

cover

Simulated tactile and depth data with Taxim and pyrender

Object NameSize (MB)Link
002_master_chef_can64.3[Link]
003_cracker_box63.2[Link]
004_sugar_box61.2[Link]
005_tomato_soup_can63.8[Link]
006_mustard_bottle63.8[Link]
007_tuna_fish_can63.2[Link]
008_pudding_box61.5[Link]
009_gelatin_box60.3[Link]
010_potted_meat_can62.8[Link]
011_banana63.7[Link]
012_strawberry64.2[Link]
013_apple63.4[Link]
014_lemon63.4[Link]
017_orange63.2[Link]
019_pitcher_base64.5[Link]
021_bleach_cleanser62.6[Link]
024_bowl65.1[Link]
025_mug64.2[Link]
029_plate66.3[Link]
035_power_drill64.7[Link]
036_wood_block60.1[Link]
037_scissors64.2[Link]
042_adjustable_wrench64.7[Link]
043_phillips_screwdriver63.9[Link]
048_hammer64.1[Link]
055_baseball63.5[Link]
056_tennis_ball63.4[Link]
072-a_toy_airplane65.5[Link]
072-b_toy_airplane63.8[Link]
077_rubiks_cube61.3[Link]

YCBSight-Real

cover

Collected tactile and depth data from real world experiments

Object NameSize (GB)Link
002_master_chef_can0.97[Link]
004_sugar_box1.15[Link]
005_tomato_soup_can1.09[Link]
010_potted_meat_can1.09[Link]
021_bleach_cleanser1.23[Link]
036_wood_block1.02[Link]

Data directory format

YCBSight-Sim
├── obj1
│   ├── gt_contact_mask
│   │   ├── <idx>.npy
│   │   └── ...
│   ├── gt_height_map
│   │   ├── <idx>.npy
│   │   └── ...
│   ├── gelsight
│   │   ├── <idx>.jpg
│   │   └── ...
│   ├── pose.txt
│   ├── depthCam.npy
│   └── depthCam.pdf
├── obj2
└── ...
YCBSight-Real
├── obj1
│   ├── gelsight
│   │   ├── gelsight_<idx>_<timestamp>.jpg
│   │   └── ...
│   ├── depth
│   │   └── depth_0_<timestamp>.tif
│   ├── pc
│   │   └── pc_0_<timestamp>.npy
│   ├── rgb
│   │   ├── rgb_<idx>_<timestamp>.jpg
│   │   └── ...
│   ├── robot.csv
│   ├── tf.json
│   └── obj1.mp4
├── obj2
└── ...

Dependencies

The visualization and data processing are implemented in python3 and require numpy, scipy, matplotlib, cv2.

To install dependencies: pip install -r requirements.txt.

Data Visualization

Local Shape Reconstruction from Touch with Lookup Table

Local Shape Reconstruction from Touch with FCRN network

Please refer to this repo (pytorch version) and this repo (tensorflow version).

License

This dataset is licensed under a Creative Commons Attribution-ShareAlike 4.0 International License, with the accompanying processing code licensed under the MIT License.

Citation

If you use YCB-Sight dataset in your research, please cite:

@article{suresh2021efficient,
  title={Efficient shape mapping through dense touch and vision},
  author={Suresh, Sudharshan and Si, Zilin and Mangelson, Joshua G and Yuan, Wenzhen and Kaess, Michael},
  journal={arXiv preprint arXiv:2109.09884},
  year={2021}
}