Awesome

Cloth-Splatting: 3D Cloth State Estimation from RGB Supervision

Project Page| Paper

block

Installation

Docker Image

We use python3.10 and cuda 12.1 for our experiments. In case you want to set up a custom environment, you can use the following commands to create a new conda environment and install the required cuda version.

conda create -n cloth-splatting python=3.10
conda activate cloth-splatting
conda install cuda -c nvidia/label/cuda-12.1.0

For the torch dependencies we use torch 2.2.0.

pip install torch==2.2.0 torchvision --index-url https://download.pytorch.org/whl/cu121

For the installation of the torch_geometric dependencies, for more information refer to the official installation guide.

pip install torch_geometric
pip install pyg_lib torch_scatter torch_sparse torch_cluster torch_spline_conv -f https://data.pyg.org/whl/torch-2.2.0+cu121.html

For the remaining pip dependencies, you can install them using the requirements.txt file.

pip install -r requirements.txt

For the submodules, you can install them using the following commands.

git submodule update --init --recursive
pip install -e submodules/depth-diff-gaussian-rasterization
pip install -e submodules/simple-knn

Data

For synthetic scenes:
The dataset provided here can be used with MD-Splatting to enable novel view synthesis and dense tracking. After downloading the dataset, extract the files to the data folder. The folder structure should look like this:

├── data
│   | final_scenes 
│     ├── scene_1
│     ├── scene_2 
│     ├── ...

Training

To train models for all scenes from the paper, run the following script:

./run_scripts/run_all.sh

Rendering

Run the following script to render images for all scenes.

./run_scripts/render_all.sh

Run Scripts

There are some other useful scripts in the run_scripts directory. Some of it is messy and needs to be cleaned up, but they'll allow you to easily run ablations and log the results.

Contributions

Some source code of ours is borrowed from 3DGS, k-planes,HexPlane, TiNeuVox, 4DGS. We appreciate the excellent works of these authors.

Citation

@inproceedings{
  longhini2024clothsplatting,
  title={Cloth-Splatting: 3D State Estimation from {RGB} Supervision for Deformable Objects},
  author={Alberta Longhini and Marcel B{\"u}sching and Bardienus Pieter Duisterhof and Jens Lundell and Jeffrey Ichnowski and M{\r{a}}rten Bj{\"o}rkman and Danica Kragic},
  booktitle={8th Annual Conference on Robot Learning},
  year={2024},
  url={https://openreview.net/forum?id=WmWbswjTsi}
}