Home

Awesome

Click to Move

Click to Move: Controlling Video Generation with Sparse Motion

Pytorch implementation of our paper Click to Move: Controlling Video Generation with Sparse Motion In ICCV 2021. Please cite with the following Bibtex code:

@inproceedings{ardino2021click,
  title={Click to Move: Controlling Video Generation with Sparse Motion},
  author={Ardino, Pierfrancesco and De Nadai, Marco and Lepri, Bruno and Ricci, Elisa and Lathuili{\`e}re, St{\'e}phane},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  pages={14749--14758},
  year={2021}
}

Please follow the instructions to run the code.

Scripts

1. Installation

conda env create -f c2m.yml
conda activate c2m
cd cityscapesScripts
pip install -e .

WIP

2. Data Preprocessing

2.1 Generate instance segmentation

We apply a modified version of Panoptic-deeplab to get the corresponding semantic and instance maps. You can find it into panoptic_deeplab folder. For this work we have used the HRNet backbone. You can download it from here.

Cityscapes

2.2 Generate object trajectories

3 Train the model

We store the configuration of the model as a YAML configuration file. You can have a look at a base configuration in src/config/c2m_journal_cityscapes.yaml. The training file takes as input the following parameters:

Our code support multi-gpu training using DistributedDataParallel. Here's an example of how you can run the code with one or more gpus.

Single gpu

python train.py --device_ids 0 --config config/c2m_journal_cityscapes.yaml

Multi gpu

python -m torch.distributed.run --nnodes=1 --nproc_per_node=2 train.py --device_ids 0,1 --config config/c2m_journal_cityscapes.yaml The example considers a scenario with a single node and two gpus per node. Please change according to your needs. For more information check the DDP example

4 Test the model

python test.py --device_ids 0 --config config/c2m_journal_cityscapes.yaml