Home

Awesome

UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model (CVPR 2024)

Python 3.10.9 PyTorch 2.2.0a0 CUDA 12.0

This repository contains the PyTorch implementation of our paper titled UnSAMFlow: Unsupervised Optical Flow Guided by Segment Anything Model, accepted by CVPR 2024.

Authors: Shuai Yuan, Lei Luo, Zhuo Hui, Can Pu, Xiaoyu Xiang, Rakesh Ranjan, Denis Demandolx.

demo image

Disclaimer

Our code is developed on our internal AI platform and has not been tested on regular linux systems. Some of the code depends on internal tools and packages that we cannot share here, but we will talk about ways to work around.

Datasets

Due to copyright issues, please download the dataset from the official websites.

Segment Anything Model

Please follow the official repo to infer SAM masks for all samples. We use the default ViT-H SAM model. The code for generating full segmentation and finding key objects from SAM masks is also included in sam_inference.py.

Usage

We provide scripts and code to run each of our experiments. Before running the experiments, please redefine the input/output directories in the scripts. For each row in our final result tables (Tabs. 1-2 in the paper), please run the following to reproduce those results.

Ours (baseline):

# KITTI
python3 train.py -c configs/kitti_base.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

# Sintel
python3 train.py -c configs/sintel_base.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

Ours (+aug):

# KITTI
python3 train.py -c configs/kitti_aug.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

# Sintel
python3 train.py -c configs/sintel_aug.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

Ours (+aug +hg):

# KITTI
python3 train.py -c configs/kitti_aug+hg.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

# Sintel
python3 train.py -c configs/sintel_aug+hg.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

Ours (+aug +hg +mf):

# KITTI
python3 train.py -c configs/kitti_aug+hg+mf.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

# Sintel
python3 train.py -c configs/sintel_aug+hg+mf.json --n_gpu=N_GPU --exp_folder=EXP_FOLDER

Code credits

The overall structure of this code is adapted from the official SemARFlow github repo, appeared in their publication SemARFlow: Injecting Semantics into Unsupervised Optical Flow Estimation for Autonomous Driving.

License

The majority of UnSAMFlow is licensed under CC-BY-NC, however portions of the project are available under separate license terms: SemARFlow and ARFlow are licensed under the MIT license.

Copyright (c) Meta Platforms, Inc. and affiliates.