Awesome
MAVOS
Efficient Video Object Segmentation via Modulated Cross-Attention Memory
<p align="center"> <img src="https://i.imgur.com/waxVImv.png" alt="Oryx Video-ChatGPT"> </p>Abdelrahman Shaker, Syed Talal Wasim, Martin Danelljan, Salman Khan, Ming-Hsuan Yang and Fahad Khan
Mohamed bin Zayed University of AI, ETH Zurich, University of California - Merced, Yonsei University, Google Research, Linköping University
<!-- [![Website](https://img.shields.io/badge/Project-Website-87CEEB)](site_url) -->Latest Updates
2024/09/02
: MAVOS checkpoints, training, and evaluation code are now available.2024/08/30
: MAVOS has been accepted at WACV 2025! 🎊2024/03/27
: Our technical report on MAVOS has been published on arXiv.
Intro
- MAVOS is a transformer-based VOS method that achieves real-time FPS and reduced GPU memory for long videos.
- MAVOS increases the speed by 7.6x over the baseline DeAOT, while significantly reducing the GPU memory by 87% on long videos with comparable segmentation performance on short and long video datasets.
Examples
https://github.com/user-attachments/assets/ca2902ed-0b82-4129-89c3-3824c782818a
https://github.com/user-attachments/assets/d4d5b77c-5fa3-4fbb-a94c-df7f8ccb3413
<br>Requirements
- Python3
- pytorch >= 1.7.0 and torchvision
- opencv-python
- Pillow
- Pytorch Correlation. Recommend to install from source instead of using
pip
:git clone https://github.com/ClementPinard/Pytorch-Correlation-extension.git cd Pytorch-Correlation-extension python setup.py install cd -
Model Zoo
Pre-trained models of our project can be found in MODEL_ZOO.md.
Getting Started
-
Prepare a valid environment follow the requirements.
-
We use the pre-trained weights of DeAOT-L on static images as baseline (Recommended). No need for pretraining. If you want to pre-train MAVOS from scratch, consider the following dataset preperation:
-
Prepare datasets:
Please follow the below instruction to prepare datasets in each corresponding folder.
-
Static
datasets/Static: pre-training dataset with static images. Guidance can be found in AFB-URR, which we referred to in the implementation of the pre-training.
-
YouTube-VOS
A commonly-used large-scale VOS dataset.
datasets/YTB/2019: version 2019, download link.
train
is required for training. -
DAVIS
A commonly-used small-scale VOS dataset.
datasets/DAVIS: TrainVal (480p) contains both the training and validation split. Test-Dev (480p) contains the Test-dev split. The full-resolution version is also supported for training and evaluation but not required.
-
-
Prepare ImageNet pre-trained encoders
Select and download below checkpoints into pretrain_models:
- MobileNet-V2 (default encoder)
- ResNet-50
- Swin-Base
-
-
Training: the training script will fine-tune the pre-trained models using 4 GPUs on both
YouTube-VOS 2019 train
andDAVIS-2017 train
, resulting in a model that can generalize to different domains. -
Evaluation : the evaluation script will evaluate the models on LVOS, DAVIS, and LTV. The results will be packed into Zip files. For calculating scores, please use official LVOS toolkit (for Val), DAVIS toolkit (for Val). For the Long-Time Video dataset, use the same DAVIS toolkit and replace --davis_path to long_video videos with the corresponding annotations.
Results on Long videos benchmarks
LVOS val set
<img src="source/LVOS.png" width="90%"/>LTV
<img src="source/LTV.png" width="90%"/>Acknowledgment
Our code base is based on the AOT repository. We thank the authors for their open-source implementation.
The computations were enabled by resources provided by the National Academic Infrastructure for Supercomputing in Sweden (NAISS) at Alvis partially funded by the Swedish Research Council through grant agreement no. 2022-06725, the LUMI supercomputer hosted by CSC (Finland) and the LUMI consortium, and by the Berzelius resource provided by the Knut and Alice Wallenberg Foundation at the National Supercomputer Centre.
Citations
Please consider citing our paper in your publications if it helps your research.
@article{Shaker2024MAVOS,
title={Efficient Video Object Segmentation via Modulated Cross-Attention Memory},
author={Shaker, Abdelrahman and Wasim, Syed and Danelljan, Martin and Khan, Salman and Yang, Ming-Hsuan and Khan, Fahad Shahbaz},
journal={arXiv:2403.17937},
year={2024}
}
License
This project is released under the BSD-3-Clause license. See LICENSE for additional details.