Home

Awesome

<p align="center" width="100%"> <img src='https://github-production-user-asset-6210df.s3.amazonaws.com/44519745/246222783-fdda535f-e132-4fdd-8871-2408cd29a264.png' width="50%"> </p>

OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning

<p align="left"> <a href="https://arxiv.org/abs/2306.11249" alt="arXiv"> <img src="https://img.shields.io/badge/arXiv-2306.11249-b31b1b.svg?style=flat" /></a> <a href="https://github.com/chengtan9907/OpenSTL/blob/master/LICENSE" alt="license"> <img src="https://img.shields.io/badge/license-Apache--2.0-%23002FA7" /></a> <!-- <a href="https://huggingface.co/OpenSTL" alt="Huggingface"> <img src="https://img.shields.io/badge/huggingface-OpenSTL-blueviolet" /></a> --> <a href="https://openstl.readthedocs.io/en/latest/" alt="docs"> <img src="https://readthedocs.org/projects/openstl/badge/?version=latest" /></a> <a href="https://github.com/chengtan9907/OpenSTL/issues" alt="docs"> <img src="https://img.shields.io/github/issues-raw/chengtan9907/SimVPv2?color=%23FF9600" /></a> <a href="https://github.com/chengtan9907/OpenSTL/issues" alt="resolution"> <img src="https://img.shields.io/badge/issue%20resolution-1%20d-%23B7A800" /></a> <a href="https://img.shields.io/github/stars/chengtan9907/OpenSTL" alt="arXiv"> <img src="https://img.shields.io/github/stars/chengtan9907/OpenSTL" /></a> </p>

📘Documentation | 🛠️Installation | 🚀Model Zoo | 🤗Huggingface | 👀Visualization | 🆕News

Introduction

OpenSTL is a comprehensive benchmark for spatio-temporal predictive learning, encompassing a broad spectrum of methods and diverse tasks, ranging from synthetic moving object trajectories to real-world scenarios such as human motion, driving scenes, traffic flow, and weather forecasting. OpenSTL offers a modular and extensible framework, excelling in user-friendliness, organization, and comprehensiveness. The codebase is organized into three abstracted layers, namely the core layer, algorithm layer, and user interface layer, arranged from the bottom to the top. We support PyTorch Lightning implementation OpenSTL-Lightning (recommended) and naive PyTorch version OpenSTL.

<p align="center" width="100%"> <img src='https://github.com/chengtan9907/OpenSTL/assets/34480960/4f466441-a78a-405c-beb6-00a37e3d3827' width="90%"> </p> <!-- <p align="center" width="100%"> <img src='https://github-production-user-asset-6210df.s3.amazonaws.com/44519745/246222226-61e6b8e8-959c-4bb3-a1cd-c994b423de3f.png' width="90%"> </p> --> <p align="right">(<a href="#top">back to top</a>)</p>

Overview

<details open> <summary>Major Features and Plans</summary> </details> <details open> <summary>Code Structures</summary> </details>

News and Updates

[2023-12-15] OpenSTL-Lightning (OpenSTL v1.0.0) is released.

[2023-09-23] The OpenSTL paper has been accepted by NeurIPS 2023 Dataset and Benchmark Track! arXiv / Zhihu.

[2023-06-19] OpenSTL v0.3.0 is released and will be enhanced in #25.

Installation

This project has provided an environment setting file of conda, users can easily reproduce the environment by the following commands:

git clone https://github.com/chengtan9907/OpenSTL
cd OpenSTL
conda env create -f environment.yml
conda activate OpenSTL
python setup.py develop
<details close> <summary>Dependencies</summary> </details>

Please refer to install.md for more detailed instructions.

Getting Started

Please see get_started.md for the basic usage. Here is an example of single GPU non-distributed training SimVP+gSTA on Moving MNIST dataset.

bash tools/prepare_data/download_mmnist.sh
python tools/train.py -d mmnist --lr 1e-3 -c configs/mmnist/simvp/SimVP_gSTA.py --ex_name mmnist_simvp_gsta

Tutorial on using Custom Data

For the convenience of users, we provide a tutorial on how to train, evaluate, and visualize with OpenSTL on custom data. This tutorial enables users to quickly build their own projects using OpenSTL. For more details, please refer to the tutorial.ipynb in the examples/ directory.

We also provide a Colab demo of this tutorial:

<a href="https://colab.research.google.com/drive/19uShc-1uCcySrjrRP3peXf2RUNVzCjHh?usp=sharing" target="_parent"><img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open In Colab"/></a>

<p align="right">(<a href="#top">back to top</a>)</p>

Overview of Model Zoo and Datasets

We support various spatiotemporal prediction methods and provide benchmarks on various STL datasets. We are working on add new methods and collecting experiment results.

<p align="right">(<a href="#top">back to top</a>)</p>

Visualization

We present visualization examples of ConvLSTM below. For more detailed information, please refer to the visualization.

<div align="center">
Moving MNISTMoving FMNIST
<div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-video/mmnist_ConvLSTM.gif' height="auto" width="260" ></div><div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-video/moving_fashionmnist_ConvLSTM.gif' height="auto" width="260" ></div>
Moving MNIST-CIFARKittiCaltech
<div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-video/moving_mnist_cifar_ConvLSTM.gif' height="auto" width="260" ></div><div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-video/kitticaltech_ConvLSTM.gif' height="auto" width="260" ></div>
KTHHuman 3.6M
<div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-video/kth20_ConvLSTM.gif' height="auto" width="260" ></div><div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-video/human_ConvLSTM.gif' height="auto" width="260" ></div>
Traffic - in flowTraffic - out flow
<div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-traffic/taxibj_in_flow_ConvLSTM.gif' height="auto" width="260" ></div><div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-traffic/taxibj_out_flow_ConvLSTM.gif' height="auto" width="260" ></div>
Weather - TemperatureWeather - Humidity
<div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-weather-5-625/weather_temperature_5_625_ConvLSTM.gif' height="auto" width="360" ></div><div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-weather-5-625/weather_humidity_5_625_ConvLSTM.gif' height="auto" width="360" ></div>
Weather - Latitude WindWeather - Cloud Cover
<div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-weather-5-625/weather_wind_latitude_ConvLSTM.gif' height="auto" width="360" ></div><div align=center><img src='https://github.com/chengtan9907/OpenSTL/releases/download/vis-weather-5-625/weather_cloud_cover_5_625_ConvLSTM.gif' height="auto" width="360" ></div>
BAIR Robot PushingKinetics-400
<div align=center><img src='https://github-production-user-asset-6210df.s3.amazonaws.com/44519745/257872182-4f31928d-2ebc-4407-b2d4-1fe4a8da5837.gif' height="auto" width="260" ></div><div align=center><img src='https://github-production-user-asset-6210df.s3.amazonaws.com/44519745/257872560-00775edf-5773-478c-8836-f7aec461e209.gif' height="auto" width="260" ></div>
</div>

License

This project is released under the Apache 2.0 license. See LICENSE for more information.

Acknowledgement

OpenSTL is an open-source project for STL algorithms created by researchers in CAIRI AI Lab. We encourage researchers interested in video and weather prediction to contribute to OpenSTL! We borrow the official implementations of ConvLSTM, PredNet, PredRNN variants, E3D-LSTM, MAU, PhyDNet, MMVP, and SwinLSTM.

Citation

If you are interested in our repository or our paper, please cite the following paper:

@inproceedings{tan2023openstl,
  title={OpenSTL: A Comprehensive Benchmark of Spatio-Temporal Predictive Learning},
  author={Tan, Cheng and Li, Siyuan and Gao, Zhangyang and Guan, Wenfei and Wang, Zedong and Liu, Zicheng and Wu, Lirong and Li, Stan Z},
  booktitle={Conference on Neural Information Processing Systems Datasets and Benchmarks Track},
  year={2023}
}
@inproceedings{gao2022simvp,
  title={Simvp: Simpler yet better video prediction},
  author={Gao, Zhangyang and Tan, Cheng and Wu, Lirong and Li, Stan Z},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={3170--3180},
  year={2022}
}
@article{tan2022simvpv2,
  title={SimVP: Towards Simple yet Powerful Spatiotemporal Predictive Learning},
  author={Tan, Cheng and Gao, Zhangyang and Li, Siyuan and Li, Stan Z},
  journal={arXiv preprint arXiv:2211.12509},
  year={2022}
}
@inproceedings{tan2023temporal,
  title={Temporal attention unit: Towards efficient spatiotemporal predictive learning},
  author={Tan, Cheng and Gao, Zhangyang and Wu, Lirong and Xu, Yongjie and Xia, Jun and Li, Siyuan and Li, Stan Z},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={18770--18782},
  year={2023}
}

Contribution and Contact

For adding new features, looking for helps, or reporting bugs associated with OpenSTL, please open a GitHub issue and pull request with the tag "new features", "help wanted", or "enhancement". Feel free to contact us through email if you have any questions.

<p align="right">(<a href="#top">back to top</a>)</p>