Home

Awesome

<p align="center"> <img src="static/logo.png" width="150" style="margin-bottom: 0.2;"/> <p> <h2 align="center"> <a href="http://arxiv.org/abs/2410.06234">TEOChat: Large Language and Vision Assistant for Temporal Earth Observation Data</a></h2> <h5 align="center"> If you like our project, please give us a star ⭐ on GitHub for latest update. </h2> <h5 align="center">

hf_dataset hf_model hf_space <br> arXiv License Hits

</h5>

📰 News

Table of Contents

😮 Highlights

TEOChat is the first language and vision assistant that can engage in conversation about sequences of temporal earth observation imagery, and exhibits impressive performance on multiple temporal instruction-following tasks.

<img src="assets/figure1.png"/>

📚 TEOChatlas: A new instruction-following dataset for temporal EO data

We introduce a new instruction-following dataset for temporal EO data called TEOChatlas which we use to train TEOChat. TEOChatlas contains 554,071 examples spanning dozens of temporal instruction-following tasks.

<img src="assets/figure2.png"/>

🤖 TEOChat: A new vision-language model for temporal EO data

We design TEOChat to use a LLaVA-style architecture, combining a temporally shared vision encoder with a LLaMA 2 LLM connected through an MLP vision-language projector

<img src="assets/figure3.png"/>

🤗 Demo

Gradio Web UI

We provide an online demo in Huggingface Spaces.

You can also run the demo locally by running the following command:

python videollava/serve/teochat_demo.py 
<img src="assets/demo.gif" width="500" />

🚀 Main Results

We demonstrate that TEOChat:

Temporal Tasks

<p align="left"> <img src="assets/table1.png" width=80%> </p>

Zero-shot Temporal Tasks and Comparison with Proprietary Foundation Models

<p align="left"> <img src="assets/table3-4.png" width=80%> </p>

Single Image Tasks

<p align="left"> <img src="assets/table5.png" width=80%> </p>

🛠️ Requirements and Installation

git clone https://github.com/ermongroup/TEOChat.git
cd TEOChat
conda create -n teochat python=3.9 -y
conda activate teochat
pip install --upgrade pip  # enable PEP 660 support
pip install -e .
pip install git+https://github.com/facebookresearch/pytorchvideo

🗝️ Training & Validating

The training & validating instructions, including how to download the TEOChatlas dataset, are in TRAIN_AND_VALIDATE.md.

👍 Acknowledgement

🔒 License

✏️ Citation

If you find our paper and code useful in your research, please consider giving a star :star: and citation :pencil:.

@article{irvin2024teochat,
  title={TEOChat: A Large Vision-Language Assistant for Temporal Earth Observation Data},
  author={Irvin, Jeremy Andrew and Liu, Emily Ruoyu and Chen, Joyce Chuyi and Dormoy, Ines and Kim, Jinyoung and Khanna, Samar and Zheng, Zhuo and Ermon, Stefano},
  journal={arXiv preprint arXiv:2410.06234},
  year={2024}
}