Awesome

🆕 Updates in Oct 2024

• HuggingFace demo and model checkpoint, see <a href="https://huggingface.co/spaces/yuhangzang/ContextDet-Demo">here</a>
• The CODE dataset for evaluation, see here
• ContextDET training scripts, see here (waiting to be cleaned up)

🌟 Contextual Object Detection

Recent Multimodal Large Language Models (MLLMs) are remarkable in vision-language tasks, such as image captioning and question answering, but lack the essential perception ability, <i>i.e</i>., object detection. In this work, we address this limitation by introducing a novel research problem of <strong>contextual object detection</strong>--understanding visible objects within different human-AI interactive contexts. Three representative scenarios are investigated, including the language cloze test, visual captioning, and question answering.

Comparison with Related Works

😎 Method

We present ContextDET, a novel <i>generate-then-detect</i> framework, specialized for contextual object detection. ContextDET is end-to-end and consists of three key architectural components:

1. a visual encoder that extracts high-level image representations and computes visual tokens,
2. a pre-trained LLM that decodes multimodal contextual tokens with a task-related multimodal prefix, and
3. a visual decoder that predicts matching scores and bounding boxes for conditional queries linked to contextual object words.

The new <strong>generate-then-detect</strong> framework enables us to detect object words within human vocabulary.

🥰 Qualitative Examples

💻 Try Demo

🤗 You can try our demo on <a href="https://huggingface.co/spaces/yuhangzang/ContextDet-Demo">HuggingFace spaces</a>. To avoid waiting in the queue and speed up your inference, consider <a href="https://huggingface.co/spaces/yuhangzang/ContextDet-Demo?duplicate=true">duplicating the space</a> and use GPU resources.

🤗 If you want to try the demo on your own computer with GPU, follow these steps

1. Install the required python packages:

pip install -r requirements.txt

2. Download the checkpoint file from the following <a href="https://drive.google.com/file/d/1ko_QPvhaHpmi7ASrkaLNSakJ2MYHMqFG/view?usp=share_link">URL</a> and save it in your local directory.
3. Now, you're ready to run the demo. Execute the following command:

python app.py

You are expected to see the following web page:

📝 Citation

We would be grateful if you consider citing our work if you find it useful:

@article{zang2023contextual,
  author = {Zang, Yuhang and Li, Wei and Han, Jun and Zhou, Kaiyang and Loy, Chen Change},
  title = {Contextual Object Detection with Multimodal Large Language Models},
  journal = {arXiv preprint arXiv:2305.18279},
  year = {2023}
}

📋 Liscense

This project is licensed under <a rel="license" href="https://github.com/yuhangzang/ContextDET/blob/master/LICENSE">S-Lab License 1.0</a>. Redistribution and use for non-commercial purposes should follow this license.

😃 Acknowledgement

We acknowledge the use of the following public code in this project: ¹DETR, ²Deformable DETR, ³DETA, ⁴OV DETR, ⁵BLIP2.

📧 Contact

If you have any questions, please feel free to contact Yuhang Zang <b>(zang0012 AT ntu.edu.sg)</b>.