Home

Awesome

:sauropod: Grounding DINO


Grounding DINO Methods | GitHub arXiv YouTube

Grounding DINO Demos | Open In Colab YouTube HuggingFace space YouTube YouTube

Extensions | Grounding DINO with Segment Anything; Grounding DINO with Stable Diffusion; Grounding DINO with GLIGEN

PWC
PWC
PWC
PWC

Official PyTorch implementation of Grounding DINO, a stronger open-set object detector. Code is available now!

:bulb: Highlight

:fire: News

<details open> <summary><font size="4"> Description </font></summary> <a href="https://arxiv.org/abs/2303.05499">Paper</a> introduction. <img src=".asset/hero_figure.png" alt="ODinW" width="100%"> Marrying <a href="https://github.com/IDEA-Research/GroundingDINO">Grounding DINO</a> and <a href="https://github.com/gligen/GLIGEN">GLIGEN</a> <img src="https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/GD_GLIGEN.png" alt="gd_gligen" width="100%"> </details>

:star: Explanations/Tips for Grounding DINO Inputs and Outputs

:label: TODO

:hammer_and_wrench: Install

If you have a CUDA environment, please make sure the environment variable CUDA_HOME is set. It will be compiled under CPU-only mode if no CUDA available.

pip install -e .

:arrow_forward: Demo

CUDA_VISIBLE_DEVICES=6 python demo/inference_on_a_image.py \
  -c /path/to/config \
  -p /path/to/checkpoint \
  -i .asset/cats.png \
  -o "outputs/0" \
  -t "cat ear." \
  [--cpu-only] # open it for cpu mode

See the demo/inference_on_a_image.py for more details.

Web UI

We also provide a demo code to integrate Grounding DINO with Gradio Web UI. See the file demo/gradio_app.py for more details.

Notebooks

:luggage: Checkpoints

<!-- insert a table --> <table> <thead> <tr style="text-align: right;"> <th></th> <th>name</th> <th>backbone</th> <th>Data</th> <th>box AP on COCO</th> <th>Checkpoint</th> <th>Config</th> </tr> </thead> <tbody> <tr> <th>1</th> <td>GroundingDINO-T</td> <td>Swin-T</td> <td>O365,GoldG,Cap4M</td> <td>48.4 (zero-shot) / 57.2 (fine-tune)</td> <td><a href="https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha/groundingdino_swint_ogc.pth">Github link</a> | <a href="https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swint_ogc.pth">HF link</a></td> <td><a href="https://github.com/IDEA-Research/GroundingDINO/blob/main/groundingdino/config/GroundingDINO_SwinT_OGC.py">link</a></td> </tr> <tr> <th>2</th> <td>GroundingDINO-B</td> <td>Swin-B</td> <td>COCO,O365,GoldG,Cap4M,OpenImage,ODinW-35,RefCOCO</td> <td>56.7 </td> <td><a href="https://github.com/IDEA-Research/GroundingDINO/releases/download/v0.1.0-alpha2/groundingdino_swinb_cogcoor.pth">Github link</a> | <a href="https://huggingface.co/ShilongLiu/GroundingDINO/resolve/main/groundingdino_swinb_cogcoor.pth">HF link</a> <td><a href="https://github.com/IDEA-Research/GroundingDINO/blob/main/groundingdino/config/GroundingDINO_SwinB.cfg.py">link</a></td> </tr> </tbody> </table>

:medal_military: Results

<details open> <summary><font size="4"> COCO Object Detection Results </font></summary> <img src=".asset/COCO.png" alt="COCO" width="100%"> </details> <details open> <summary><font size="4"> ODinW Object Detection Results </font></summary> <img src=".asset/ODinW.png" alt="ODinW" width="100%"> </details> <details open> <summary><font size="4"> Marrying Grounding DINO with <a href="https://github.com/Stability-AI/StableDiffusion">Stable Diffusion</a> for Image Editing </font></summary> See our example <a href="https://github.com/IDEA-Research/GroundingDINO/blob/main/demo/image_editing_with_groundingdino_stablediffusion.ipynb">notebook</a> for more details. <img src=".asset/GD_SD.png" alt="GD_SD" width="100%"> </details> <details open> <summary><font size="4"> Marrying Grounding DINO with <a href="https://github.com/gligen/GLIGEN">GLIGEN</a> for more Detailed Image Editing. </font></summary> See our example <a href="https://github.com/IDEA-Research/GroundingDINO/blob/main/demo/image_editing_with_groundingdino_gligen.ipynb">notebook</a> for more details. <img src=".asset/GD_GLIGEN.png" alt="GD_GLIGEN" width="100%"> </details>

:sauropod: Model: Grounding DINO

Includes: a text backbone, an image backbone, a feature enhancer, a language-guided query selection, and a cross-modality decoder.

arch

:hearts: Acknowledgement

Our model is related to DINO and GLIP. Thanks for their great work!

We also thank great previous work including DETR, Deformable DETR, SMCA, Conditional DETR, Anchor DETR, Dynamic DETR, DAB-DETR, DN-DETR, etc. More related work are available at Awesome Detection Transformer. A new toolbox detrex is available as well.

Thanks Stable Diffusion and GLIGEN for their awesome models.

:black_nib: Citation

If you find our work helpful for your research, please consider citing the following BibTeX entry.

@inproceedings{ShilongLiu2023GroundingDM,
  title={Grounding DINO: Marrying DINO with Grounded Pre-Training for Open-Set Object Detection},
  author={Shilong Liu and Zhaoyang Zeng and Tianhe Ren and Feng Li and Hao Zhang and Jie Yang and Chunyuan Li and Jianwei Yang and Hang Su and Jun Zhu and Lei Zhang},
  year={2023}
}