Home

Awesome

<p> <a href="./README_CN.md">[δΈ­ζ–‡η‰ˆζœ¬]</a> </p> We currently receive a bunch of issues, our team will check and solve them one by one, please stay tuned.

INTERN-2.5: Multimodal Multitask General Large Model

PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC PWC

The official implementation of

InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions.

[Paper] [Blog in Chinese]

Highlights

Related Projects

Foundation Models

Autonomous Driving

Application in Challenges

News

History

Introduction

"INTERN-2.5" is a powerful multimodal multitask general model jointly released by SenseTime and Shanghai AI Laboratory. It consists of large-scale vision foundation model "InternImage", pre-training method "M3I-Pretraining", generic decoder "Uni-Perceiver" series, and generic encoder for autonomous driving perception "BEVFormer" series.

<div align=left> <img src='./docs/figs/intern_pipeline_en.png' width=900> </div>

Applications

πŸŒ… Image Modality Tasks

"INTERN-2.5" achieved an impressive Top-1 accuracy of 90.1% on the ImageNet benchmark dataset using only publicly available data for image classification. Apart from two undisclosed models trained with additional datasets by Google and Microsoft, "INTERN-2.5" is the only open-source model that achieves a Top-1 accuracy of over 90.0%, and it is also the largest model in scale worldwide.

"INTERN-2.5" outperformed all other models worldwide on the COCO object detection benchmark dataset with a remarkable mAP of 65.5, making it the only model that surpasses 65 mAP in the world.

"INTERN-2.5" also demonstrated world's best performance on 16 other important visual benchmark datasets, covering a wide range of tasks such as classification, detection, and segmentation, making it the top-performing model across multiple domains.

Performance

<table border="1" width="90%"> <tr align="center"> <th colspan="1"> Image Classification</th><th colspan="2"> Scene Classification </th><th colspan="1">Long-Tail Classification</th> </tr> <tr align="center"> <th>ImageNet</th><th>Places365</th><th>Places 205</th><th>iNaturalist 2018</th> </tr> <tr align="center"> <th>90.1</th><th>61.2</th><th>71.7</th><th>92.3</th> </tr> </table> <table border="1" width="90%"> <tr align="center"> <th colspan="4"> Conventional Object Detection</th><th colspan="3">Long-Tail Object Detection </th><th colspan="1">Autonomous Driving Object Detection</th><th colspan="1">Dense Object Detection</th> </tr> <tr align="center"> <th>COCO</th><th>VOC 2007</th><th>VOC 2012</th><th>OpenImage</th><th>LVIS minival</th><th>LVIS val</th><th>BDD100K</th><th>nuScenes</th><th>CrowdHuman</th> </tr> <tr align="center"> <th>65.5</th><th>94.0</th><th>97.2</th><th>74.1</th><th>65.8</th><th>63.2</th><th>38.8</th><th>64.8</th><th>97.2</th> </tr> </table> <table border="1" width="90%"> <tr align="center"> <th colspan="3">Semantic Segmentation</th><th colspan="1">Street Segmentation</th><th colspan="1">RGBD Segmentation</th> </tr> <tr align="center"> <th>ADE20K</th><th>COCO Stuff-10K</th><th>Pascal Context</th><th>CityScapes</th><th>NYU Depth V2</th> </tr> <tr align="center"> <th>62.9</th><th>59.6</th><th>70.3</th><th>86.1</th><th>69.7</th> </tr> </table> <br>

🌁 πŸ“– Image and Text Cross-Modal Tasks

Image-Text Retrieval: "INTERN-2.5" can quickly locate and retrieve the most semantically relevant images based on textual content requirements. This capability can be applied to both videos and image collections and can be further combined with object detection boxes to enable a variety of applications, helping users quickly and easily find the required image resources. For example, it can return the relevant images specified by the text in the album.

Image-To-Text: "INTERN-2.5" has a strong understanding capability in various aspects of visual-to-text tasks such as image captioning, visual question answering, visual reasoning, and optical character recognition. For example, in the context of autonomous driving, it can enhance the scene perception and understanding capabilities, assist the vehicle in judging traffic signal status, road signs, and other information, and provide effective perception information support for vehicle decision-making and planning.

Performance

<table border="1" width="90%"> <tr align="center"> <th colspan="1">Image Captioning</th><th colspan="2">Fine-tuning Image-Text Retrieval</th><th colspan="1">Zero-shot Image-Text Retrieval</th> </tr> <tr align="center"> <th>COCO Caption</th><th>COCO Caption</th><th>Flickr30k</th><th>Flickr30k</th> </tr> <tr align="center"> <th>148.2</th><th>76.4</th><th>94.8</th><th>89.1</th> </tr> </table> <br>

Released Models

<details> <summary> Open-source Visual Pretrained Models </summary> <br> <div>
namepretrainpre-training resolution#paramdownload
InternImage-LImageNet-22K384x384223Mckpt
InternImage-XLImageNet-22K384x384335Mckpt
InternImage-HJoint 427M384x3841.08Bckpt
InternImage-G-384x3843Bckpt
</div> </details> <details> <summary> ImageNet-1K Image Classification </summary> <br> <div>
namepretrainresolutionacc@1#paramFLOPsdownload
InternImage-TImageNet-1K224x22483.530M5Gckpt | cfg
InternImage-SImageNet-1K224x22484.250M8Gckpt | cfg
InternImage-BImageNet-1K224x22484.997M16Gckpt | cfg
InternImage-LImageNet-22K384x38487.7223M108Gckpt | cfg
InternImage-XLImageNet-22K384x38488.0335M163Gckpt | cfg
InternImage-HJoint 427M640x64089.61.08B1478Gckpt | cfg
InternImage-G-512x51290.13B2700Gckpt | cfg
</div> </details> <details> <summary> COCO Object Detection and Instance Segmentation </summary> <br> <div>
backbonemethodschdbox mAPmask mAP#paramFLOPsdownload
InternImage-TMask R-CNN1x47.242.549M270Gckpt | cfg
InternImage-TMask R-CNN3x49.143.749M270Gckpt | cfg
InternImage-SMask R-CNN1x47.843.369M340Gckpt | cfg
InternImage-SMask R-CNN3x49.744.569M340Gckpt | cfg
InternImage-BMask R-CNN1x48.844.0115M501Gckpt | cfg
InternImage-BMask R-CNN3x50.344.8115M501Gckpt | cfg
InternImage-LCascade1x54.947.7277M1399Gckpt | cfg
InternImage-LCascade3x56.148.5277M1399Gckpt | cfg
InternImage-XLCascade1x55.348.1387M1782Gckpt | cfg
InternImage-XLCascade3x56.248.8387M1782Gckpt | cfg
backbonemethodbox mAP (val/test)#paramFLOPsdownload
InternImage-HDINO (TTA)65.0 / 65.42.18BTODOTODO
InternImage-GDINO (TTA)65.3 / 65.53BTODOTODO
</div> </details> <details> <summary> ADE20K Semantic Segmentation </summary> <br> <div>
backbonemethodresolutionmIoU (ss/ms)#paramFLOPsdownload
InternImage-TUperNet512x51247.9 / 48.159M944Gckpt | cfg
InternImage-SUperNet512x51250.1 / 50.980M1017Gckpt | cfg
InternImage-BUperNet512x51250.8 / 51.3128M1185Gckpt | cfg
InternImage-LUperNet640x64053.9 / 54.1256M2526Gckpt | cfg
InternImage-XLUperNet640x64055.0 / 55.3368M3142Gckpt | cfg
InternImage-HUperNet896x89659.9 / 60.31.12B3566Gckpt | cfg
InternImage-HMask2Former896x89662.5 / 62.91.31B4635Gckpt | cfg
</div> </details> <details> <summary> Main Results of FPS </summary> <br> <div>

Export classification model from pytorch to tensorrt

Export detection model from pytorch to tensorrt

Export segmentation model from pytorch to tensorrt

nameresolution#paramFLOPsbatch 1 FPS (TensorRT)
InternImage-T224x22430M5G156
InternImage-S224x22450M8G129
InternImage-B224x22497M16G116
InternImage-L384x384223M108G56
InternImage-XL384x384335M163G47

Before using mmdeploy to convert our PyTorch models to TensorRT, please make sure you have the DCNv3 custom operator builded correctly. You can build it with the following command:

export MMDEPLOY_DIR=/the/root/path/of/MMDeploy

# prepare our custom ops, you can find it at InternImage/tensorrt/modulated_deform_conv_v3
cp -r modulated_deform_conv_v3 ${MMDEPLOY_DIR}/csrc/mmdeploy/backend_ops/tensorrt

# build custom ops
cd ${MMDEPLOY_DIR}
mkdir -p build && cd build
cmake -DCMAKE_CXX_COMPILER=g++-7 -DMMDEPLOY_TARGET_BACKENDS=trt -DTENSORRT_DIR=${TENSORRT_DIR} -DCUDNN_DIR=${CUDNN_DIR} ..
make -j$(nproc) && make install

# install the mmdeploy after building custom ops
cd ${MMDEPLOY_DIR}
pip install -e .

For more details on building custom ops, please refering to this document.

</div> </details>

Citations

If this work is helpful for your research, please consider citing the following BibTeX entry.

@article{wang2022internimage,
  title={InternImage: Exploring Large-Scale Vision Foundation Models with Deformable Convolutions},
  author={Wang, Wenhai and Dai, Jifeng and Chen, Zhe and Huang, Zhenhang and Li, Zhiqi and Zhu, Xizhou and Hu, Xiaowei and Lu, Tong and Lu, Lewei and Li, Hongsheng and others},
  journal={arXiv preprint arXiv:2211.05778},
  year={2022}
}

@inproceedings{zhu2022uni,
  title={Uni-perceiver: Pre-training unified architecture for generic perception for zero-shot and few-shot tasks},
  author={Zhu, Xizhou and Zhu, Jinguo and Li, Hao and Wu, Xiaoshi and Li, Hongsheng and Wang, Xiaohua and Dai, Jifeng},
  booktitle={CVPR},
  pages={16804--16815},
  year={2022}
}

@article{zhu2022uni,
  title={Uni-perceiver-moe: Learning sparse generalist models with conditional moes},
  author={Zhu, Jinguo and Zhu, Xizhou and Wang, Wenhai and Wang, Xiaohua and Li, Hongsheng and Wang, Xiaogang and Dai, Jifeng},
  journal={arXiv preprint arXiv:2206.04674},
  year={2022}
}

@article{li2022uni,
  title={Uni-Perceiver v2: A Generalist Model for Large-Scale Vision and Vision-Language Tasks},
  author={Li, Hao and Zhu, Jinguo and Jiang, Xiaohu and Zhu, Xizhou and Li, Hongsheng and Yuan, Chun and Wang, Xiaohua and Qiao, Yu and Wang, Xiaogang and Wang, Wenhai and others},
  journal={arXiv preprint arXiv:2211.09808},
  year={2022}
}

@article{yang2022bevformer,
  title={BEVFormer v2: Adapting Modern Image Backbones to Bird's-Eye-View Recognition via Perspective Supervision},
  author={Yang, Chenyu and Chen, Yuntao and Tian, Hao and Tao, Chenxin and Zhu, Xizhou and Zhang, Zhaoxiang and Huang, Gao and Li, Hongyang and Qiao, Yu and Lu, Lewei and others},
  journal={arXiv preprint arXiv:2211.10439},
  year={2022}
}

@article{su2022towards,
  title={Towards All-in-one Pre-training via Maximizing Multi-modal Mutual Information},
  author={Su, Weijie and Zhu, Xizhou and Tao, Chenxin and Lu, Lewei and Li, Bin and Huang, Gao and Qiao, Yu and Wang, Xiaogang and Zhou, Jie and Dai, Jifeng},
  journal={arXiv preprint arXiv:2211.09807},
  year={2022}
}

@inproceedings{li2022bevformer,
  title={Bevformer: Learning bird’s-eye-view representation from multi-camera images via spatiotemporal transformers},
  author={Li, Zhiqi and Wang, Wenhai and Li, Hongyang and Xie, Enze and Sima, Chonghao and Lu, Tong and Qiao, Yu and Dai, Jifeng},
  booktitle={ECCV},
  pages={1--18},
  year={2022},
}