Awesome

CVinW Readings

``Computer Vision in the Wild (CVinW)'' is an emerging research field. This writeup provides a quick introduction of CVinW and maintains a collection of papers on the topic. If you find some missing papers or resources, please open issues or pull requests (recommended).

Table of Contents

• What is Computer Vision in the Wild (CVinW)?
  • Goals of CVinW
  • Task Transfer Scenarios are Broad
  • Task Transfer Cost is Low
  • Benchmarks
  • News
• Papers on Task-level Transfer with Pre-trained Models
  • Image Classification in the Wild
  • Object Detection in the Wild
  • Segmentation in the Wild
  • Video Classification in the Wild
  • Grounded Image Generation in the Wild
  • Large Multimodal Models
  • Others
• Papers on Efficient Model Adaptation
  • Parameter-Efficient Methods
  • Others
• Papers on Out-of-domain Generalization
  • Surveys
  • Out-of-domain Generalization
  • Robust Models
• Acknowledgements

What is Computer Vision in the Wild?

:star: Goals of CVinW

Developing a transferable foundation model/system that can effortlessly adapt to a large range of visual tasks in the wild. It comes with two key factors: (i) The task transfer scenarios are broad, and (ii) The task transfer cost is low. The main idea is illustrated as follows, please see the detailed description in ELEVATER paper.

:one: Task Transfer Scenarios are Broad

We illustrate and compare CVinW with other settings using a 2D chart in Figure 1, where the space is constructed with two orthogonal dimensions: input image distribution and output concept set. The 2D chart is divided into four quadrants, based on how the model evaluation stage is different from model development stage. For any visual recognition problems at different granularity such as image classification, object detection and segmentation, the modeling setup cann be categorized into one of the four settings. We see an emerging trend on moving towards CVinW. Interested in the various pre-trained vision models that move towards CVinW? please check out Section :fire:``Papers on Task-level Transfer with Pre-trained Models''.

:two: Task Transfer Cost is Low

One major advantage of pre-trained models is the promise that they can transfer to downstream tasks effortlessly. The model adaptation cost is considered in two orthogonal dimensions: sample-efficiency and parameter-efficiency, as illustrated in Figure 2. The bottom-left corner and top-right corner is the most inexpensive and expensive adaptation strategy, respectively. One may interpolate and make combinations in the 2D space, to get different model adaptation methods with different cost. To efficient adapt large vision models of the gradaully increaseing size, we see an emerging need on efficient model adaptation. Interested in contributing your smart efficient adaptation algorithms and see how it differs from existing papers? please check out Section :snowflake:``Papers on Efficient Model Adaptation'' .

:cinema: Benchmarks

:loudspeaker: News

• [09/2023] 🔥 Discover the fascinating journey of "Multimodal Foundation Models: From Specialists to General-Purpose Assistants" 🌐 Dive into the evolution of large models in #ComputerVision & #VisionLanguage! This is based on our CVPR 2023 Tutorial, where you could find videos and slides of the core chapters. For its preceding paper, please check out Vision-Language Pre-training: Basics, Recent Advances, and Future Trends

• [02/2023] Organizing the 2nd Workshop @ CVPR2023 on Computer Vision in the Wild (CVinW), where two new challenges are hosted to evaluate the zero-shot, few-shot and full-shot performance of pre-trained vision models in downstream tasks:
  • ``Segmentation in the Wild (SGinW)'' Challenge evaluates on 25 image segmentation tasks.
  • ``Roboflow 100 for Object Detection in the Wild'' Challenge evaluates on 100 object detection tasks.

$\qquad$ <img src="images/cvpr-2023-logo.jpeg" width=10%/> [Workshop] $\qquad$ <img src="images/sginw.jpg" width=10%/> [SGinW Challenge] $\qquad$ <img src="images/rf100.png" width=10%/> [RF100 Challenge]

• [09/2022] Organizing ECCV Workshop Computer Vision in the Wild (CVinW), where two challenges are hosted to evaluate the zero-shot, few-shot and full-shot performance of pre-trained vision models in downstream tasks:
  • ``Image Classification in the Wild (ICinW)'' Challenge evaluates on 20 image classification tasks.
  • ``Object Detection in the Wild (ODinW)'' Challenge evaluates on 35 object detection tasks.

$\qquad$ <img src="images/eccv2022-logo.png" width=10%/> [Workshop] $\qquad$ <img src="images/icinw100.jpg" width=10%/> [ICinW Challenge] $\qquad$ <img src="images/odinw.jpg" width=10%/> [ODinW Challenge]

:fire: Papers on Task-level Transfer with Pre-trained Models

:orange_book: Image Classification in the Wild

:orange_book: Object Detection in the Wild

:orange_book: 3D Object Detection in the Wild

:orange_book: Segmentation in the Wild

:orange_book: 3D Segmentation in the Wild

CLIP2Scene: Towards Label-Efficient 3D Scene Understanding by CLIP Runnan Chen, Youquan Liu, Lingdong Kong, Xinge Zhu, Yuexin Ma, Yikang Li, Yuenan Hou, Yu Qiao, Wenping Wang CVPR 2023. [paper] [code]

Towards Label-Free Scene Understanding by Vision Foundation Models Runnan Chen, Youquan Liu, Lingdong Kong, Nenglun Chen, Xinge Zhu, Yuexin Ma, Tongliang Liu, Wenping Wang NeurIPS 2023. [paper] [code]

Segment Any Point Cloud Sequences by Distilling Vision Foundation Models Youquan Liu, Lingdong Kong, Jun Cen, Runnan Chen, Wenwei Zhang, Liang Pan, Kai Chen, Ziwei Liu NeurIPS 2023 (Spotlight). [paper] [code] [project]

:orange_book: Video Classification in the Wild

:orange_book: Other Visual Recognition in the Wild

:orange_book: Grounded Image Generation in the Wild

:new: This is a new research topic: grounded image generation based on any open-set concept, include text and visual prompt. All the text-to-image pre-trained generation models allow open-set prompting at the image-level, and thus belong to ``Grounded Image Generation in the Wild'' by default. This paper collection focuses on more fine-grained controlability in the image generation, such as specifying new concept at the the level of bounding box, masks, edge/depth maps etc.

:orange_book: Large Multimodal Models

:new: This is a new research topic: build general-purpose multimodal assistants based on large language models (LLM). One prominent example is OpenAI Multimodal GPT-4. A comphrensive list paper list is compiled at Awesome-Multimodal-Large-Language-Models. Our collection maintains the a brief list for the completenes of CVinW.

:snowflake: Papers on Efficient Model Adaptation

:blue_book: Parameter-Efficient Methods

$\colorbox{powderblue}{Prompt}$   $\colorbox{tomato}{Adapter}$

:blue_book: Other Efficient Model Adaptation Methods

:eye: Papers on Out-of-domain Generalization

:green_book: Surveys

:green_book: Out-of-domain Generalization

:green_book: Robust Models

:beers: Acknowledgements

We thank all the authors above for their great works! Related Reading List

• [Awesome Detection Transformer]
• [Awesome Prompting Papers in Computer Vision]

If you find this repository useful, please consider giving a star :star: and cite the related papers :beer::

@article{li2022elevater,
    title={ELEVATER: A Benchmark and Toolkit for Evaluating Language-Augmented Visual Models},
    author={Li, Chunyuan and Liu, Haotian and Li, Liunian Harold and Zhang, Pengchuan and Aneja, Jyoti and Yang, Jianwei and Jin, Ping and Hu, Houdong and Liu, Zicheng and Lee, Yong Jae and Gao, Jianfeng},
    journal={Neural Information Processing Systems},
    year={2022}
}

@article{li2023multimodal,
    title={Multimodal Foundation Models: From Specialists to General-Purpose Assistants},
    author={Li, Chunyuan and Gan, Zhe and Yang, Zhengyuan and Yang, Jianwei and Li, Linjie and Wang, Lijuan and Gao, Jianfeng},
    journal={arXiv:2309.10020},
    year={2023}
}

@article{gan2022vision,
  title={Vision-language pre-training: Basics, recent advances, and future trends},
  author={Gan, Zhe and Li, Linjie and Li, Chunyuan and Wang, Lijuan and Liu, Zicheng and Gao, Jianfeng},
  journal={Foundations and Trends{\textregistered} in Computer Graphics and Vision},
  volume={14},
  number={3--4},
  pages={163--352},
  year={2022},
  publisher={Now Publishers, Inc.}
}