Awesome
MarineInst20M
The official dataset repository of "MarineInst: A Foundation Model for Marine Image Analysis with Instance Visual Description". ECCV 2024.
π’ News
[July.10 2024] We release our MarineInst20M dataset and corresponding codes to reproduce our annotations!
[July.2 2024] MarineInst is accepted by ECCV 2024
with two Strong Accept
.
Dataset construction flow
Dataset construction flow:
<p align="center"> <img src="figs/dataset_flow.png" width="100%"></a> <br> Dataset construction flow of our MarineInst20M. </p>Dataset statistics:
<p align="center"> <img src="figs/statistics.jpg" width="100%"></a> <br> Statistics of each component in our MarineInst20M. </p>Key Contributions:
- MarineInst20M - First large-scale Marine dataset (million-level) with instance masks enable marine instance description (instance segmentation + instance captioning).
- Combination of wide public marine datasets/websites (around 50 sources) - We try our best to collect/gather the marine image dataset (with various formats of annotations) and images from public marine websites.
- Instruction-following training data - formulation of paired instance-caption to support various instruction-following understanding tasks.
Potential applications of MarineInst20M dataset:
- Scale up marine sea creature recognition.
- Biological monitoring and monitoring.
- Support a large range of downstream marine visual understanding tasks.
- Interdisciplinary research.
- More complicated systems (
instance-level visual language model
,contrallable image synthesis
,underwater image enhancement
,3D reconstruction
andvideo understanding
).
The directory structure of our MarineInst20M should be this:
βββMarineInst20M
βββ Flickr
βββ Human-annotated
βββ Model-generated # image urls and annotations
βββ Shutterstock
βββ Human-annotated
βββ Model-generated
βββ Gettyimages
βββ Human-annotated
βββ Model-generated
βββ Private_Data # our private data and images from YouTube or Webimages
βββ YouTube_data
βββ Webimages
βββ ...
βββ Public_Datasets # we convert the annotations of existing public datasets to masks
βββ DeepFish
βββ IOCFish5K
βββ ...
βββ Public_Websites # we provide the urls and corresponding annotations for images from public websites
βββ EOL
βββ FishDB
βββ ...
We provide corresponding README
file under each folder to provide more information. We provide the details and corresponding jsons for constructing our MarineInst20M. Please note that we provide the instance mask annotation in COCO RLE
format.
Acknowledgement
- SAM Please check this great open-source work if you are not familiar with foundation models. We thank their contributions to the whole community.
- BLIP2 Please check this great open-source work if you are not familiar with VLMs!
- MarineGPT Domain-specific VLM to generate captions based on the generated instance masks.
- SALT Our internal labeling tool is mainly modified from SALT.
- MiniGPT-4 A powerful and open-source MLLM!
Citation
If you find our work useful in your research, please consider citing:
@article{ziqiang2024marineinst,
title={MarineInst: A Foundation Model for Marine Image Analysis with Instance Visual Description},
author={Ziqiang Zheng, Yiwe Chen, Huimin Zeng, Tuan-Anh Vu, Binh-Son Hua, Sai-Kit Yeung},
journal={European Conference on Computer Vision (ECCV)},
year={2024},
publisher={Springer}
}