Home

Awesome

COCO-Stuff 10K dataset v1.1 (outdated)

Holger Caesar, Jasper Uijlings, Vittorio Ferrari

Overview

<img src="http://calvin.inf.ed.ac.uk/wp-content/uploads/data/cocostuffdataset/cocostuff-examples.png" alt="COCO-Stuff example annotations" width="100%"> Welcome to official homepage of the COCO-Stuff [1] dataset. COCO-Stuff augments the popular COCO [2] dataset with pixel-level stuff annotations. These annotations can be used for scene understanding tasks like semantic segmentation, object detection and image captioning.

Overview

Highlights

Updates

Results

The current release of COCO-Stuff-10K publishes both the training and test annotations and users report their performance individually. We invite users to report their results to us to complement this table. In the near future we will extend COCO-Stuff to all images in COCO and organize an official challenge where the test annotations will only be known to the organizers.

For the updated table please click here.

MethodSourceClass-average accuracyGlobal accuracyMean IOUFW IOU
FCN-16s [3][1]34.0%52.0%22.7%-
Deeplab VGG-16 (no CRF) [4][1]38.1%57.8%26.9%-
FCN-8s [3][6]38.5%60.4%27.2%-
DAG-RNN + CRF [6][6]42.8%63.0%31.2%-
OHE + DC + FCN+ [5][5]45.8%66.6%34.3%51.2%
Deeplab ResNet (no CRF) [4]-45.5%65.1%34.4%50.4%
W2V + DC + FCN+ [5][5]45.1%66.1%34.7%51.0%

Dataset

FilenameDescriptionSize
cocostuff-10k-v1.1.zipCOCO-Stuff dataset v. 1.1, images and annotations2.0 GB
cocostuff-10k-v1.1.jsonCOCO-Stuff dataset v. 1.1, annotations in JSON format (optional)62.3 MB
cocostuff-labels.txtA list of the 1+91+91 classes in COCO-Stuff2.3 KB
cocostuff-readme.txtThis document6.5 KB
Older files
cocostuff-10k-v1.0.zipCOCO-Stuff dataset version 1.0, including images and annotations2.6 GB

Usage

To use the COCO-Stuff dataset, please follow these steps:

  1. Download or clone this repository using git: git clone https://github.com/nightrome/cocostuff10k.git
  2. Open the dataset folder in your shell: cd cocostuff10k
  3. If you have Matlab, run the following commands:
  1. Alternatively run the following Linux commands or manually download and unpack the dataset:

MAT Format

The COCO-Stuff annotations are stored in separate .mat files per image. These files follow the same format as used by Tighe et al.. Each file contains the following fields:

JSON Format

Alternatively, we also provide stuff and thing annotations in the COCO-style JSON format. The thing annotations are copied from COCO. We encode every stuff class present in an image as a single annotation using the RLE encoding format of COCO. To get the annotations:

Label Names & Indices

To be compatible with COCO, version 1.1 of COCO-Stuff has 91 thing classes (1-91), 91 stuff classes (92-182) and 1 class "unlabeled" (0). Note that 11 of the thing classes from COCO 2015 do not have any segmentation annotations. The classes desk, door and mirror could be either stuff or things and therefore occur in both COCO and COCO-Stuff. To avoid confusion we add the suffix "-stuff" to those classes in COCO-Stuff. The full list of classes can be found here.

The older version 1.0 of COCO-Stuff had 80 thing classes (2-81), 91 stuff classes (82-172) and 1 class "unlabeled" (1).

Label Hierarchy

The hierarchy of labels is stored in CocoStuffClasses. To visualize it, run CocoStuffClasses.showClassHierarchyStuffThings() (also available for just stuff and just thing classes) in Matlab. The output should look similar to the following figure: <img src="https://github.com/nightrome/cocostuff10k/blob/master/dataset/cocostuff-labelhierarchy.png?raw=true" alt="COCO-Stuff label hierarchy" width="100%">

Semantic Segmentation Models

To encourage further research of stuff and things we provide the trained semantic segmentation model (see Sect. 4.4 in [1]).

DeepLab VGG-16

Use the following steps to download and setup the DeepLab [4] semantic segmentation model trained on COCO-Stuff. It requires deeplab-public-ver2, which is built on Caffe:

  1. Install Cuda. I recommend version 7.0. For version 8.0 you will need to apply the fix described here in step 3.
  2. Download deeplab-public-ver2: git submodule update --init models/deeplab/deeplab-public-ver2
  3. Compile and configure deeplab-public-ver2 following the author's instructions. Depending on your system setup you might have to install additional packages, but a minimum setup could look like this:
  1. Configure the COCO-Stuff dataset:
  1. Download the base VGG-16 model:
  1. Run cd models/deeplab && ./run_cocostuff_vgg16.sh to train and test the network on COCO-Stuff.

DeepLab ResNet 101

The default Deeplab model performs center crops of size 513*513 pixels of an image, if any side is larger than that. Since we want to segment the whole image at test time, we choose to resize the images to 513x513, perform the semantic segmentation and then rescale it elsewhere. Note that without the final step, the performance might differ slightly.

  1. Follow steps 1-4 of the DeepLab VGG-16 section above.
  2. Download the base ResNet model:
  1. Rescale the images and annotations:
  1. Run ./run_cocostuff_resnet101.sh to train and test the network on COCO-Stuff.

Annotation Tool

In [1] we present a simple and efficient stuff annotation tool which was used to annotate the COCO-Stuff dataset. It uses a paintbrush tool to annotate SLICO superpixels (precomputed using the code of Achanta et al.) with stuff labels. These annotations are overlaid with the existing pixel-level thing annotations from COCO. We provide a basic version of our annotation tool:

Misc

References

Licensing

COCO-Stuff is a derivative work of the COCO dataset. The authors of COCO do not in any form endorse this work. Different licenses apply:

Contact

If you have any questions regarding this dataset, please contact us at holger-at-it-caesar.com.