Home

Awesome

Image Conditioned Noise Model (ICNM)

Created by Ishan Misra

Based on the CVPR 2016 Paper - "Seeing through the Human Reporting Bias : Visual Classifiers from Noisy Human-Centric Labels"

You can download the paper here.

Why use it?

This codebase can help you replicate the results from the paper and train new models. In cases where your image labels are noisy, our method provides orthogonal gains over existing techniques. It may help boost performance of your baseline classification system. This paper introduces a way to learn visually correct classifiers from noisy labels. It is an Image Conditioned Noise Model which can estimate label noise without clean labels.

Introduction

Our code base is a mix of Python and C++ and uses the Caffe framework.

It is heavily derived from the visual concepts codebase by Saurabh Gupta, and the Fast-RCNN codebase by Ross Girshick. It also uses the MS COCO PythonAPI from Piotr Dollar.

Citing

If you find our code useful in your research, please consider citing:

   @inproceedings{MisraNoisy16,
    Author = {Ishan Misra and C. Lawrence Zitnick and Margaret Mitchell and Ross Girshick},
    Booktitle = {CVPR},
    Title = {{Seeing through the Human Reporting Bias:  Visual Classifiers from Noisy Human-Centric Labels}},
    Year = {2016},
   }

Contents

  1. Requirements: software
  2. Installation
  3. Download ground truth
  4. Usage
  5. Training and Experiment scripts
  6. Extra downloads

Requirements: software

  1. Requirements for Caffe and pycaffe (see: Caffe installation instructions)

Note: Caffe must be built with support for Python layers and OpenCV.

# In your Makefile.config, make sure to have this line uncommented
WITH_PYTHON_LAYER := 1
USE_OPENCV := 1
  1. Python packages you might not have: python-opencv, nltk. You can skip nltk if you download pre-computed ground-truth files (links below).

Installation

  1. Clone the ICNM repository
# Make sure to clone with --recursive
git clone --recursive https://github.com/imisra/icnm.git
  1. We'll call the directory that you cloned ICNM into ICNM_ROOT The following subdirectories should exist as soon as you clone

  1. Build Caffe and pycaffe

    cd $ICNM_ROOT/caffe-icnm
# Now follow the Caffe installation instructions here:
# http://caffe.berkeleyvision.org/installation.html

# If you're experienced with Caffe and have all of the requirements installed
# and your Makefile.config in place, then simply do:
make -j8 pycaffe #makes caffe and pycaffe with 8 processes in parallel

  2. Download pre-computed ICNM classifiers here and unzip. This will populate the folder $ICNM_ROOT/experiments/latentNoise/cache with caffe model files.

  3. Download COCO Dataset and annotations

    wget http://msvocds.blob.core.windows.net/coco2014/train2014.zip
wget http://msvocds.blob.core.windows.net/coco2014/val2014.zip
wget http://msvocds.blob.core.windows.net/annotations-1-0-3/captions_train-val2014.zip

  4. Extract all of these zips into one directory named $ICNM_ROOT/data/coco. You can optionally extract them in another location, say $COCO_ROOT and create a symlink to that location.

    unzip train2014.zip
unzip val2014.zip
unzip captions_train-val2014.zip

  5. It should have this basic structure

    $ICNM_ROOT/data/coco/images               # images
$ICNM_ROOT/data/coco/images/train2014     # images
$ICNM_ROOT/data/coco/images/val2014       # images
$ICNM_ROOT/data/coco/annotations          # json files with annotations

  6. [Optional] Download baseline models for COCO here. Unzipping this in $ICNM_ROOT will populate the folder $ICNM_ROOT/experiments/latentNoise/cache with caffe model files.

Download ground truth

Use these steps to train and test our model on the COCO dataset.

We bundle up training labels, evaluation labels, and other files used for training/testing our network. This step makes it so that the nltk package is not used. It also includes the json files for the 20k test split used in our paper. Following the MILVC paper, we call this split as valid2. It is half the size of the official COCO 2014 val set. Download it here

This should give you the following files

Usage

Once you have models and data downloaded, you can run the main_test.py file. This should reproduce the results from the paper by first classifying 20k images from the valid2 set and then evaluating the result. The file has documentation to understand the various types of evaluation and baseline methods. Since, this process is time consuming, we also provide the pre-computed detections and evaluations in the Extra downloads section for the baseline and our method.

Training and Experiment scripts

Scripts to reproduce the experiments in the paper (up to stochastic variation) are provided in $ICNM_ROOT/experiments/scripts. To train the methods, ensure that the label ground truth files and image paths in the prototxt are correct. Example, in the $ICNM_ROOT/experiments/baselines/classification_finetune.prototxt file, the following paths should exist.

mil_data_param {
  label_file: "./data/labels_captions_coco_vocabS1k_train.h5"
  source: "./data/ids_captions_coco_vocabS1k_train.txt"
  root_dir: "./data/coco/images/train2014"
}

Log files for experiments can be downloaded from Experiment logs. After unzipping they are located in experiments/baseline/logs and experiments/latentNoise/logs.

Note: Our models (in the experiments/latentNoise subdirectory), require you to train the baseline (experiments/baseline) models first, for two epochs, and are initialized using these models. For the COCO dataset, one epoch is roughly 80k iterations with a batch size of 1. Thus, our models are initialized by baseline models trained for 160k iterations.

Additional details on the data layer: The data layer used is from the visual concepts codebase. Briefly, it takes in the following parameters

Extra downloads