Home

Awesome

Banner

NeWT: Natural World Tasks

This repository contains resources for working with the NeWT dataset.

Benchmarking Representation Learning for Natural World Image Collections

Source code for reproducing the experiments in the CVPR 2021 paper can be found in the benchmark directory.

iNaturalist 2021 Dataset

The iNat2021 dataset is available here.

NeWT 2021 Dataset

The NeWT 2021 dataset is a collection of 164 binary classification tasks. The images for all tasks have been combined into a single directory, and all images are stored as 3 channel jpegs. A single csv file contains the information for all tasks. See below for descriptions and download links.

Annotation Format

Image metadata and task data are stored in a csv file. Each row of the csv file corresponds to an image, and each image belongs to exactly 1 task. The csv has the following columns:

Example Data

Example rows from newt2021_labels.csv with task_subcluster field:

idtask_clustertask_subclustertasklabeltext_labelsplitheightwidth
d80eb625-4982-4d34-ad9c-d957f565111eappearanceageml_age_coopers_hawk0not_adulttrain425640
6c3eddb5-345f-444c-9957-1bee9c5aada2appearanceageml_age_coopers_hawk0not_adulttrain430640

The image files for these respective rows are accessible at:

<br>

Example rows from newt2021_labels.csv without task_subcluster field:

idtask_clustertask_subclustertasklabeltext_labelsplitheightwidth
9a12feb3-bfbb-48cb-8227-21f5a8a4530ccontextnanml_bio_raptor_utility_pole0negtrain640540
2cb20e06-9072-42fe-bd09-8557dfb591dccontextnanml_bio_raptor_utility_pole0negtrain426640

The image files for these respective rows are accessible at:

Example Code

The following example code demonstates how to iterate over the 164 binary classification tasks. There are placeholder commands for extracting features from the images and training binary classifiers.

import os

import imageio
import numpy as np
import pandas as pd

newt_labels_fp = "newt2021_labels.csv"
newt_image_dir = "newt2021_images/"

newt_df = pd.read_csv(newt_labels_fp)
newt_df['filepath'] = newt_df['id'].apply(
    lambda image_id: os.path.join(newt_image_dir, image_id + ".jpg")
)

task_results = []
for task_name, task_df in newt_df.groupby('task'):
    
    X_train = []
    X_test = []
    y_train = []
    y_test = []
    for _, image_row in task_df.iterrows():
        
        image = imageio.imread(image_row['filepath'])
        image_label = image_row['label']

        # Place holder for feature extraction
        image_feature = np.random.default_rng().uniform(-1,1,1000) 
        
        if image_row['split'] == 'train':
            X_train.append(image_feature)
            y_train.append(image_label)
        else:
            X_test.append(image_feature)
            y_test.append(image_label)
    
    X_train = np.array(X_train)
    X_test = np.array(X_test)
    y_train = np.array(y_train)
    y_test = np.array(y_test)
    
    # Placeholder for training and evaluating a classifier
    task_acc = np.random.default_rng().uniform(0,1,1)[0]
    
    task_results.append({
        'task' : task_name,
        'task_cluster' : task_df.iloc[0]['task_cluster'],
        'task_subcluster' : task_df.iloc[0]['task_subcluster'],
        'acc' : task_acc
    })

task_results_df = pd.DataFrame(task_results)
print(f"Mean Accuracy: {task_results_df['acc'].mean():.3f}")

for task_cluster, mean_acc in task_results_df.groupby('task_cluster')['acc'].mean().iteritems():
    print(f"{task_cluster:10} {mean_acc:0.3f}")

The following code block demonstrates how to iterate over the tasks within each task cluster and subcluster. Partial output is provided after the code block.

import os

import pandas as pd

####
# Load in the NeWT dataset
newt_labels_fp = "newt2021_labels.csv"
newt_image_dir = "newt2021_images/"

newt_df = pd.read_csv(newt_labels_fp)
newt_df['filepath'] = newt_df['id'].apply(
    lambda image_id: os.path.join(newt_image_dir, image_id + ".jpg")
)

####
# Print some basic info for the task clusters
newt_task_clusters = newt_df['task_cluster'].unique()
print("NeWT Task Clusters")
for i, task_cluster_name in enumerate(newt_task_clusters):
    task_cluster_df = newt_df[newt_df['task_cluster'] == task_cluster_name]
    task_subclusters = task_cluster_df['task_subcluster'].dropna().unique()
    num_subclusters = task_subclusters.shape[0]
    num_tasks = task_cluster_df['task'].unique().shape[0]
    print(f"{i+1:1d}. {task_cluster_name:10s} ({num_subclusters:2d} subclusters) : {num_tasks:3d} tasks")
    if num_subclusters > 0:
        for task_subcluster_name in task_subclusters:
            task_subcluster_df = task_cluster_df[task_cluster_df['task_subcluster'] == task_subcluster_name]
            num_tasks = task_subcluster_df['task'].unique().shape[0]
            print(f"\t{task_subcluster_name:10s}: {num_tasks:3d} tasks")
    print()
print()
        
####
# Choose a specific task cluster and print basic info about the tasks
task_cluster_name = 'appearance'
assert task_cluster_name in newt_task_clusters, f"{task_cluster_name} is not a valid NeWT task cluster"
task_cluster_df = newt_df[newt_df['task_cluster'] == task_cluster_name]

num_subclusters = task_cluster_df['task_subcluster'].dropna().unique().shape[0]
num_tasks = task_cluster_df['task'].unique().shape[0]

if num_subclusters > 0:
    print(f"The task `{task_cluster_name}` has {num_tasks:3d} total tasks, grouped into {num_subclusters} subclusters")
else:
    print(f"The task `{task_cluster_name}` has {num_tasks:3d} total tasks")
    

print(f"Tasks included in the `{task_cluster_name}` cluster:")
for i, (task_name, task_df) in enumerate(task_cluster_df.groupby('task')):
    
    print(f"{i+1:3d}. {task_name:55s}")
    
    # Get positive and negative examples for this task
    task_positive_df = task_df[task_df['label'] == 1]
    task_negative_df = task_df[task_df['label'] == 0]
    
    # Get the text label for the positive and negative classes
    # NOTE: these are not unique across the tasks. 
    positive_text_label = task_positive_df.iloc[0]['text_label']
    negative_text_label = task_negative_df.iloc[0]['text_label']
    print(f"\tpositive class text label: {positive_text_label}")
    print(f"\tnegative class text label: {negative_text_label}")
    print(f"\tpositive images: {task_positive_df.shape[0]:3d}, negative images: {task_negative_df.shape[0]:3d}")
    print()
    
    # Get `train` and `test` examples for this task
    for split in ['train', 'test']:
        task_split_df = task_df[task_df['split'] == split]
        
        task_split_pos_df = task_split_df[task_split_df['label'] == 1]
        task_split_neg_df = task_split_df[task_split_df['label'] == 0]
        
        print(f"\tpositive {split:5s} images: {task_split_pos_df.shape[0]:3d}")
        print(f"\tnegative {split:5s} images: {task_split_neg_df.shape[0]:3d}")
        print()

Example output from the above code block:

NeWT Task Clusters
1. context    ( 0 subclusters) :   8 tasks

2. appearance ( 4 subclusters) : 132 tasks
   	attribute :   7 tasks
   	age       :  14 tasks
   	species   : 102 tasks
   	health    :   9 tasks

3. gestalt    ( 0 subclusters) :   6 tasks

4. behavior   ( 0 subclusters) :  16 tasks

5. counting   ( 0 subclusters) :   2 tasks


The task `appearance` has 132 total tasks, grouped into 4 subclusters
Tasks included in the `appearance` cluster:
  1. fgvcx_icassava_healthy_vs_sick                         
   	positive class text label: sick
   	negative class text label: healthy
   	positive images: 200, negative images: 200
   
   	positive train images: 100
   	negative train images: 100
   
   	positive test  images: 100
   	negative test  images: 100

  2. fgvcx_plant_pathology_healthy_vs_sick                  
   	positive class text label: sick
   	negative class text label: healthy
   	positive images: 200, negative images: 200
   
   	positive train images: 100
   	negative train images: 100
   
   	positive test  images: 100
   	negative test  images: 100

  3. inat_non_species_black_eastern_gray_squirrel           
   	positive class text label: black_squirrel
   	negative class text label: regular_squirrel
   	positive images:  77, negative images: 100
   
   	positive train images:  50
   	negative train images:  50
   
   	positive test  images:  27
   	negative test  images:  50

  4. inat_non_species_dead_common_garter_snake              
   	positive class text label: common_garter_snake
   	negative class text label: gopher_snake
   	positive images: 100, negative images: 100
   
   	positive train images:  50
   	negative train images:  50
   
   	positive test  images:  50
   	negative test  images:  50
...

Data Download

The dataset files are available through the AWS Open Data Program:

Reference

If you find our work useful in your research please consider citing our paper:

@inproceedings{van2021benchmarking,
  title={Benchmarking Representation Learning for Natural World Image Collections},
  author={Van Horn, Grant and Cole, Elijah and Beery, Sara and Wilber, Kimberly and Belongie, Serge and Mac Aodha, Oisin},
  booktitle={Computer Vision and Pattern Recognition},
  year={2021}
}