Awesome

OnlyPlanes: Incrementally Tuning Synthetic Training Datasets for Satellite Object Detection

Project Links: [Paper] [Project Page] [Video] [Blog]

Abstract: This paper addresses the challenge of solely using synthetic data to train and improve computer vision models for detecting airplanes in satellite imagery by iteratively tuning the training dataset. The domain gap for computer vision algorithms remains a continued struggle to generalize when learning only from synthetic images and produce results comparable to models learning from real images. We present OnlyPlanes, a synthetic satellite image training dataset of airplanes in a top-view that contains 12,500 images, 132,967 aircraft instances, with 80 fine grain attributes. We detail the synthetic rendering process to procedurally-generate and render training images and introduce the synthetic learning feedback loop that iteratively finetunes the training dataset to improve model performance. Experiments show the prior performance degradations and false positives when testing the model on real-world images were corrected when incrementally incorporating new features to the training dataset such as diverse plane formations and altitude variations. We demonstrate the promising future for continually improving models by tuning the synthetic training data developed in an iterative fashion to effectively close the synthetic to real domain gap. The OnlyPlanes dataset, source code and trained models are available at https://github.com/naivelogic/OnlyPlanes.

Example Inference Results from OnlyPlanes Models tested on Real-World Datasets

Dataset Overview

OnlyPlanes is a small dataset (less than 50k images and less than 20GB in weight) for airplane object and instant segmentation computer vision task from satellite imagery.

Models Overview

The training methodology implemented involved a Faster-RCNN and a Mask-RCNN model solely trained on the OnlyPlanes synthetic dataset. The Faster-RCNN model is used for object detection and the Mask-RCNN model is used for instance segmentation.

Benchmarking

To evaluate the performance of OnlyPlanes we tested the performance on detecting airplanes in other notable datasets. Benchmark datasets include: RarePlanes Real Test, DIOR Test, iSAID Val, and NWPU VHR-10.

To test the performance of the model the below benchmark datasets were used.

• iSAID | paper | dataset | binary plane json

  • Note: We used the pre-processed version of the dataset from CATNet approach where images were split into 512 x 512 patches and extreme aspect ratios from the official toolkit were corrected.
  • Test dataset statistics: 11,752 images | 6,613 airplane instances from the validation dataset.

• RarePlanes | paper | dataset | binary plane json

  • Note: for evaluation only RarePlanes Real Test dataset was used. Instructions to download the RarePlanes dataset RarePlanes Public User Guide. Additionally, since the official RarePlanes repository is no longer available (summer 2022), refer to the unofficial mirror repo.
  • Test dataset statistics: 2,710 images and 6,812 airplane instances from the real test dataset.

• NWPU VHR10 | paper | dataset | binary plane json

  • Test dataset statistics: 650 images and 757 airplane instances from the positive image set.

• DIOR | paper | dataset | binary plane json

  • Note: we first converted the DIOR VOC to COCO annotations using only the we used the test.txt Horizontal Bounding Boxes annotations.
  • Test dataset statistics: 2,932 images and 8,042 airplane instances from the test dataset.

Here is the file structure used for these benchmark datasets

Benchmark Datasets
├── data
│   ├── dior
│   │   ├── Annotations
│   │   ├── ImageSets
│   │   ├── JPEGImages-test
│   │   └── JPEGImages-trainval
│   ├── rareplanes
│   │   └── real
|   │       ├── metadata_annotations
|   |       │   ├── instances_test_aircraft.json
|   |       │   └── instances_test_role.json
|   │       └── test
|   |           └── PS-RGB_tiled
│   ├── isaid
│   │   ├── annotations
│   │   |   ├── instances_val.json  # original - not used
|   │   │   └── val_binary_plane_coco_annotations_ALL.json # used for benchmarking
│   │   └── val
    └── nwpu_vhr10
        ├── ground truth
        └── positive image set

Getting Started

This step we install all the necessary dependencies to train an object detector on a local machine. For this workshop we will be utilizing detectron2 that runs on PyTorch as the framework for training DNNs. Quick start

• Step 1: clone workshop repo (git required): git clone https://github.com/naivelogic/OnlyPlanes.git
• Step 2: navigate to repo folder: cd OnlyPlanes
• Step 3: install dependencies with Anaconda: sh scripts/00_install_env.sh
• Step 4: verify installation: conda activate OP

Requirements

• Python > 3.5
• Pytorch > 1.9
• torchvision that matches the PyTorch installation. You can install them together at pytorch.org to make sure of this.
• Cuda > 11.0
• Detectron2
• Make sure the CUDA, Pytorch and Detectron2 have the correct CUDA version if using GPU

Usage

This project contains various scripts to reproduce the results in the paper.

Dataset Preparations

Download and unzip the training and evaluation datasets either locally or in an Azure blob container. For each of the zip datasets, contains the coco annotations .json file and images in the data folder. Start by downloading the OnlyPlanes dataset using the following commands:

sudo chmod u+x scripts/01a_download_OnlyPlanes_dataset.sh
sh scripts/01a_download_OnlyPlanes_dataset.sh

.
└── OnlyPlanes
    ├── LICENSE.txt
    ├── coco_annotations
    │   ├── all_labels_coco_annotations.json    # all plane labels including fine-grain attributes
    │   ├── binary_plane_coco_annotations.json  # use this for **training on binary planes** 1 class
    │   ├── role_coco_annotations.json          
    │   ├── civil_role_coco_annotations.json    
    │   ├── OnlyPlanes_Role_Counts.csv
    │   └── OnlyPlanes_metadata.csv
    └── images
        ├── 1.png
        └── ...

Train model on Synthetic OnlyPlanes dataset

OnlyPlanes paper provides an object detection and a segmentation model. You can train the model by yourself or directly use the snapshot provided by us. The following script lets you train them using the configurations defined in the OnlyPlanes paper.

To train the dataset run the following, also update the training config .yaml files as necessary (e.g., number of classes)

conda activate OP

TRAIN_IMG_DIR=/path/to/image/directory         #(e.g., datasets/OnlyPlanes/images)
TRAIN_COCO_JSON=/path/to/coco_annotations.json #(e.g., datasets/OnlyPlanes/coco_ds/OnlyPlanes_binary_plane_annotations.json)

VAL_IMG_DIR=/path/to/image/directory         
VAL_COCO_JSON=/path/to/coco_annotations.json 

OUTPUT_FOLDER=/path/to/output_training_results
TRAINCONFIG=/path/to/training_configs.yaml    #(e.g., configs/frcnn_r50_fpn_demo.yaml)
TRAIN_METHOD=bbox #bbox segm (rotated and obb not supported yet)
python train.py --output-folder $OUTPUT_FOLDER --config-file $TRAINCONFIG \
                --train_img_dir $TRAIN_IMG_DIR --train_coco_json $TRAIN_COCO_JSON \
                --val_img_dir $VAL_IMG_DIR --val_coco_json $VAL_COCO_JSON \
                --train_method $TRAIN_METHOD

Evaluation on real images

The inference.py script has been provided to demo the performance of the trained detector or segmentation on a folder containing test images of real world aircrafts.

INPUT_DIR=/path/to/images
OUTPUT_DIR=/path/to/save/predicted_images
CONF=0.5
USE_GPU=True
CKPT_PATH=/path/to/model/model_final.pth
NUM_IMAGES=2

python inference.py --input_dir ${INPUT_DIR} --output_dir ${OUTPUT_DIR} --conf ${CONF} \
                    --use_gpu ${USE_GPU} --ckpt_path ${CKPT_PATH} --num_images ${NUM_IMAGES}

License

The source code of this repository is released only for academic use. See the license file for details.

Notes

The codes of this repository are built upone the folloiwng open sources. Thanks to the authors for sharing the code!

• Pre-trained machine learning models and tools from Detectron2
• Synthetic dataset is generated by Microsoft Mixed Reality

Citation

If you find this project useful or utilize the OnlyPlanes Dataset for your research, please kindly cite our paper.

@article{hale2022OnlyPlanes,
  title={OnlyPlanes: Incrementally Tuning Synthetic Training Datasets for Satellite Object Detection},
  author={Hale, Phillip and Tiday, Luke and Urbina, Pedro},
  number={arXiv:TBD},
  year={2022}
}