Home

Awesome

FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors

This repository is an official implementation of the paper Feature Aggregated Queries for Transformer-based Video Object Detectors.

Installation

The codebase is built on top of Deformable DETR.

Requirements

Usage

Dataset preparation

  1. Please download ILSVRC2015 DET and ILSVRC2015 VID dataset from here. Then we covert jsons of two datasets by using the code. The joint json of two datasets is provided. The After that, we recommend to symlink the path to the datasets to datasets/. And the path structure should be as follows:
code_root/
└── data/
    └── vid/
        ├── Data
            ├── VID/
            └── DET/
        └── annotations/
        	  ├── imagenet_vid_train.json
            ├── imagenet_vid_train_joint_30.json
        	  └── imagenet_vid_val.json

Training

We use ResNet50 and ResNet101 as the network backbone. We train our FAQ with ResNet50 as backbone as following:

Training on single node

  1. Train SingleBaseline. You can download COCO pretrained weights from Deformable DETR.
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh $1 r50 $2 configs/r50_train_single.sh
  1. Train FAD. Using the model weights of SingleBaseline as the resume model.
GPUS_PER_NODE=8 ./tools/run_dist_launch.sh $1 r50 $2 configs/r50_train_multi.sh

Training on slurm cluster

If you are using slurm cluster, you can simply run the following command to train on 1 node with 8 GPUs:

GPUS_PER_NODE=8 ./tools/run_dist_slurm.sh <partition> r50 8 configs/r50_train_multi.sh

Evaluation

You can get the config file and pretrained model of FAQ (the link is in "Main Results" session), then put the pretrained_model into correponding folder.

code_root/
└── exps/
    └── our_models/
        ├── COCO_pretrained_model
        ├── exps_single
        └── exps_multi

And then run following command to evaluate it on ImageNET VID validation set:

GPUS_PER_NODE=8 ./tools/run_dist_launch.sh $1 eval_r50 $2 configs/r50_eval_multi.sh

Citing FAQ

If you find FAQ useful in your research, please consider citing:

@misc{cui2023faq,
      title={FAQ: Feature Aggregated Queries for Transformer-based Video Object Detectors}, 
      author={Yiming Cui and Linjie Yang},
      year={2023},
      eprint={2303.08319},
      archivePrefix={arXiv},
      primaryClass={cs.CV}
}