Home

Awesome

[NeurIPS 2022 Spotlight] RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection

arXiv GitHub Stars GitHub Forks Hits visitors

Updates

<!--- **Update on Jan. 28th, 2023**: I updated the link of the pre-trained model in **Inference on Custom Images**. This updated one is fine-tuned on HICO. The old one is the pre-trained parameters. Sorry about the mistake. The code is still under pre-release review because we encountered a minor technical issue. It will be out this week. Sincere apologies! -->

Todo List

Note that if you can not get access to the links provided below, try using another browser or contact me by e-mail.

Model Outline

This repo contains the implementation of various methods to resolve HOI detection (not limited to RLIP), aiming to serve as a benchmark for HOI detection. Below methods are included in this repo:

<!--- a newly-proposed Cross-Modal Relation Detection method for HOI detection. -->

Citation

If you find our work inspiring or our code/annotations useful to your research, please cite:

@inproceedings{Yuan2022RLIP,
  title={RLIP: Relational Language-Image Pre-training for Human-Object Interaction Detection},
  author={Yuan, Hangjie and Jiang, Jianwen and Albanie, Samuel and Feng, Tao and Huang, Ziyuan and Ni, Dong and Tang, Mingqian},
  booktitle={Advances in Neural Information Processing Systems (NeurIPS)},
  year={2022}
}

@inproceedings{Yuan2023RLIPv2,
  title={RLIPv2: Fast Scaling of Relational Language-Image Pre-training},
  author={Yuan, Hangjie and Zhang, Shiwei and Wang, Xiang and Albanie, Samuel and Pan, Yining and Feng, Tao and Jiang, Jianwen and Ni, Dong and Zhang, Yingya and Zhao, Deli},
  booktitle={Proceedings of the IEEE/CVF International Conference on Computer Vision},
  year={2023}
}

@inproceedings{Yuan2022OCN,
  title={Detecting Human-Object Interactions with Object-Guided Cross-Modal Calibrated Semantics},
  author={Hangjie Yuan and Mang Wang and Dong Ni and Liangpeng Xu},
  booktitle={AAAI},
  year={2022}
}

Inference on Custom Images

To facilitate the use of custom images without annotations, I have implemented a version of code that supports this. In terms of the pre-trained model, I am using the best-performing RLIP-ParSe. To begin, please place your images in the folder custom_imgs. Then, you could try running the code below:

cd /PATH/TO/RLIP
# RLIP-ParSe
bash scripts/Inference_on_custom_imgs.sh

After successfully running the code, the generated results will be available in the folder custom_imgs/result. I have tested the code on a single Tesla A100 with batch sizes of 1, 2, 3 and 4. Note that, by default, we saved all the detection results (64 pairs). In most cases, it is possible to set a threshold for the verb_scores (multiplication of relation scores and object scores) in the saved results, which will enable their use in your own work. You can do it by yourself and tune the threshold for your work. As a recommendation, 0.25 (0.5*0.5) might be a good start to try out.

Annotation Preparation

DatasetSettingDownload
VGPre-trainingLink
HICO-DETFew-shot 1%, 10%Link
HICO-DETZero-shot (UC-NF, UC-RF)*Link
HICO-DETRelation Label Noise (10%, 30%, 50%)Link

Note: β‘  * Zero-shot (NF) do not need any HICO-DET annotations for fine-tuning, so we only provide training annotations for the UC-NF and UC-RF setting.

Pre-training Dataset (Visual Genome) preparation

Firstly, we could download VG dataset from the official link, inclduing images Part I and Part II. (Note: If the official website is not working, you can use the link that I provide: Images and Images2.) The annotations after pre-processing could be downloaded from the link above, which is used for pre-training. Note that this is generated from scene_graphs.json file by several pre-processing steps to remove redundant triplets. Also, several settings mentioned below also need the annotations that we provide. VG dataset and its corresponding annotations should be organized as follows:

VG
 |─ annotations
 |   |β€” scene_graphs_after_preprocessing.json
 |   :
 |β€” images
 |   |β€” 2409818.jpg
 |   |β€” n102412.jpg
 :   :

Downstream Dataset preparation

1. HICO-DET

HICO-DET dataset can be downloaded here. After finishing downloading, unpack the tarball (hico_20160224_det.tar.gz) to the data directory.

Instead of using the original annotations files, we use the annotation files provided by the PPDM authors. The annotation files can be downloaded from here. The downloaded annotation files have to be placed as follows.

qpic
 |─ data
 β”‚   └─ hico_20160224_det
 |       |─ annotations
 |       |   |─ trainval_hico.json
 |       |   |─ test_hico.json
 |       |   └─ corre_hico.npy
 :       :

2. V-COCO

First clone the repository of V-COCO from here, and then follow the instruction to generate the file instances_vcoco_all_2014.json. Next, download the prior file prior.pickle from here. Place the files and make directories as follows.

qpic
 |─ data
 β”‚   └─ v-coco
 |       |─ data
 |       |   |─ instances_vcoco_all_2014.json
 |       |   :
 |       |─ prior.pickle
 |       |─ images
 |       |   |─ train2014
 |       |   |   |─ COCO_train2014_000000000009.jpg
 |       |   |   :
 |       |   └─ val2014
 |       |       |─ COCO_val2014_000000000042.jpg
 |       |       :
 |       |─ annotations
 :       :

The annotation file has to be converted to the HOIA format. The conversion can be conducted as follows.

PYTHONPATH=data/v-coco \
        python convert_vcoco_annotations.py \
        --load_path data/v-coco/data \
        --prior_path data/v-coco/prior.pickle \
        --save_path data/v-coco/annotations

Note that only Python2 can be used for this conversion because vsrl_utils.py in the v-coco repository shows a error with Python3.

V-COCO annotations with the HOIA format, corre_vcoco.npy, test_vcoco.json, and trainval_vcoco.json will be generated to annotations directory.

RLIP Pre-training

Since RLIP pre-trained on VG and COCO dataset, we provide a series of pre-trained weights for you to use. Weights in the table below are used to initialize ParSe/ParSeD/RLIP-ParSe/RLIP-ParSeD for pre-training or fine-tuning.

ModelPre-training ParadigmPre-training DatasetBackboneBase DetectorDownload
MDETR-ParSeModulated DetectionGoldG+ResNet-101DETRLink
ParSeDObject DetectionVGResNet-50DDETRLink
ParSeDObject DetectionCOCOResNet-50DDETRLink
ParSeObject DetectionCOCOResNet-50DETRLink (Query128) <br> Link (Query200)
ParSeObject DetectionCOCOResNet-101DETRLink (Query128)
RLIP-ParSeDRLIPVGResNet-50DDETRLink
RLIP-ParSeDRLIPCOCO + VGResNet-50DDETRLink
RLIP-ParSeRLIPCOCO + VGResNet-50DETRLink

With respect to the first, third and fourth line of the pre-trained weights, they are produced from the original codebase. For further reference, you could visit DDETR and MDETR. The weights provided above are transformed from original codebases. With respect to the last three models' weights, optionally, you can pre-train the model yourself by running the corresponding script:

cd /PATH/TO/RLIP
# RLIP-ParSe
bash scripts/Pre-train_RLIP-ParSe_VG.sh
# RLIP-ParSeD
bash scripts/Pre-train_RLIP-ParSeD_VG.sh

Note that above scripts contain the installation of dependencies, which could be done independently. For the --pretrained parameter in script, you could ignore it to pre-train from scratch or use ParSeD parameters pre-trained on COCO.

1. Fully-finetuning

Weights in the table below are fully-fined weights of ParSe/ParSeD/RLIP-ParSe/RLIP-ParSeD using pre-trained weights from the above table.

ModelPre-training ParadigmPre-training DatasetBackboneBase DetectorFull / Rare / Non-RareDownload
ParSeDRLIPCOCOResNet-50DDETR29.12 / 22.23 / 31.17Link
ParSeRLIPCOCOResNet-50DETR31.79 / 26.36 / 33.41Link
ParSeRLIPCOCOResNet-101DETR32.76 / 28.59 / 34.01Link
RLIP-ParSeDRLIPVGResNet-50DDETR29.21 / 24.45 / 30.63Link
RLIP-ParSeDRLIPCOCO + VGResNet-50DDETR30.70 / 24.67 / 32.50Link
RLIP-ParSeRLIPCOCO + VGResNet-50DETR32.84 / 26.85 / 34.63Link

2. Few-shot (0, 1%, 10%)

The scripts are identical to those for fully fine-tuning. The major difference is that we need to add --few_shot_transfer 10 \ for 10% data of few-shot transfer and --few_shot_transfer 1 \ for 1% data of few-shot transfer. Note that we only fine-tune for 10 epochs with the lr dropping at 7th epoch. Thus, you need to change --lr_drop and --epochs in the script accordingly.

cd /PATH/TO/RLIP
# RLIP-ParSeD on HICO
bash scripts/Fine-tune_RLIP-ParSeD_HICO.sh
# RLIP-ParSe on HICO
bash scripts/Fine-tune_RLIP-ParSe_HICO.sh
# ParSe on HICO
bash scripts/Fine-tune_ParSe_HICO.sh
# ParSeD on HICO
bash scripts/Fine-tune_ParSeD_HICO.sh

When there is no extra data provided (0 percent of few-shot transfer), please refer to zero-shot NF setting, but performance is present here.

ModelPre-training ParadigmPre-training DatasetBackboneBase DetectorDataFull / Rare / Non-RareDownload
RLIP-ParSeDRLIPCOCO + VGResNet-50DDETR013.92 / 11.20 / 14.73Link*
RLIP-ParSeDRLIPCOCO + VGResNet-50DDETR1%18.30 / 16.22 / 18.92Link
RLIP-ParSeDRLIPCOCO + VGResNet-50DDETR10%22.09 / 15.89 / 23.94Link
RLIP-ParSeRLIPCOCO + VGResNet-50DETR015.40 / 15.08 / 15.50Link*
RLIP-ParSeRLIPCOCO + VGResNet-50DETR1%18.46 / 17.47 / 18.76Link
RLIP-ParSeRLIPCOCO + VGResNet-50DETR10%22.59 / 20.16 / 23.32Link

Note: β‘  * means that the checkpoints are the same as the ones in the RLIP Pre-training table, since they do not involve any fine-tuning.

3. Zero-shot (NF, UC-RF, UC-NF)

With respect to NF setting, it is actually a testing procedure after loading the pre-trained weights. We could run the script below.

cd /PATH/TO/RLIP
# Zero-shot NF setting with RLIP-ParSe/RLIP-ParSeD
bash scripts/NF_Zero_shot.sh

With respect to UC-RF and UC-NF setting, training is required. We could run the script below by adding --zero_shot_setting UC-RF \ or --zero_shot_setting UC-NF \. Note that for UC-NF setting, we only fine-tunes for 40 epochs (lr dropping at 30th epoch) to avoid overfitting. Thus, you need to change --lr_drop 30 \ and --epochs 40 \ in the script accordingly.

cd /PATH/TO/RLIP
# RLIP-ParSeD on HICO
bash scripts/Fine-tune_RLIP-ParSeD_HICO.sh
# RLIP-ParSe on HICO
bash scripts/Fine-tune_RLIP-ParSe_HICO.sh
# ParSe on HICO
bash scripts/Fine-tune_ParSe_HICO.sh
# ParSeD on HICO
bash scripts/Fine-tune_ParSeD_HICO.sh
ModelPre-training ParadigmPre-training DatasetBackboneBase DetectorSettingFull / Rare / Non-RareDownload
RLIP-ParSeRLIPCOCO + VGResNet-50DETRUC-RF30.52 / 19.19 / 33.35Link
RLIP-ParSeRLIPCOCO + VGResNet-50DETRUC-NF26.19 / 20.27 / 27.67Link

Evaluation

The mAP on HICO-DET under the Full set, Rare set and Non-Rare Set will be reported during the training process.

The results for the official evaluation of V-COCO must be obtained by the generated pickle file of detection results.

cd /PATH/TO/RLIP
python generate_vcoco_official.py \
        --param_path /PATH/TO/CHECKPOINT \
        --save_path vcoco.pickle \
        --hoi_path /PATH/TO/VCOCO/DATA \

Then you should run following codes after modifying the path to get the final performance:

cd /PATH/TO/RLIP
python datasets/vsrl_eval.py

Acknowledgement

Part of this work's implemention refers to several prior works including OCN, QPIC, CDN, DETR, DDETR and MDETR.