Awesome
Category Query Learning for Human-Object Interaction Classification
This repository contains the official implementation of the paper "Category Query Learning for Human-Object Interaction Classification" (CVPR 2023).
<div align="center"> <img src=".github/cql_overview.png" width="900px" /> <p>Overview of our method</p> </div>Unlike most previous HOI methods that focus on learning better human-object features, we propose a novel and complementary approach called category query learning. Such queries are explicitly associated to interaction categories, converted to image specific category representation via a transformer decoder, and learnt via an auxiliary image-level classification task.
Preparation
Dependencies
You can install the python libraries with this command:
pip install -r requirements.txt
Actually torch>=1.5.1
is probably fine, but we have not tested all versions.
Dataset
HICO-DET
The images in HICO-DET dataset can be downloaded here. After finishing downloading, unpack the tarball (hico_20160224_det.tar.gz
) to the data
directory.
The annotation files can be downloaded from here. Note the training annotation is slightly different than previous versions as some HOI instances with the same human and object are merged as one, and this may affect the performance by about 0.1 mAP. The downloaded annotation files should be placed as follows.
qpic
|─ data
│ └─ hico_20160224_det
| |─ annotations
| | |─ trainval_hico.json
| | |─ test_hico.json
| | └─ corre_hico.npy
: :
V-COCO
First clone the repository of V-COCO from here, and then follow the instruction to generate the file instances_vcoco_all_2014.json
. Next, download the prior file prior.pickle
from here. Place the files and make directories as follows.
qpic
|─ data
│ └─ v-coco
| |─ data
| | |─ instances_vcoco_all_2014.json
| | :
| |─ prior.pickle
| |─ images
| | |─ train2014
| | | |─ COCO_train2014_000000000009.jpg
| | | :
| | └─ val2014
| | |─ COCO_val2014_000000000042.jpg
| | :
| |─ annotations
: :
Following most previous methods, the annotation file have to be converted to the HOIA format. The conversion can be conducted as follows.
PYTHONPATH=data/v-coco \
python convert_vcoco_annotations.py \
--load_path data/v-coco/data \
--prior_path data/v-coco/prior.pickle \
--save_path data/v-coco/annotations
Note that only Python2 can be used for this conversion because vsrl_utils.py
in the v-coco repository shows a error with Python3.
V-COCO annotations with the HOIA format, corre_vcoco.npy
, test_vcoco.json
, and trainval_vcoco.json
will be generated to annotations
directory.
Pre-trained parameters for initialization
We also use the COCO-pretrained parameters of DETR, same as previous methods. You can download the DETR parameters from here for the ResNet50 backbone, and here for the ResNet101 backbone. For HICO-DET, convert the parameters with the following command.
python convert_parameters.py \
--load_path params/detr-r50-e632da11.pth \
--save_path params/detr-r50-pre-hico.pth
For V-COCO, convert the pre-trained parameters with the following command.
python convert_parameters.py \
--load_path params/detr-r50-e632da11.pth \
--save_path params/detr-r50-pre-vcoco.pth \
--dataset vcoco
Training
HICO-DET
After the preparation, you can start the training with the following command.
For the HICO-DET training.
bash cfg/hicodet.sh
See this script for detail about training configurations on HICO-DET.
V-COCO
For the V-COCO training.
bash cfg/vcoco.sh
See this script for detail about training configurations on V-COCO.
Evaluation
The evaluation is automatically conducted at the end of each epoch during the training. The results are written in the log.txt
file in the output directory. However, they are not evaluated by the standard evaluation tools of HICO-DET and V-COCO. For this, additional steps need to be performed, which we describe below.
HICO-DET
On HICO-DET, you can also conduct the evaluation with trained parameters as follows.
bash cfg/eval_hicodet.sh
Then we perform the official evaluation on HICO-DET. For this, we follow the evaluation steps introduced in PPDM.
V-COCO
For the official evaluation of V-COCO, a pickle file of detection results have to be generated. You can generate the file as follows.
bash cfg/eval_vcoco.sh
Then we perform the official evaluation steps on V-COCO with this pickle file using the official evaluation toolkit. You can refer to CDN for detailed steps.
Results and trained models
HICO-DET:
Full (D) | Rare (D) | Non-rare (D) | Full(KO) | Rare (KO) | Non-rare (KO) | Checkpoint | Config | Training log | |
---|---|---|---|---|---|---|---|---|---|
QPIC+CQL | 31.07 | 25.21 | 32.82 | 33.74 | 28.03 | 35.45 | ckpt | cfg | log |
QPIC+CQL* | 31.72 | 27.40 | 33.01 | 34.25 | 28.77 | 35.89 | ckpt | cfg | log |
D: Default, KO: Known object. *: Additional trick is applied in this setting. Please see the config for more details.
Note that the results in the training log are not exactly equal to the standard evaluation results. Please see the evaluation part above.
V-COCO:
Scenario 1 | Scenario 2 | Checkpoint | Config | Training log | |
---|---|---|---|---|---|
QPIC+CQL | 63.57 | 65.70 | ckpt | cfg | log |
Citation
Please kindly consider citing our paper if it helps your research.
@inproceedings{xie2023category,
title={Category Query Learning for Human-Object Interaction Classification},
author={Xie, Chi and Zeng, Fangao and Hu, Yue and Liang, Shuang and Wei, Yichen},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={15275--15284},
year={2023}
}
Acknowledgement
Some of this work's code is built upon several prior works including DETR, QPIC and GEN-VLKT.