Awesome

Regularized Unrolling Network (RU-Net)

This is an official implementation for CVPR 2022 paper "RU-Net: Regularized Unrolling Network for Scene Graph Generation".

Installation
Dataset setup
Config Introduction
Training
Validation and Testing
Models

Introduction

This project is based on the code of Scene Graph Benchmark in Pytorch.

Method overview

overall structure

Installation

Clone this repo

git clone https://github.com/siml3/RU-Net.git
cd RU-Net

Create a conda virtual environment and activate it (optional)

conda create -n runet -y
conda activate runet

Prepare proper pytorch environment for your device.

Install dependencies

conda install ipython
conda install scipy
conda install h5py
pip install ninja yacs cython matplotlib tqdm opencv-python-headless overrides


git clone https://github.com/cocodataset/cocoapi.git
cd cocoapi/PythonAPI
python setup.py build_ext install
cd ../../

# GPU and Cuda environment is necessary for proper compilation of this repo
git clone https://github.com/NVIDIA/apex.git
cd apex
python setup.py install --cuda_ext --cpp_ext
cd ../

Compile extensions and install this repo in develop mode
```
python setup.py build develop
```

Dataset

Please refer to the DATASET.md

Training

Detector Fine-tune

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10001 --nproc_per_node=4 tools/detector_pretrain_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 8 TEST.IMS_PER_BATCH 4 DTYPE "float32" SOLVER.MAX_ITER 50000 SOLVER.STEPS "(30000, 45000)" SOLVER.VAL_PERIOD 2000 SOLVER.CHECKPOINT_PERIOD 2000 MODEL.RELATION_ON False SOLVER.PRE_VAL False OUTPUT_DIR /path/to/output

Training on Scene Graph Generation

The training scripts are available in the folder scripts

If you have multi gpus, try the command as the follows for SGCLS:

# four cards
PYTHONPATH=$PWD CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10022 --nproc_per_node=4 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.PRETRAINED_DETECTOR_CKPT checkpoints/pretrained_faster_rcnn/model_final.pth OUTPUT_DIR "exp/RU-Net" SOLVER.IMS_PER_BATCH 12 TEST.IMS_PER_BATCH 4 DTYPE "float16" SOLVER.PRE_VAL True MODEL.ROI_RELATION_HEAD.PREDICTOR RUNetPredictor MODEL.ROI_RELATION_HEAD.MP_LAYER_NUM 5 SOLVER.BASE_LR 0.0025 GLOVE_DIR glove MODEL.MODEL.ROI_RELATION_HEAD.L21_LOSS 0.7

Testing

gpu_id=0,1
port=10086
gpu_num=2
output_dir=""

PYTHONPATH=$PWD CUDA_VISIBLE_DEVICES=${gpu_id} python -m torch.distributed.launch --master_port ${port} --nproc_per_node=${gpu_num} tools/relation_test_net.py --config-file "${output_dir}/config.yml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False OUTPUT_DIR "${output_dir}" TEST.IMS_PER_BATCH ${gpu_num} DTYPE "float16" GLOVE_DIR glove MODEL.ROI_RELATION_HEAD.L21_LOSS 0.7

Models

Here we provide our pretrained model via BaiduNetDisk:

Link：https://pan.baidu.com/s/1zmHPKPpeUioVQXGCwn9rAA Extraction code：1010

Citation

If RU-Net is helpful for your research, we'd really appreciate it if you could cite this paper:

@inproceedings{lin2022ru,
  title={RU-Net: Regularized Unrolling Network for Scene Graph Generation},
  author={Lin, Xin and Ding, Changxing and Zhang, Jing and Zhan, Yibing and Tao, Dacheng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19457--19466},
  year={2022}
}