Home

Awesome

Regularized Unrolling Network (RU-Net)

LICENSE Python PyTorch

This is an official implementation for CVPR 2022 paper "RU-Net: Regularized Unrolling Network for Scene Graph Generation".

Contents

  1. Installation
  2. Dataset setup
  3. Config Introduction
  4. Training
  5. Validation and Testing
  6. Models

Introduction

This project is based on the code of Scene Graph Benchmark in Pytorch.

Method overview

overall structure

Installation

git clone https://github.com/siml3/RU-Net.git
cd RU-Net
conda create -n runet -y
conda activate runet

Dataset

Please refer to the DATASET.md

Training

Detector Fine-tune

CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10001 --nproc_per_node=4 tools/detector_pretrain_net.py --config-file "configs/e2e_relation_detector_X_101_32_8_FPN_1x.yaml" SOLVER.IMS_PER_BATCH 8 TEST.IMS_PER_BATCH 4 DTYPE "float32" SOLVER.MAX_ITER 50000 SOLVER.STEPS "(30000, 45000)" SOLVER.VAL_PERIOD 2000 SOLVER.CHECKPOINT_PERIOD 2000 MODEL.RELATION_ON False SOLVER.PRE_VAL False OUTPUT_DIR /path/to/output 

Training on Scene Graph Generation

The training scripts are available in the folder scripts

If you have multi gpus, try the command as the follows for SGCLS:

# four cards
PYTHONPATH=$PWD CUDA_VISIBLE_DEVICES=0,1,2,3 python -m torch.distributed.launch --master_port 10022 --nproc_per_node=4 tools/relation_train_net.py --config-file "configs/e2e_relation_X_101_32_8_FPN_1x.yaml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False MODEL.PRETRAINED_DETECTOR_CKPT checkpoints/pretrained_faster_rcnn/model_final.pth OUTPUT_DIR "exp/RU-Net" SOLVER.IMS_PER_BATCH 12 TEST.IMS_PER_BATCH 4 DTYPE "float16" SOLVER.PRE_VAL True MODEL.ROI_RELATION_HEAD.PREDICTOR RUNetPredictor MODEL.ROI_RELATION_HEAD.MP_LAYER_NUM 5 SOLVER.BASE_LR 0.0025 GLOVE_DIR glove MODEL.MODEL.ROI_RELATION_HEAD.L21_LOSS 0.7 

Testing

gpu_id=0,1
port=10086
gpu_num=2
output_dir=""

PYTHONPATH=$PWD CUDA_VISIBLE_DEVICES=${gpu_id} python -m torch.distributed.launch --master_port ${port} --nproc_per_node=${gpu_num} tools/relation_test_net.py --config-file "${output_dir}/config.yml" MODEL.ROI_RELATION_HEAD.USE_GT_BOX True MODEL.ROI_RELATION_HEAD.USE_GT_OBJECT_LABEL False OUTPUT_DIR "${output_dir}" TEST.IMS_PER_BATCH ${gpu_num} DTYPE "float16" GLOVE_DIR glove MODEL.ROI_RELATION_HEAD.L21_LOSS 0.7

Models

Here we provide our pretrained model via BaiduNetDisk:

Link:https://pan.baidu.com/s/1zmHPKPpeUioVQXGCwn9rAA Extraction code:1010

Citation

If RU-Net is helpful for your research, we'd really appreciate it if you could cite this paper:

@inproceedings{lin2022ru,
  title={RU-Net: Regularized Unrolling Network for Scene Graph Generation},
  author={Lin, Xin and Ding, Changxing and Zhang, Jing and Zhan, Yibing and Tao, Dacheng},
  booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
  pages={19457--19466},
  year={2022}
}