Awesome

HOP-REVERIE-Challenge

This respository is the code of REVERIE-Challenge using HOP. The code is based on Recurrent-VLN-BERT. Thanks to Yicong Hong for releasing the Recurrent-VLN-BERT code.

Prerequisites

Installation

• Install docker Please check here to install docker.
• Create container To pull the image:

  docker pull starrychiao/hop-recurrent:v1
  

  If your CUDA version is 11.3, you can pull the image:

  docker push starrychiao/vlnbert-2022-3090:tagname
  

  To create the container:

  docker run -it --ipc host  --shm-size=1024m --gpus all --name your_name  --volume "your_directory":/root/mount/Matterport3DSimulator starrychiao/hop-recurrent:v1
  

• Set up

  docker start "your container id or name"
  docker exec -it "your container id or name" /bin/bash
  cd /root/mount/Matterport3DSimulator
  

Data Preparation

Please follow the instructions below to prepare the data in directories:

• MP3D navigability graphs: connectivity

  • Download the connectivity maps .

• MP3D image features: img_features

  • Download the Scene features (ResNet-152-Places365).

• REVERIE data: data_v2

  • Download the REVERIE data.
  • Download the object features [reverie_obj_feats_v2.pkl] from Google Drive or Baidu Cloud Disk [code: nubg].

• objpos.json

• After downloading data you should see the following folder structure:

    ├── data_v2
    │   └── BBoxS
    │       └── reverie_obj_feats_v2.pkl
    │   └── BBoxes_v2
    ├── REVERIE_train.json
    ├── REVERIE_val_seen.json
    ├── REVERIE_val_unseen.json
    ├── REVERIE_test.json
    └── objpos.json

Initial HOP weights

• Pre-trained HOP weights: load/hop
  • Download the pytorch_model.bin from here.

Training

bash run/agent.bash

Evaluating

• To generate submit_test.json

bash run/test.bash

• You can also evaluate results on REVERIE seen and REVERIE unseen splits.

python ./r2r_src/eval.py