Home

Awesome

Visual-Graph-Memory

This is an official GitHub Repository for paper "Visual Graph Memory with Unsupervised Representation for Visual Navigation", which is accepted as a regular paper (poster) in ICCV 2021.

Setup

Requirements

The source code is developed and tested in the following setting.

Please refer to habitat-sim and habitat-lab for installation.

To set the environment, run:

pip install -r requirements.txt

Habitat Data (Gibson, MP3D) Setup

Most of the scripts in this code build the environments assuming that the gibson/mp3d dataset is in habitat-lab/data/ folder.

The recommended folder structure of habitat-api (or habitat-lab):

habitat-api (or habitat-lab)
  └── data
      └── datasets
      │   └── pointnav
      │       └── gibson
      │           └── v1
      │               └── train
      │               └── val
      └── scene_datasets
          └── gibson_habitat
              └── *.glb, *.navmeshs  

otherwise, you should edit the data path in these lines.

VGM Demonstration

To visualize the VGM generation, run:

python vgm_demo.py --gpu 0 --num-proc 2

This command will show the online VGM generation during random exploration. The rendering window will show the generated VGM and the observations as follows:

vgm_demo_1 vgm_demo_1

Note that the top-down map and pose information are only used for visualization, not for the graph generation.

Imitation Learning

  1. Data generation
    python collect_IL_data.py --ep-per-env 200 --num-procs 4 --split train --data-dir /path/to/save/data
    
    This will generate the data for imitation learning. You can find some examples of the collected data in IL_data folder, and look into them with show_IL_data.ipynb.
  2. Training
    python train_bc.py --config configs/vgm.yaml --stop --gpu 0
    
  3. Evaluation

Reinforcement Learning

The reinforcement learning code is highly based on habitat-lab/habitat_baselines. To train the agent with reinforcement learning (PPO), run:

python train_rl.py --config configs/vgm.yaml --version EXPERIMENT_NAME --diff hard --render --stop --gpu 0

Evaluation

We provide evaluation code and the pretrained model.

python evaluate_random.py --config configs/vgm.yaml --version-name test --eval-ckpt VGM_ILRL.pth --stop --diff hard

You can use "evaluate_dataset.py" to evaluate VGM on public image-goal nav dataset (https://github.com/facebookresearch/image-goal-nav-dataset)

git clone https://github.com/facebookresearch/image-goal-nav-dataset.git
python evaluate_dataset.py --config configs/vgm.yaml --version-name test --eval-ckpt VGM_ILRL.pth --stop --diff hard

In the above dataset, the provided pretrained model shows following performances.

Easy(SR)Easy(SPL)Medium(SR)Medium(SPL)Hard(SR)Hard(SPL)Overall(SR)Overall(SPL)
0.760.400.760.560.620.490.710.48

Also, our VGM model shows following performances on NRNS Image-Goal Navigation dataset (https://meerahahn.github.io/nrns/data)

- Straight

Easy(SR)Easy(SPL)Medium(SR)Medium(SPL)Hard(SR)Hard(SPL)Overall(SR)Overall(SPL)
0.810.540.820.700.670.540.770.60

- Curved

Easy(SR)Easy(SPL)Medium(SR)Medium(SPL)Hard(SR)Hard(SPL)Overall(SR)Overall(SPL)
0.810.460.790.600.620.470.740.51