Home

Awesome

Learning to Learn how to Learn: Self-Adaptive Visual Navigation using Meta-Learning

By Mitchell Wortsman, Kiana Ehsani, Mohammad Rastegari, Ali Farhadi and Roozbeh Mottaghi (Oral Presentation at CVPR 2019).

CVPR 2019 Paper | Video | BibTex

IntuitionExamples

There is a lot to learn about a task by actually attempting it! Learning is continuous, i.e. we learn as we perform. Traditional navigation approaches freeze the model during inference (top row in the intuition figure above). In this paper, we propose a self-addaptive agent for visual navigation that learns via self-supervised interaction with the environment (bottom row in the intuition figure above).

Citing

If you find this project useful in your research, please consider citing:

@InProceedings{Wortsman_2019_CVPR,
  author={Mitchell Wortsman and Kiana Ehsani and Mohammad Rastegari and Ali Farhadi and Roozbeh Mottaghi},
  title={Learning to Learn How to Learn: Self-Adaptive Visual Navigation Using Meta-Learning},
  booktitle = {The IEEE Conference on Computer Vision and Pattern Recognition (CVPR)},
  month = {June},
  year = {2019}
}

Results

ModelSPL ≥ 1Success ≥ 1SPL ≥ 5Success ≥ 5
SAVN16.15 ± 0.540.86 ± 1.213.91 ± 0.528.70 ± 1.5
Scene Priors15.47 ± 1.135.13 ± 1.311.37 ± 1.622.25 ± 2.7
Non-Adaptive A3C14.68 ± 1.833.04 ± 3.511.69 ± 1.921.44 ± 3.0

Setup

tar -xzf pretrained_models.tar.gz
tar -xzf data.tar.gz

The data folder contains:

Note that the starting positions and scenes for the test and validation set may be found in test_val_split.

If you wish to access the RGB images in addition to the ResNet features, replace thor_offline_data with thor_offlline_data_with_images. If you wish to run your model on the image files, add the command line argument --images_file_name images.hdf5.

Evaluation using Pretrained Models

Use the following code to run the pretrained models on the test set. Add the argument --gpu-ids 0 1 to speed up the evaluation by using GPUs.

SAVN

python main.py --eval \
    --test_or_val test \
    --episode_type TestValEpisode \
    --load_model pretrained_models/savn_pretrained.dat \
    --model SAVN \
    --results_json savn_test.json 

cat savn_test.json 

Scene Priors

python main.py --eval \
    --test_or_val test \
    --episode_type TestValEpisode \
    --load_model pretrained_models/gcn_pretrained.dat \
    --model GCN \
    --glove_dir ./data/gcn \
    --results_json scene_priors_test.json

cat scene_priors_test.json 

Non-Adaptvie-A3C

python main.py --eval \
    --test_or_val test \
    --episode_type TestValEpisode \
    --load_model pretrained_models/nonadaptivea3c_pretrained.dat \
    --results_json nonadaptivea3c_test.json

cat nonadaptivea3c_test.json

The result may vary depending on system and set-up though we obtain:

ModelSPL ≥ 1Success ≥ 1SPL ≥ 5Success ≥ 5
SAVN16.1342.2014.3030.09
Scene Priors14.8636.9011.4924.70
Non-Adaptive A3C14.1032.4010.7319.16

The results in the initial submission (shown below) were the best (in terms of success on the validation set). After the initial submission, we trained the model 5 times from scratch to obtain error bars, which you may find in results.

ModelSPL ≥ 1Success ≥ 1SPL ≥ 5Success ≥ 5
SAVN16.1342.1013.1930.54
Non-Adaptive A3C13.7332.9010.8820.66

How to Train your SAVN

You may train your own models by using the commands below.

Training SAVN

python main.py \
    --title savn_train \
    --model SAVN \
    --gpu-ids 0 1 \
    --workers 12

Training Non-Adaptvie A3C

python main.py \
    --title nonadaptivea3c_train \
    --gpu-ids 0 1 \
    --workers 12

How to Evaluate your Trained Model

You may use the following commands for evaluating models you have trained.

SAVN

python full_eval.py \
    --title savn \
    --model SAVN \
    --results_json savn_results.json \
    --gpu-ids 0 1
    
cat savn_results.json

Non-Adaptive A3C

python full_eval.py \
    --title nonadaptivea3c \
    --results_json nonadaptivea3c_results.json \
    --gpu-ids 0 1
    
cat nonadaptivea3c_results.json

Random Agent

python main.py \
    --eval \
    --test_or_val test \
    --episode_type TestValEpisode \
    --title random_test \
    --agent_type RandomNavigationAgent \
    --results_json random_results.json
    
cat random_results.json