Awesome
Auxiliary Tasks and Exploration Enable ObjectNav
Joel Ye, Dhruv Batra, Abhishek Das, and Erik Wijmans.
Project Site: https://joel99.github.io/objectnav/
This repo is a code supplement for the paper "Auxiliary Tasks and Exploration Enable ObjectNav." The code is a fork of Habitat-Lab and borrows heavily from habitat-pointnav-aux.
Overview
The primary code contributions from the paper are located in:
- Auxiliary Tasks:
auxiliary_tasks.py
andsupervised_auxiliary_tasks.py
- Exploration Reward:
coverage.py
- Tether logic (denoted as "split" in code):
rollout_storage.py
,ppo.py
,multipolicy.py
- Experimental Configurations:
habitat_baselines/config/objectnav/full/*.yaml
- Misellaneous business logic:
ppo_trainer.py
- Scripts and Analysis:
./scripts/*
Requirements
The code is known to be compatible with Python 3.6, Habitat Lab v0.1.5 and Habitat Sim v0.1.6 (headless). We recommend you use conda to initialize the code as follows:
- Install habitat-sim. In the habitat-sim repo, run:
conda create -n <ENV_NAME> python=3.6 cmake=3.14.0
conda activate <ENV_NAME>
pip install -r requirements.txt
conda install habitat-sim=0.1.6 headless -c conda-forge -c aihabitat
Then, in this repo, run:
conda env update -f environment.yml --name <ENV_NAME>
Note: If you run into a pip error along the lines of a missing metadata file, it is likely an improperly cleaned package (e.g. numpy-1.19.5), just remove it.
Note 2: You may also need to run python setup.py develop --all
as in the habitat-lab installation process.
Data
The experiments were done on the Matterport3D dataset. Please refer to habitat-sim for the scene dataset and habitat-lab for the ObjectNav dataset.
Pre-trained Weights
You can download pretrained agent and rednet weights here. We provide the best checkpoint for agents as measured by their performance with RedNet segmentation:
base-full.31.pth
: 6-action agent.base4-full.33.pth
: 4-action agent.split_clamp.31.pth
: 6 + Tether agent.rednet_semmap_mp3d_tuned.pth
: RedNet weights.
Training
We train our models for around 8 GPU-weeks, on a SLURM-managed cluster. On such a cluster, you can launch your run with:
sbat ./scripts/ddppo_8gpu.sh <args>
With the appropriate configuration file. The code in run.py
is set up to parse out the arguments, so to train with the configuration .../objectnav/full/base.yaml
, write:
sbat ./scripts/ddppo_8gpu.sh base full
On a machine without SLURM, see, e.g. ./scripts/train_no_ckpt.sh
.
Evaluation
You can evaluate a trained checkpoint by configuring the arguments in ./scripts/eval_on.sh
. This forwards to habitat_baselines/run.py
, which accepts a number of flags. For example:
- Add a
-v
flag to generate videos of your agent. - Add a
-ra
flag to record trajectories and hidden states for subsequent analysis.
See run.py
for all arguments. Note that an evaluation on the full validation split can take a while (>1 GPU-day).
Citation
If you use this work, you can cite it as
@misc{ye2021auxiliary,
title={Auxiliary Tasks and Exploration Enable ObjectNav},
author={Joel Ye and Dhruv Batra and Abhishek Das and Erik Wijmans},
year={2021},
eprint={2104.04112},
archivePrefix={arXiv},
primaryClass={cs.CV}
}
Additional Notes
- The 300-episode subset mentioned throughout analysis was created by sampling episodes uniformly across the scenes in train and val splits, using
scripts/dataset_slicer.py
- The
.py
notebooks inscripts/
start with a comment denoting their use. - Though the configurations default to mixed precision training, it is not particularly faster in our experiments.
- The UNLICENSE applies to this project's changes on top of the habitat-lab codebase.