Awesome

A Behavioral Approach to Visual Navigation with Graph Localization Networks

Paper: A Behavioral Approach to Visual Navigation with Graph Localization Networks
Website: https://graphnav.stanford.edu/
Video: https://youtu.be/nN3B1F90CFM

Citing

@INPROCEEDINGS{Savarese-RSS-19, 
    AUTHOR    = {Kevin Chen AND Juan Pablo de Vicente AND Gabriel Sepulveda AND Fei Xia AND Alvaro Soto AND Marynel Vázquez AND Silvio Savarese}, 
    TITLE     = {A Behavioral Approach to Visual Navigation with Graph Localization Networks}, 
    BOOKTITLE = {Proceedings of Robotics: Science and Systems}, 
    YEAR      = {2019}, 
    ADDRESS   = {FreiburgimBreisgau, Germany}, 
    MONTH     = {June}, 
    DOI       = {10.15607/RSS.2019.XV.010} 
} 

Environment Setup

GraphNav Setup

# Need python 2 when using ROS
conda create -n graphnav python=3.7

# Install requirements
pip install -r requirements.txt

# Add graphnav
export PYTHONPATH=$PYTHONPATH:<path-to-graphnav>/src

Gibson Setup

Install Gibson v2. Follow the directions here for ROS-specific instructions.

Example Gibson installation below using Anaconda. For more details, see Gibson.

conda create -n graphnav-gibson-py2 python=2.7
conda activate graphnav-gibson-py2
pip install -e .  # From GibsonEnvV2 directory
conda deactivate

If not done already, execute the ROS-specific Gibson instructions.

Example of path setup below. Put these in a script if you would like.

export PYTHONPATH=$PYTHONPATH:<path-to-GibsonEnvV2>
export PYTHONPATH=$PYTHONPATH:<path-to-graphnav-gibson-py2-venv>/lib/python2.7/site-packages  # Path to virtual env if you used one (use Python 2)
export PYTHONPATH=$PYTHONPATH:<path-to-graphnav>/src

# Make sure to remove these from PATH as specified in the Gibson ROS instructions (if using Anaconda)
echo $PATH | grep -oP "[^:;]+" | grep conda

Add ROS packages for Gibson and graphnav, just like the ROS-specific Gibson instructions.

# Gibson
ln -s <path-to-GibsonEnvV2>/examples/ros/gibson-ros/ ~/catkin_ws/src/
cd ~/catkin_ws && catkin_make

ln -s <path-to-graphnav> ~/catkin_ws/src/semnav_ros
cd ~/catkin_ws && catkin_make

Dataset Setup

Download the data (collected from Gibson 1) from here. This data is used for training the behaviors networks.

Replace the following in graphnav/config.py:

DATASET_ROOT = '/data/graphnav/trajectory-data'                                                                                                                                                                                            
LOG_ROOT = '/data/graphnav/experiments'                                                                                                                                                                                                    
STANFORD_JSON = '<path-to-graphnav>/data/semantic_labels.json'                                                                                                                                                                     
MAPS_ROOT = '<path-to-graphnav>/maps/v0.2'

For faster data loader initialization in each run:

# Edit the directory in build_dataset_cache.py
python build_dataset_cache.py --dataset_type frame_by_frame

Behavior Networks

Training

Example train command:

python learning/behavior_net/behavior_trainer.py \
    --batch_size 32 \
    --learning_rate 0.0001 \
    --dataset v0.2 \
    --n_workers 3 \
    --behaviornet_type behavior_rnn \
    --dataset_type temporal \
    --print_freq 250 \
    --val_freq 1000 \
    --ckpt_freq 1000 \
    --n_epochs 500 \
    --log_dir v0.2/behavior_rnn/tl \
    --behavior_id tl \
    --n_frames_per_sample 20

An example config for different behavior networks:

# behaviornet_type, behavior_id, dataset_type
behaviornets = [
        ('behavior_rnn', 'tl', 'temporal'),
        ('behavior_rnn', 'tr', 'temporal'),
        ('behavior_cnn', 'cf', 'temporal'),
        ('behavior_rnn', 's', 'temporal'),
        ('behavior_cnn', 'fd', 'temporal'),
        ]

Graph Localization Network

Training

python learning/graph_net/graph_net_trainer.py \
    --dataset v0.2 \
    --learning_rate 1e-4 \
    --n_workers 3 \
    --behaviornet_type graph_net \
    --dataset_type graph_net \
    --print_freq 100 \
    --val_freq 2000 \
    --ckpt_freq 2000 \
    --n_epochs 500 \
    --n_frames_per_sample 20 \
    --log_dir v0.2/graph_net \
    --aggregate_method sum \
    --use_gn_augmentation

Rollouts in Gibson

The code has been tested with ROS Kinetic (Python 2.7).

Setup for each run

Yaml

Edit the Gibson configuration yaml file. Pay particular attention to the model ID. Also make sure fov: 2.62 or 150 degrees (depending on Gibson 1 or Gibson 2) and resolution: 320.

Launch file

Edit the ROS launch file. Make sure the area yaml matches with the model ID specified in the turtlebot_rgbd.yaml and points to a yaml from the this repo.

Load checkpointed models

Edit graph_net_navigation_brain.py to load the desired checkpointed models.

Launching Gibson + ROS

roslaunch gibson2-ros turtlebot_gt_navigation.launch
rosrun semnav_ros navigation_planner.py
rosrun semnav_ros graph_net_navigation_brain.py --dataset_type graph_net  # No particle filter