Awesome
Waypoint Predictor Training for Discrete-Continuous-VLN
Prerequisites
-
Please follow Discrete-Continuous-VLN to set up your environments, prepare scene dataset of MP3D, download the adapted mp3d connectivity graphs, and the pretrained ddppo ResNet encoder. Data and model path should be similar to Discrete-Continuous VLN. Download the adapted mp3d graphs from here.
-
Change the data path
/home/vlnce/vln-ce/data/
in the codes to your above data path. -
Change the
RAW_GRAPH_PATH
in the codes to your unzipped adapted mp3d connectivity graphs.
Preparing Training Data
- Run
gen_training_data/get_images_inputs.py
to get the RGBD inputs of the waypoint predictor, which will be saved attraining_data/rgbd_fov90
. - Run
gen_training_data/get_nav_dict.py
to get the computed navigability dict of each node, which will be saved atgen_training_data/nav_dicts
. - Run
gen_training_data/test_twm0.2_obstacle_first.py
to get the direct training data for training waypoint predictor, which will be saved attraining_data
.
Running
Training and Evaluation
Please run bash run_waypoint.bash
to train the waypoint predictor. If you only want to evaluate trained model, change --TRAINEVAL
to eval
. Modify the checkpoint_load_path
in waypoint_predictor.py
to evaluate different models.
Citation
Please cite our paper:
@InProceedings{Hong_2022_CVPR,
author = {Hong, Yicong and Wang, Zun and Wu, Qi and Gould, Stephen},
title = {Bridging the Gap Between Learning in Discrete and Continuous Environments for Vision-and-Language Navigation},
booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
month = {June},
year = {2022}
}