Awesome

PRET: Planning with Directed Fidelity Trajectory for Vision and Language Navigation

Requirements

1. Install Matterport3D simulator.

   Put connectivity in ./(or create a soft link). Put build and Matterport3D data v1 in the ./data direcotry(or create a soft link).

2. Install python requirements.

   timm==0.9.5
   transformers==4.28.1
   torch==1.13.0, do not use >=2.0
   
   numpy
   pandas
   matplotlib
   python-opencv
   
   tqdm
   pyyaml
   networkx
   jsonlines
   

3. Download datasets, image features and model checkpoints from here. Download the data.zip and log.zip and unzip them.

Finally, the directory layout should looks like:

.
├── connectivity
├── data
│   ├── build
│   ├── candidate_buffer.pkl
│   ├── img_features
│   ├── pretrain_data
│   ├── pretrained_model
│   ├── R2R
│   ├── RxR
│   └── v1
├── log
│   └── commit
├── src
├── .gitignore
├── pretrain.sh
├── README.md
└── run.sh

Train

Pretrain:

sh pretrain.sh R2R
# sh pretrain.sh RxR

Fine-tune:

sh run.sh R2R
# sh run.sh RxR

Citation

@article{lu2024pret,
  title={PRET: Planning with Directed Fidelity Trajectory for Vision and Language Navigation},
  author={Lu, Renjie and Meng, Jingke and Zheng, Wei-Shi},
  journal={arXiv preprint arXiv:2407.11487},
  year={2024}
}