Home

Awesome

Hierarchical and Partially Observable Goal-driven Policy Learning with Goals Relational Graph

<img src="https://github.com/Xin-Ye-1/HRL-GRG/blob/main/overview.png" width="80%" height="80%" align=center />

This is the source code for our HRL-GRG model and the baseline methods we mentioned in the paper.

paper | project webpage

Requirements

Our code is developed and tested under the following dependencies:

Before running the code, please specify the path to the code directory in the config.json that can be found in both the grid_world, the robotic_object_search/House3D and the robotic_object_search/AI2-THOR directory.

Before running the robotic object search code on AI2-THOR, please download our pre-processed data sourced from AI2-THOR and extract at the robotic_object_search/AI2-THOR directory.

Before running the robotic object search code on House3D, please download our pre-processed data sourced from House3D and extract at the robotic_object_search/House3D directory.

Download our pre-trained models and put them in the corresponding code directories for training and/or evaluating our method.

Training

Grid-world domain

To train our model HRL-GRG in the paper, run this command:

# From grid-world/HRL-GRG/
./train.sh

To train other baseline methods mentioned in the paper, run the same command from the corresponding directories.

Robotic Object Search

AI2-THOR

To train our model HRL-GRG in the paper, run this command:

# Specify the parameters in robotic_object_search/AI2-THOR/HRL-GRG/train.sh, 
# and from robotic_object_search/AI2-THOR/HRL-GRG/
./train.sh

or

# From robotic_object_search/AI2-THOR/HRL-GRG/
python train.py \
    --pretrained_model_path=${PATH_TO_PRETRAINED_MODEL} \
    --model_path=${PATH_TO_MODEL} 

where the pretrained_model_path is ../A3C/result_pretrain/model.

House3D

To train our model HRL-GRG in the paper, run this command:

# Specify the parameters in robotic_object_search/House3D/HRL-GRG/train.sh, 
# and from robotic_object_search/House3D/HRL-GRG/
./train.sh

or

# From robotic_object_search/House3D/HRL-GRG/
python train.py \
    --default_scenes=<enviroments_to_train> \
    --default_targets=<target_objects_to_train> \
    --pretrained_model_path=${PATH_TO_PRETRAINED_MODEL} \
    --model_path=${PATH_TO_MODEL} 

where the pretrained_model_path is ../A3C/result_se_for_pretrain/model for the single environment setting, and ../A3C/result_me_for_pretrain/model for the multiple environments setting.

To train other baseline methods mentioned in the paper, run the same command from the corresponding directories.

Evaluation and Results

Grid-world domain

To evaluate our method HRL-GRG on the grid-world domain, and reproduce the results of our method on the unseen grid-world maps for seen goals as follows,

Unseen Envs Seen GoalsSRAS / MSSPL
HRL-GRG0.5728.71 / 9.030.33

run this command:

# From grid_world/HRL-GRG/
CUDA_VISIBLE_DEVICES=-1 python evaluate.py \
  --evaluate_file='../random_method/maps_16X16_v6_valid_seengoals.txt'

To reproduce the results for the unseen goals and the overall goals, specify the evaluate_file as '../random_method/maps_16X16_v6_valid_unseengoals.txt' and '../random_method/maps_16X16_v6_valid_total.txt' respectively.

To evaluate other baseline methods, run the same command from the corresponding directories.

Robotic Object Search

AI2-THOR

To evaluate our method HRL-GRG for the robotic object search task on AI2-THOR, and reproduce the results of our method on the seen environments for seen goals as follows,

Seen Env Seen GoalsSRSPL
HRL-GRG0.740.34

run this command,

# From robotic_object_search/AI2-THOR/HRL-GRG/
CUDA_VISIBLE_DEVICES=-1 python evaluate.py \
  --model_path="result_pretrain/model" \
  --evaluate_file='../random_method/ssso.txt' 

To reproduce the results for the seen environments unseen goals, unseen environments seen goals and unseen environments unseen goals, specify the evaluate_file as '../random_method/ssuo.txt', '../random_method/usso.txt' and '../random_method/usuo.txt' respectively.

To evaluate the corresponding Random method, run the following command with the evaluate_file being specified respectively.

# From robotic_object_search/AI2-THOR/random_method/
CUDA_VISIBLE_DEVICES=-1 python random_walk.py \
  --evaluate_file='../random_method/ssso.txt' 

House3D

To evaluate our method HRL-GRG for the robotic object search task on House3D,

# From robotic_object_search/House3D/HRL-GRG/
CUDA_VISIBLE_DEVICES=-1 python evaluate.py \
  --model_path="result_se_pretrain/model" \
  --evaluate_file='../random_method/1s6t.txt' 

to reproduce the results of our method on the single environment for the seen goals as follows,

Single Env Seen GoalsSRSPL
HRL-GRG0.880.33
# From robotic_object_search/House3D/HRL-GRG/
CUDA_VISIBLE_DEVICES=-1 python evaluate.py \
  --model_path="result_se_pretrain/model" \
  --evaluate_file='../random_method/1s6t_test.txt' 

to reproduce the results of our method on the single environment for the unseen goals as follows,

Single Env Unseen GoalsSRSPL
HRL-GRG0.790.21
# From robotic_object_search/House3D/HRL-GRG/
CUDA_VISIBLE_DEVICES=-1 python evaluate.py \
  --model_path="result_me_pretrain/model" \
  --evaluate_file='../random_method/4s6t.txt' 

to reproduce the results of our method on the multiple environments for the seen environments as follows,

Multiple Envs Seen EnvsSRSPL
HRL-GRG0.760.20
# From robotic_object_search/House3D/HRL-GRG/
CUDA_VISIBLE_DEVICES=-1 python evaluate.py \
  --model_path="result_me_pretrain/model" \
  --evaluate_file='../random_method/4s6t_test.txt' 

to reproduce the results of our method on the multiple environments for the unseen environments as follows,

Multiple Envs Unseen EnvsSRSPL
HRL-GRG0.620.10

To evaluate other baseline methods, run the same command from the corresponding directories.