Home

Awesome

VALAN: Vision and Language Agent Navigation

VALAN, short for Vision and Language Agent Navigation is a lightweight and scalable software framework for deep reinforcement learning based on the SEED RL architecture. The framework facilitates the development and evaluation of embodied agents for solving grounded language understanding tasks, such as Vision-and-Language Navigation and Vision-and-Dialog Navigation, in photo-realistic environments, such as Matterport3D and StreetLearn. Such tasks require agents to interpret natural language instructions/dialog to navigate in photo-realistic environments in order to achieve prescribed navigation goals. We have added a minimal set of abstractions on top of SEED RL allowing us to generalize the architecture to solve a variety of other RL problems.

This package contains the implementations of the following problems:

See Mehta et al. for details about our implementation for Touchdown and the data supporting it.

For a detailed description of the architecture please read Lansing et al. Please cite the paper if you use the code from this repository in your work.

Bibtex

@article{lansing2019valan,
    title={VALAN: Vision and Language Agent Navigation},
    author={Larry Lansing and Vihan Jain and Harsh Mehta and Haoshuo Huang and Eugene Ie},
    year={2019},
    eprint={1912.03241},
    archivePrefix={arXiv},
    primaryClass={cs.LG}
}

Prerequisites

Before getting started, we need two main packages to run VALAN: docker and git.

Usage

Running on local machine

Running on distributed environment (e.g., GCP)

We provide the tool and a concrete example to run VALAN on GCP with distributed learning. Note that training the AI Platform requires signing up for a GCP account and will will incurr charges for using the compute resources. See: https://cloud.google.com/ai-platform

Disclaimer

This is not an official Google product.