Home

Awesome

<img src="doc/xworld_logo.png" alt="XWorld" width="834" height="192">

<img src="doc/simple_race_2.png" width="150" height="250"> <img src="doc/atari.png" width="191" height="250"> <img src="doc/xworld2d.png" width="250" height="250"> <img src="doc/xworld3d.png" width="250" height="250">

This repository contains a collection of simulators for Reinforcement Learning research.

DifficultyNameDescriptionThread-compatible?*Optional?PolicyTeacher?
EasySimpleGameA simple 1D array-walking game.YesNoDiscreteNo
Easy, MediumSimpleRaceA simple synthetic car racing game.YesNoDiscreteNo
Medium, HardAtariWrappers for the Arcade Learning Environment (ALE) environment. For stability, we use a fork version.YesYesDiscreteNo
Medium, HardXWorld2DA 2D world for an agent to learn vision and language abilities.NoNoDiscrete<br>ContinuousYes
HardXWorld3DA 3D world for an agent to learn vision and language abilities.NoYesDiscrete<br>ContinuousYes

(*If yes, then multithreading can be used; otherwise multiprocessing is needed.)

Architecture

XWorld features a teacher infrastructure implemented as a scheduler of multiple Finite State Machines (FSMs). The idea is that given the environment, the teacher can propose a task sampled (by some heuristics) from a task set. Each task - formulated as an FSM - has several stages, and the teacher does different things in different stages. The transition from one stage to another is determined by the envionment state, e.g., whehter the agent is idle or whether it has achieved the goal. Each stage returns several things including the next stage and the teacher's action. Currently, we define language (strings) as the teacher's sole action. However, the teacher is able to change the environment (e.g., adding/deleting objects, changing the map size, etc.) within each stage.

<img src="doc/xworld_arch.png">

The above figure illustrates the architecture. The motivation is to let the users flexibly write simple Python scripts to configure the environment maps and tasks.

Currently, the teacher is only incorporated into XWorld2D and XWorld3D.

Requirements

Dependencies

The following softwares must be installed before building XWorld.

Boost, Glog, GFlags, GTest, and Python

In Ubuntu 14.04 and 16.04, you can do

sudo apt-get install libboost-all-dev libgflags-dev libgoogle-glog-dev libgtest-dev python-dev

Build

First get this git repository

git clone https://github.com/PaddlePaddle/XWorld

Suppose the directory is xworld_path, then do

cd <xworld_path>
mkdir -p build
cd build
cmake [<optional parameters>] ..

For example,

cd ~/XWorld; mkdir build; cd build
cmake ..

Finally, in the build directory do

make
make test

By default, XWorld only builds the first three games: SimpleGame, SimpleRace, and XWorld2D.

Optionally, you can install Atari by:

cmake -DWITH_ATARI=ON ..

which will automatically download and build Atari.

You can also install XWorld3D by:

cmake -DWITH_XWORLD3D=ON ..

Usage

Python interface

We provide a set of simple Python APIs for interacting with the simulators. After building XWorld, you need to export the path of the python module:

export PYTHONPATH=<xworld_path>/python:$PYTHONPATH

You can add the above line to ~/.bashrc to avoid doing the export in the future.

To get started, several examples of the simulator Python APIs can be found in

<xworld_path>/python/examples

C++ interface

Alternatively, several C++ examples (run the .sh scripts inside) can be found in

<xworld_path>/examples

These examples use the individual class constructors to create games. However, we also provide a unified simulator interface for creating games in a more convenient way (like in Python). A demo of the unified C++ simulator interface for multi-process simulation can be found in

<xworld_path>/examples/demo_interface.cpp

Generally, C++ APIs are more flexible but expose more details compared to the Python APIs.

Flags of a game

Option flags are passed into a game via different ways for the two interfaces:

For descriptions of the flags of a game, please take a look at the README file under the game directory.

Citations

If you use XWorld2D for research, consider citing

If you use XWorld3D for research, consider citing

If you use our wrappers of the third-party simulators, please follow their original guide for citation.

License

This repository has the Apache2.0 license, except that the third-party simulator ALE has its own license.