Home

Awesome

About

Reversi reinforcement learning by AlphaGo Zero methods.

@mokemokechicken's training hisotry is Challenge History.

If you can share your achievements, I would be grateful if you post them to Performance Reports.

Environment

Modules

Reinforcement Learning

This AlphaGo Zero implementation consists of three worker self, opt and eval.

Evaluation

For evaluation, you can play reversi with the BestModel.

Data

If you want to train the model from the beginning, delete the above directories.

How to use

Setup

install libraries

pip install -r requirements.txt

install libraries with Anaconda

cp requirements.txt conda-requirements.txt
conda env create -f environment.yml
source activate reversi-a0
conda install --yes --file conda-requirements.txt

If you want use GPU,

pip install tensorflow-gpu

set environment variables

Create .env file and write this.

KERAS_BACKEND=tensorflow

Windows Setup

This instruction is written by @GCRhoads, Thanks!

Required: 64-bit windows

Procedure verified for Windows 8.1. Not yet tested for other versions.

Note: Windows uses backslashes not forward slashes in path names.

  1. Change the first line (if necessary) of "src\reversi_zero\agent\player.py" to from asyncio.futures import Future

  2. Install the 64-bit version of Python 3.5 (the 32-bit version is not sufficient). You have two options

Note: For some strange reason, both Python 3.5 and Anaconda get installed in a hidden folder. To access them, you first have to go to the Control Panel, select Folder Options, and on the View tab, click on the circle next to "Show hidden files, folders, or drives" in the Advanced settings section. Anaconda gets installed in C:\ProgramData\Anaconda3. The direct download option installs Python in (I believe) C:\Users\<your user name>\AppData\Local\Program\Python.

  1. Install Visual C++ 2015 build tools. You could install the entire 2015 version (not the 2017 version that Microsoft tries to force on you) of Visual Studio but this is a large download and install, most of which you don't need. Download visual C++ build tools. Double-click on the downloaded file to run the installer.

  2. Rewrite all uses of an f-strings. The python source code for this project uses numerous f-strings, a feature new to Python 3.6. Since we need Python 3.5 (required by the windows version of tensorflow), use your editor's search feature to find every occurrence of an f-string and rewrite it using string.format().

  3. Install the libraries From either the Anaconda prompt or from a command window in the top level folder where you put this distribution, enter the following.

pip install -r requirements.txt
  1. Install tensor-flow

If you have a gpu compatible with tensor-flow (see the list on the tensor-flow web site), then your code will execute much faster if you install the gpu version. To install the gpu-version enter the following in either the Anaconda prompt or the command window.

pip3 install -- upgrade tensorflow-gpu

If you do not have a compatible gpu, then you will have to settle for the slow cpu-only version. To install this, enter the following in either the Anaconda prompt or the command window.

pip3 install -- upgrade tensorflow
  1. set environment variables Create a .env file and write the following line in this file.
KERAS_BACKEND=tensorflow

Now you should be good to go.

Strongest Model

Now, "challenge 5 model" and "ch5 config" are strongest in my models. If you want to play with it,

rm -rf data/model/next_generation/
sh ./download_model.sh 5
# run as wxPython GUI
python src/reversi_zero/run.py play_gui -c config/ch5.yml

If you want to use as a NBoard engine(see below "Run as NBoard2.0 Engine"), please use nboard_engine -c config/ch5.yml for the Command.

Past Models

Please remove( or rename) data/model/next_generation/ directory if you want to use "BestModel" at data/model/model_best_*.

Download Trained BestModel

Download trained BestModel(trained by bellow Challenge 1) for example.

sh ./download_best_model.sh

Download Trained the Newest Model

Download trained the newest model(trained by Challenge 2, 3, 4, 5) as BestModel.

sh ./download_model.sh <version>

ex)

sh ./download_model.sh 5

Configuration

'AlphaGo Zero' method and 'AlphaZero' method

I think the main difference between 'AlphaGo Zero' and 'AlphaZero' is whether using eval or not. It is able to change these methods by configuration.

AlphaGo Zero method

AlphaZero method

policy distribution of self-play

In DeepMind's paper, it seems that policy(π) data saved by self-play are distribution in proportion to pow(N, 1/tau). After the middle of the game, the tau becomes 0, so the distribution is one-hot.

PlayDataConfig#save_policy_of_tau_1 = True means that the saved policy's tau is always 1.

other important hyper-parameters (I think)

If you find a good parameter set, please share in the github issues!

PlayDataConfig

PlayConfig, PlayWithHumanConfig

TrainerConfig

Basic Usages

For training model, execute Self-Play, Trainer and Evaluator.

Self-Play

python src/reversi_zero/run.py self

When executed, Self-Play will start using BestModel. If the BestModel does not exist, new random model will be created and become BestModel.

options

Trainer

python src/reversi_zero/run.py opt

When executed, Training will start. A base model will be loaded from latest saved next-generation model. If not existed, BestModel is used. Trained model will be saved every 2000 steps(mini-batch) after epoch.

options

Evaluator

python src/reversi_zero/run.py eval

When executed, Evaluation will start. It evaluates BestModel and the latest next-generation model by playing about 200 games. If next-generation model wins, it becomes BestModel.

options

Play Game

python src/reversi_zero/run.py play_gui

Note: Mac pyenv environment

play_gui uses wxPython. It can not execute if your python environment is built without Framework. Try following pyenv install option.

env PYTHON_CONFIGURE_OPTS="--enable-framework" pyenv install 3.6.3

For Anaconda users:

conda install python.app
pythonw src/reversi_zero/run.py play_gui
<img src="doc/img/play_gui.png" width="50%">

When executed, ordinary reversi board will be displayed and you can play against BestModel. After BestModel moves, numbers are displayed on the board.

Run as NBoard2.0 Engine

NBoard is a very good reversi GUI and has strong reversi engines, which runs on Windows, Mac, and Linux (JRE required).

<img src="doc/img/nboard.png" width="50%">

It can add external engines that implement NBoard Protocol.

How to add this model as an external engine to NBoard

<img src="doc/img/add_to_nboard.png" width="50%">

convenient way to evaluate your model

NBoard cannot play with two different engines (maybe). However, it can select different engines of play-engine and analysis-engine.

So, convenient way to evaluate your model is for example,

I have little confidence about hint protocol as analysis-engine (there is odd behavior), but work in my environment.

Auto Evaluation with other reversi AIs

reversi-arena is a system for evaluating reversi AIs which implement NBoard Protocol. It is useful when playing many games with strong AI like NTest.

View Training Log in TensorBoard

1. install tensorboard

pip install tensorboard

2. launch tensorboard and access by web browser

tensorboard --logdir logs/tensorboard/

And access http://<The Machine IP>:6006/.

<img src="doc/img/tensorboard.png">

Trouble Shooting

If you can not launch tensorboard by error, try to create another new plain project which includes only tensorflow and tensorboard.

And

tensorboard --logdir <PATH TO REVERSI DIR>/logs/tensorboard/

Tips and Memo

GPU Memory

In my environment of GeForce GTX 1080, memory is about 8GB, so sometimes lack of memory happen. Usually the lack of memory cause warnings, not error. If error happens, try to change per_process_gpu_memory_fraction in src/worker/{evaluate.py,optimize.py,self_play.py},

tf_util.set_session_config(per_process_gpu_memory_fraction=0.2)

Less batch_size will reduce memory usage of opt. Try to change TrainerConfig#batch_size in NormalConfig.

Training Speed