Home

Awesome

TensorKart

self-driving MarioKart with TensorFlow

Driving a new (untrained) section of the Royal Raceway:

RoyalRaceway.gif

Driving Luigi Raceway:

LuigiRacewayVideo

The model was trained with:

With even a small training set the model is sometimes able to generalize to a new track (Royal Raceway seen above).

Dependencies

Recording Samples

  1. Start your emulator program (mupen64plus) and run Mario Kart 64
  2. Make sure you have a joystick connected and that mupen64plus is using the sdl input plugin
  3. Run record.py
  4. Make sure the graph responds to joystick input.
  5. Position the emulator window so that the image is captured by the program (top left corner)
  6. Press record and play through a level. You can trim some images off the front and back of the data you collect afterwards (by removing lines in data.csv).

record

Notes

Viewing Samples

Run python utils.py viewer samples/luigi_raceway to view the samples

Preparing Training Data

Run python utils.py prepare samples/* with an array of sample directories to build an X and y matrix for training. (zsh will expand samples/* to all the directories. Passing a glob directly also works)

X is a 3-Dimensional array of images

y is the expected joystick ouput as an array:

  [0] joystick x axis
  [1] joystick y axis
  [2] button a
  [3] button b
  [4] button rb

Training

The train.py program will train a model using Google's TensorFlow framework and cuDNN for GPU acceleration. Training can take a while (~1 hour) depending on how much data you are training with and your system specs. The program will save the best model from all epochs of training to disk when it is done.

Play

The play.py program will use the gym-mupen64plus environment to execute the trained agent against the MarioKart environment. The environment will provide the screenshots of the emulator. These images will be sent to the model to acquire the joystick command to send. The AI joystick commands can be overridden by holding the 'LB' button on the controller.

Future Work / Ideas:

Related Projects:

Xbox Game AI - Uses PYXInput for direct control of any Xbox/PC game.

SerpentAI - Game Agent Framework to create AIs for any game.

Donkey Gym - OpenAI Gym Environments for self-driving "Donkey Car".

AirSim - An Unreal Engine simulator for autonoumous vehicles.

Special Thanks To

Contributing

Open a PR! I promise I am friendly :)