Home

Awesome

Ngx - Neural network based visual generator and mixer

Ngx is an attempt at utilizing a neural network for VJing. It implements pix2pix (image-to-image translation with cGAN) as an ad-hoc next-frame prediction model that is trained with pairs of consecutive frames extracted from a video clip, so that it can generate an image sequence for an infinite duration just by repeatedly feeding frames back. It also has functionality for mixing (crossfading) two pix2pix models that gives unexpected variation and transition to generated video.

gif gif

(The gif on the right is from a tweet by @chaosgroove.)

Vimeo - Ngx demonstration

Pre-built executables

You can download a pre-built executable from Releases page. It contains pre-trained pix2pix models with Beeple's VJ clips.

Model NameOriginal clipVimeo link
Beeple 1FIBER OPTICALhttps://vimeo.com/238083470
Beeple 2redgate.v1https://vimeo.com/106577855
Beeple 3p-crawlhttps://vimeo.com/116330714
Beeple 4exhausthttps://vimeo.com/95513505
Beeple 5quicksilverhttps://vimeo.com/153493904
Beeple 6MOONVIRUShttps://vimeo.com/155970396
Beeple 7TENDRILhttps://vimeo.com/158477477
Beeple 8shifting painshttps://vimeo.com/48818536

The original video clips are licensed under a Creative Commons Attribution license (CC-BY 3.0). These pre-trained models are also attributed to the original author.

How to control

Ngx can be controlled with the on-screen controller or a MIDI controller.

NameDescriptionMIDI CC
MixCrossfading between model 1 and 277
Noise 1Noise injection (low density)78
Noise 2Noise injection (medium density)79
Noise 3Noise injection (high density)80
NameDescriptionMIDI CC
FeedbackFeedback rate81, 82
FColorFalse color effect83
Hue ShiftAmount of hue shift55, 56
InvertColor inversion84
NameMIDI notes
Model Select 141, 42, 43, 44, 57, 58, 59, 60
Model Select 273, 74, 75, 76, 89, 90, 91, 92

The default MIDI mapping is optimized for Novation Launch Control XL: You can change model with the track buttons and control parameters with the track faders. The 7th and 8th panning knobs are used to tweak the hue shift values.

How to open the project in Unity

This repository uses Git submodules to manage dependent packages. To check these submodules out, execute git submodule init && git submodule update from the command line, or specify --recursive option when initially cloning.

After cloning the repository, .pict (pix2pix weight data) files should be manuyally downloaded and copied into the Streaming Assets directory because these files are excluded from the repository to save bandwidth and storage usage. Please download the pict file package and extract it into Assets/StreamingAssets.

How to train pix2pix models

Prerequisites: Basic experience on a machine learning framework. It's recommended to have a look into the pix2pix tutorial from Machine Learning for Artists if you haven't tried pix2pix or any similar GAN model.

You can train your own pix2pix model using pix2pix-tensorflow. I personally recommend using Google Colaboratory for training. It's easy to set up, handy to use and cost effective (it's free!).

The followings are Colaboratory notebooks I used to train the default models. These notebooks are written to use Google Drive as a dataset storage.

Frequently asked questions

What are the hardware recommendations?

I used a Windows system with GeForce GTX 1070 for a live performance. It runs at around 21 fps; this might be the minimum practical spec for the system.