Home

Awesome

RunwayML library for Processing

A library to easily use RunwayML with Processing.

Prerequisites

Installation

  1. Download RunwayML.zip
  2. Unzip into Documents > Processing > libraries
  3. Restart Processing (if it was already running)

đź“˝ Watch How to Install and Use the RunwayML Processing Library

Testing installation

Before using the library please make sure RunwayML is started and is running one model.

You can test by opening your browser and navigating to http://localhost:8000/info to view info about your model.

If you see a JSON string with inputs and outputs information use the Basic Example below !

Feel free to test the Basic Example below.

Basic Example

For this example run the im2txt model.

This example will print information from RunwayML (e.g. running im2txt)

// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;

void setup() {
  // setup Runway HTTP connection using default host (localhost) and port (8000)
  runway = new RunwayHTTP(this);
}

void draw() {
  
}

// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
  println(runwayData);
}

// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
  println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
  println(message);
}

Notice the text running displayed in console.

What if we want to display an image from RunwayML ?

Let's stop im2txt and use CygleGAN. Notice on info event details about the model: the output in particular is an image with the key "image". (You can see the information within RunwayML on the top right Network area of the Model).

Images to/from RunwayML are Base64 encoded and the library's ModelUtils helps convert between this format and the familiar PImage format.

With the CycleGAN model running we can tweak the above example to display an image received from RunwayML:

// import Runway library
import com.runwayml.*;
// reference to runway instance
RunwayHTTP runway;
// reference to image received
PImage resultImage;

void setup() {
  // setup Runway HTTP connection using default host (localhost) and port (8000)
  runway = new RunwayHTTP(this);
}

void draw() {
	// if there's an image received:
	if(resultImage != null){
		// display the image
		image(resultImage,0,0);
	}  
}

// this is called when new Runway data is available
void runwayDataEvent(JSONObject runwayData){
  // try to access "image" key and convert the received that to a PImage
  try{
  	String base64Image = runwayData.getString("image");
  	resultImage = ModelUtils.fromBase64(base64Image);
  }catch(Exception e){
  	// print an error message if the above fails
  	println("error parsing received data: " + runwayData);
  	e.printStackTrace();
  }
}

// this is called each time Processing connects to Runway
// Runway sends information about the current model
public void runwayInfoEvent(JSONObject info){
  println(info);
}
// if anything goes wrong
public void runwayErrorEvent(String message){
  println(message);
}

Existing Examples

You can access the following examples via Processing > Examples > Contributed Libraries > RunwayML

Extra platforms

Additionally you can use RunwayML on a machine on the local network and connect from another device capable of running Processing.

Raspberry PI

As long as you have a Raspberry PI running Processing, all you need to do is specify the IP (and optionally port) of the computer running the RunwayML app: that's it! (e.g. if your computer's LAN IP is 192.168.0.12 you would initialize Runway like so: runway = new RunwayHTTP(this,"192.168.0.12",8000);)

To setup Processing on a Raspberry Pi please follow the official Get Started tutorial

Raspberry Pi library example preview

Android

  1. please install Android Mode for Processing (if it's not already installed on your system).
  2. allow INTERNET permissions in Processing > Android > Sketch Permissions Android library example preview
  3. run the code (e.g. attnGANAndroid)

The one major caveat at the moment is the Base64 <-> PImage conversion is handled by a separate class: ModelUtilsAndroid.

This means on Android

PImage result = ModelUtils.fromBase64(base64ImageString);

becomes:

PImage result = ModelUtilsAndroid.fromBase64(base64ImageString);

Android library example preview

(Additionally, it's even possible to write RunwayML Processing sketchs on the Android device using APDE)

Dependencies

This Processing Library manages the OSC connection to Runway relying on the oscP5 library.

Please install oscP5 before using the OSC connection with this library via Sketch > Import Library... > Add Library... > Contribution Manager Filter > oscP5

Contributing

This is still a work in progress. Contributions are welcomed!

Credits

Special thanks for mentoring and support from Cris Valenzuela, Anastasis Germanidis and Daniel Shiffman

Additional thanks to Jen Sykes for updating the DenseCap and GPT2 examples

Main library development by George Profenza

Library examples are based partially on Runway Processing Examples by: