Home

Awesome

<img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/159554721-a8db2b64-ee6e-416f-8008-aa08c077b177.png">

This code sample is to show you how to use the new Neural Network Inference (NNI) Plugin in Unreal Engine 5 which implements ONNX Runtime to allow you to add Machine Learning (ML) Models in your projects. ONNX Runtime is a library to optimize and accelerate machine learning inferencing.

<img width="600" alt="image" src="https://user-images.githubusercontent.com/46505951/159556124-9e714170-c4c1-40e2-ac85-20e214912eb0.png">

We are using open source models from the ONNX model zoo to apply a style transform to the scene during game play. Check out the before and after pictures below to see how one of the models is able to stylize the scene.

Before: <img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/162550184-0ed88615-2dae-4b9f-a429-06ce668169f5.png">

After: <img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/162550233-018c0959-febd-4909-bc2e-0ffb8dd6368f.png">

Statement of Purpose

This repository aims to grow the understanding of using ML models through the NNI Plugin in Unreal Engine 5 by providing an example of implementation and references to support the Microsoft Build conference in 2022. It is not intended to be a released product. Therefore, this repository is not for discussing ONNX Runtime, the NNI plugin or requesting new features.

Prerequisites

How to run this project

Options 1: Run with Epic Games Launcher

NOTE: If you receieve the below message click Fix Now and it will recompile the project.

<img width="500" alt="image" src="https://user-images.githubusercontent.com/46505951/164088506-35bb01f3-8ea8-4f05-a50d-7bfdc95a54d4.png"> <img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/163624383-b848e76c-9c7f-4e1c-b102-5d1bf1f93e75.png"> <img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/163624882-bcff90cf-b2d6-424f-90eb-f1127f82d4da.png">

Option 2: Open and run in Visual Studio.

<img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/163624882-bcff90cf-b2d6-424f-90eb-f1127f82d4da.png">

How to select a different model

There are 5 differnet models included from the ONNX Model Zoo. You can find them under the Content > Models folder in the UE Content Browser. To add your own models, drag and drop them into the Models folder in the Content Browser.

<img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/163625334-0cbd8360-88b3-4b8d-91e3-67af120aabeb.png">

To select one of the other preloaded models or one that you added yourself.

<img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/163626096-9500b5d1-3862-4c33-a545-87e09f4157db.png"> <img width="1000" alt="image" src="https://user-images.githubusercontent.com/46505951/163626560-0ba69e90-0f98-40ea-9339-d017e3056b80.png">

Resources

Grab open source models to add to your project:

Connect with ONNX Runtime (ORT) on Social!