Awesome

MediaPipe Unity Plugin

This is a Unity (>= 2021.3) Native Plugin to use MediaPipe (0.10.14).

The goal of this project is to port the MediaPipe API (C++) one by one to C# so that it can be called from Unity.
This approach may sacrifice performance when you need to call multiple APIs in a loop, but it gives you the flexibility to use MediaPipe instead.

With this plugin, you can

• Write MediaPipe code in C#.
• Run MediaPipe's official solution on Unity.
• Run your custom Calculator and CalculatorGraph on Unity.
  • :warning: Depending on the type of input/output, you may need to write C++ code.

:smile_cat: Hello World!

Here is a Hello World! example.
Compare it with the official code!

using Mediapipe;
using UnityEngine;

public sealed class HelloWorld : MonoBehaviour
{
    private const string _ConfigText = @"
input_stream: ""in""
output_stream: ""out""
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""in""
  output_stream: ""out1""
}
node {
  calculator: ""PassThroughCalculator""
  input_stream: ""out1""
  output_stream: ""out""
}
";

    private void Start()
    {
        using var graph = new CalculatorGraph(_ConfigText);
        using var poller = graph.AddOutputStreamPoller<string>("out");
        graph.StartRun();

        for (var i = 0; i < 10; i++)
        {
            graph.AddPacketToInputStream("in", Packet.CreateStringAt("Hello World!", i));
        }

        graph.CloseInputStream("in");
        var packet = new Packet<string>();

        while (poller.Next(packet))
        {
            Debug.Log(packet.Get());
        }
        graph.WaitUntilDone();
    }
}

For more detailed usage, see the API Overview page or the tutorial on the Getting Started page.

:hammer_and_wrench: Installation

This repository does not contain required libraries (e.g. libmediapipe_c.so, Google.Protobuf.dll, etc).
You can download them from the release page instead.

If you want to customize the package or minify the package size, you need to build them by yourself.
For a step-by-step guide, please refer to the Installation Guide on Wiki.
You can also make use of the Package Workflow on Github Actions after forking this repository.

:warning: libraries that can be built differ depending on your environment.

Supported Platforms

:warning: GPU mode is not supported on macOS and Windows.

Supported Solutions

This plugin implements the following MediaPipe Tasks C# APIs.

cf. The official available solutions

Legacy Solutions

You can also use MediaPipe Framework, which allows you to run Legacy Solutions. However, please note that support for these solutions has ended.

:plate_with_cutlery: Try the sample app

Example Solutions

Some solutions (including Legacy solutions) can be tested using the sample app. Please check Assets/MediaPipeUnity/Samples/Scenes to see which solutions have samples.

UnityEditor

Select any scenes under Assets/MediaPipeUnity/Samples/Scenes and play.

Desktop, Android, iOS

Select proper Inference Mode and Asset Loader Type from the Inspector Window.

Preferable Inference Mode

If the native libraries are built for CPU (i.e. --desktop cpu), select CPU for inference mode.
For the libraries distributed on the release page, only the CPU is available for use on Windows and macOS.

Asset Loader Type

The default Asset Loader Type is set to Local, which only works on UnityEditor.
To run it on your devices, switch it to StreamingAssets and copy the required resources under StreamingAssets (if you're using MediaPipeUnityPlugin-all.zip, the StreamingAssets directory already contains them).

See the tutorial for more details.

:book: Wiki

https://github.com/homuler/MediaPipeUnityPlugin/wiki

:scroll: LICENSE

MIT

Note that some files are distributed under other licenses.

• MediaPipe (Apache Licence 2.0)
• emscripten (MIT)
  • third_party/mediapipe_emscripten_patch.diff contains code copied from emscripten
• FontAwesome (LICENSE)
  • Sample scenes use Font Awesome fonts

See also Third Party Notices.md.