Home

Awesome

XVERSE 3D-GS UE Plugin

<a href="./LICENSE"> <img alt="License" src="https://img.shields.io/badge/License-Apache_2.0-blue.svg"></a>

English | 中文

<img src="Media/image/XVERSE.jpg" width="600" />

Introduction

XVERSE 3D Gaussian Splatting UE plugin（XVERSE 3D-GS UE Plugin）is an Unreal Engine 5 (UE5) based plugin developed by XVERSE Technology Inc. (Zhenshen, China), aiming to provide real-time visualization, management, editing, and scalable hybrid rendering of Gaussian Splatting model, which is a recently arised technique for reconstructing 3D scenes from multiple photos, more details can refer to here.

Currently, our plugin is implemented based on UE5 Niagara, fully exploiting the advantage of the features provided by UE5 to realize efficient rendering and managing for Guassian Splatting models.

Since we are at early access, current accessible features are summarized below:

We will actively release new features in this repo, please stay tuned. Some future updates will contain:

Getting Started

Video Tutorial

Youtube（English）：

B站（中文）：

System Requirements

Plugin download

  1. Open github Release page
<img src="Plugins/Media/a1.jpg" width="500" />
  1. Download the zip corresponding to the UE version you are using
<img src="Plugins/Media/a2.jpg" width="500" />
  1. Unzip and move folder XV3dGS to folder Plugins
<img src="Plugins/Media/a3.jpg" width="500" />

Demo Data Download Links:

We have provided two sets of test data available for download at:

Included in these downloads are demo_fox_1080p.mp4 and demo_office_1080p.mp4, serving as two sample MP4 files for testing XV3DTools' local training capabilities.

demo_fox_gs.ply and demo_office_gs.ply are 3DGS models trained by XV3DTools, which can be imported into Unreal Engine (UE) using this plugin for real-time rendering.

<img src="Media/image/t1.jpg" width="500" />

Open demo project

This repo contains a demo project with an example scene and level

  1. download using git clone
git clone https://github.com/xverse-engine/XV3DGS-UEPlugin.git
  1. Open XV3DGS.uproject to start UE
  2. Open demo level
<img src="Media/image/S3.jpg" width="500" />

Import your guassian splatting model

After v1.1.3, this plugin supports dragging multiple ply files directly into the Content Browser and importing them.

Click on the 'Load Assets' button on the navigation bar or navigate to Window/XV3dgs. <img src="Media/image/t4.jpg" width="500" />

This action will bring up an integrated tool page by default set to load the PLY page. Click on the 'Load PLY' button.

<img src="Media/image/t3.jpg" width="500" />

This will trigger a resource selection window to appear.

<img src="Media/image/a2.jpg" width="500" />

Choose your Gaussian model file (in .ply format), then it will automaticly generate LOD and loaded into your Content Browser as a blueprint.

<img src="Media/image/a3.jpg" width="500" />

Drag the blueprint into Level Viewport and you can see your Gaussian scene.

<img src="Media/image/a4.jpg" width="500" />

Local Training on Windows Platform

We develop the XV3DTools(V1.1.1) to train MP4 videos into Gaussian Splatting ply.

Please Note: Installation of CUDA 11 or higher is required. CUDA Download Link

XV3DTools Usage Steps:

  1. Download XV3DTools(V1.1.1) tool from the cloud drive to your local machine:
<img src="Media/image/t10.jpg" width="500" />
  1. Click on the icon to open the interface and switch to the 'training' page.
<img src="Media/image/t4.jpg" width="500" />

  1. Click button to set the parameter "XV3DTools Folder" to the local path where XV3DTools tool is located. The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  2. Click button to select the MP4 file for processing and set it as the parameter "Input MP4 Path" (requires video frame rate > 20). The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  3. Click button to choose the output folder and set it as the parameter "Output Folder". The path can only be composed of English words. no chinese words and no special words such as "# $ %".

  4. click Start Training button.

<img src="Media/image/t6.jpg" width="500" />
  1. Before training, you can open "Advanced Options" and click "Refresh Estimated Time" button to estimate the training time. The parameter "Training iteration per image" can be adjusted to control the number of iterations per image.
<img src="Media/image/t7.jpg" width="500" />
  1. Training takes time. Once completed, locate the trained ply file in the output folder.
<img src="Media/image/t5.jpg" width="500" />

Training Speed Reference:

Packaging UE level for Windows Using Our Plugins

You can refer to the following tutorial to package your own UE levels into exe files that can be run on windows System.

packing tutorial

We provide a packaged Windows executable demo, you can download pack_win_exe.zip from release, simply double-click the exe file to start it.

Feature Introduction

RTS (Rotation, Translation, Scaling) Transform

Select the 3D-GS asset and drag it into the scene. then you can Transform it.

<img src="Media/image/FA0.jpg" width="500" />

You can also select a 3D-GS actor in the level viewport and transform it using GIZMO.

<img src="Media/image/a6.gif" width="500" />

Model Clipping by Gaussian model

  1. Click to select your Gaussian model blueprint, then click to select the "Clip Volume Component", and open the "Visible" box in the "Clip Volume Component".
<img src="Media/image/FA1.jpg" width="500" />
  1. With the "Clip Volume Component" selected, you can see a crop volume in the window. You can modify the rotation, translation, and scale of the volume by adjusting the Transform attribute. Alternatively, use the keyboard shortcuts WER to visually adjust the GIZMO for rotation, translation, and scale.
<img src="Media/image/FA2.jpg" width="500" />
  1. Keep the "Clip Volume Component" selected, and choose different cropping options in the Crop Type. The default state is no cropping, but you can change it to crop the inside or outside of the volume. The example below shows cropping the outside of the volume.
<img src="Media/image/FA3.jpg" width="500" />
  1. Click the "Crop GSPoints" button to clip the specified portion and obtain a clean Niagara.
<img src="Media/image/FA4.jpg" width="500" />
  1. This clean Niagara can be used for VFX. For more details, please refer to our VFX tutorial (Coming soon).
<img src="Media/image/FA5.jpg" width="500" />

Dynamic illumination

Dynamic lighting is not default opened.

you can enable it by turning on the withRelighting button.

<img src="Media/image/FC2.jpg" width="500" />

Supports all the UE light types

<img src="Media/image/FC1.jpg" width="500" />

The light source can be moved at will

<img src="Media/image/FC.gif" width="500" />

Change color

Modify the AlbedoTint parameter to change the overall color.

<img src="Media/image/FC3.jpg" width="500" /> <img src="Media/image/FB.gif" width="500" />

LOD

For imported point clouds, LOD is automatically calculated, with the default being to explicitly display the maximum LOD within the limit of 2 million points for Niagara. The parameter "VisWhichLOD" can be used to force the use of a specific LOD.

<img src="Media/image/FD1.jpg" width="500" />

Dynamic switching of LOD based on the view camera will be supported in future updates.

Release Note

v1.1.3

v1.1.2

v1.1.1

v1.1.0

v1.0.1

v1.0.0

Contributors

<a href="https://github.com/xverse-engine/XV3DGS-UEPlugin/graphs/contributors"> <img src="https://contrib.rocks/image?repo=xverse-engine/XV3DGS-UEPlugin" /> </a>