Awesome

Gauzilla

A 3D Gaussian Splatting (3DGS) renderer written in Rust for platform-agnostic WebAssembly (WASM) with lock-free multithreading.

• Uses WebGL and CPU splat sorting (based on splat) for high compatibility among web browsers
• Circumvents WASM's limitations in multithreading via the use of the lock-free bus mechanism
• Uses rfd to securely load a .ply or .splat file stored locally on the host machine
• Loads a .splat file asynchronously from a URL (CDN) without having to use async code in Rust

https://github.com/BladeTransformerLLC/gauzilla/assets/76802410/7221c93d-34ff-4c3f-8c75-b5cdeb83966b

Web Demo

See the Gauzilla Web Examples in this repository.

Introduction

The innovative rendering technique known as 3D Gaussian Splatting (3DGS) [1] represents a Machine Learning-oriented approach to 3D rendering, specifically designed for Novel View Synthesis (NVS). It facilitates the real-time photorealistic rendering of scenes reconstructed from images and videos captured using conventional smartphone cameras. Since its release in 2023 there has been a Cambrian explosion of 3DGS applications and extensions, such as 4DGS [2], D3GA [3], SLAM [4], SC-GS [5], GPS [6], GHA [7], etc (see a survey [9]).

One of the original ideas of splatting Gaussian ellipsoids onto the 2D screen as ellipses for rendering can be traced back in 2002 [8] and 3DGS uses almost the same approach for its efficient forward rendering on the GPU.

Briefly, a 3D elliptical Gaussian centered at a point p with a covariance matirx V is defined as:

The key to splatting it onto the 2D screen is to compute the 3x3 covariance matrix in ray coordinates:

where W is the view transformation (camera) matrix, V_k = R * S * S^T * R^T (covariance matrix factorized with a rotation matrix R and a scaling matrix S, both of which can be computed, respectively, from quaternions and scalers stored in the PLY file), and

is a Jacobian or the partial derivative of the camera-ray mapping at the point t_k in camera space (i.e., affine approxmation of the perspective projection).

Once the covariance matrix in ray coordinates is obtained via the simple matrix multiplications above, one can compute from the top-left 2x2 component of the matrix the major and minor axes representing the size and orientation of Gaussian ellipses, which can subsequently be rendered on the GPU efficiently using Geometry Instancing.

How to Build & Use

1. Install the nightly version of Rust: rustup toolchain install nightly
2. Install wasm-pack and sfz
3. Run ./build.sh sfz and open the locally-served URL in a web browser
4. Open a PLY file formatted for 3DGS (eg. download the official pre-trained models) or a .splat file (use this script to convert from PLY)

Orbit Camera Controls:

Left mouse button   - Rotate view around target
Middle mouse button - Zoom in/out
Right mouse button  - Move left/right/up/down

Fly Camera Controls:

Left mouse button   - Change view direction (free-look)
Middle mouse button - Move forward/backward
Right mouse button  - Move left/right/up/down

How to Deploy on Web

1. (Optional) Enable async_splat_stream feature in Cargo.toml
2. Run ./build.sh
3. Enable cross-origin isolation on the server (cf. Vercel deployment configuration or coi-serviceworker)

ToDo

• Optimize Scene::sort() and Scene::generate_texture() (eg. parallelize using wasm-bindgen-rayon)
• Implement asynch progressive splat loading/rendering for web hosting
• Allow camera controls with keyboard
• Write a WebGPU render path (cf. splatter)

References

1. Kerbl et al., "3D Gaussian Splatting for Real-Time Radiance Field Rendering," 2023.
2. Wu et al., "4D Gaussian Splatting for Real-Time Dynamic Scene Rendering," 2023.
3. Zielonka et al., "D3GA - Drivable 3D Gaussian Avatars," 2023.
4. Matsuki et al., "Gaussian Splatting SLAM," 2023.
5. Huang et al., "SC-GS: Sparse-Controlled Gaussian Splatting for Editable Dynamic Scenes," 2023.
6. Zheng et al., "GPS-Gaussian: Generalizable Pixel-wise 3D Gaussian Splatting for Real-time Human Novel View Synthesis," 2023.
7. Xu et al., "Gaussian Head Avatar: Ultra High-fidelity Head Avatar via Dynamic Gaussians," 2022.
8. Zwicker et al., "EWA Splatting," 2002.
9. Chen & Wang, "A Survey on 3D Gaussian Splatting," 2024.