Home

Awesome

Reservoir Spatio Temporal Importance Resampling (ReSTIR)

Outlines

Introduction

This is a team project implementing ReSTIR based on the research paper with the same title, published by NVIDIA in 2020. Briefly, the purpose of ReSTIR is to help rendering scenes with a lot of lights but make it much less noisy than the basic path tracing algorithm. This is a result of continuously finding and updating the most ideal light for a pixel based on its surrounding neighbor pixels and its light in previous frames.

ReSTIR explained

Please let us know if you find any errors in our understandings of the paper.

Overview

For each pixel:

  1. Select 1 light from randomly 32 chosen lights
  2. Shoot a shadow ray from the light to the pixel. If it is obscured, discard the chosen light.
  3. Compare the light used in the last iteration to the light from step 2 and choose one.
  4. Compare the lights from random adjacent pixels to light from step 3 and choose one.
  5. Shoot a shadow ray to light from step 4 and shade the pixel.

This uses a combination of Weighted Reservoir Sampling and Resampled Importance Sampling to select and compare lights.

Details

The way to execute each of the step is elaborated in algorithm 5, 4, 3, 2 and 1 in the paper, so we would not dig into those details here. However, there were some variables or details that made us scratch our head for a bit, so we would explain those as below.

Reservoir

We uses a data structure called reservoir for each pixel that holds the current light and sum of all weights seen for the light. A reservoir holds four values: weight sum of all light candidates seen so far, number of light candidates seen so far, index of the chosen light, and the adjusted weight based on weight sum and the number of light candidates seen.

Weight calculation per light candidate

Each light candidate has a weight corresponding to their chance of being chosen as the current sampled light per pixel.

Temporal reuse

When doing temporal reuse, the paper advises to clamp the number of candidates M contribution to the pixel (otherwise, this can go unbounded.) We clamp the previous frame's M to at most 20x of the current frame's reservoir's M. Without this, objects in the scene might become black, a problem we encountered.

Results

Final results

Forest scene (80 lights)

ReSTIR outputs a more converged render at the same iteration as the base method of randomly sample one light. This is clearly seen when zooming into many parts of the renders above.

ReSTIR (after 14 iterations)One random light (after 14 iterations)

Purple bedroom scene (15 lights)

ReSTIR (after 44 iterations)One random light (after 44 iterations)

Overall, we find that the ReSTIR image is slightly more converged than using the basic method of randomly selecting one light in the scene. Examples of areas that are much more converged can be seen below, including pillow, blanket, chair and table. We expect that the diffference of convergence between these two rendering methods would be larger if there are many more lights in the scene.

ReSTIR (44th frame)One random light (44th frame)

However, ReSTIR spatial reuse also makes some part of the scene looks somewhat fuzzier than the basic method.

ReSTIR (44th frame)One random light (44th frame)

Bistro scene

Intermediate results

Candidates generation results

With only ReSTIR candidates generation, we eliminate a lot of shadows that are not visible in the converged images. The method helps to brighten up the scene more quickly.

Ground truthCandidates generation (first frame)One random light (first frame)

Temporal results

Temporal results also help to brighten our renders more quickly than the base method of randomly sample one light. As mentioned, the paper advises having a clamped number of candidates seen for previous reservoir to max of k * currentReservoirWeight. Here are the effects of changing parameter k.

k = 5k = 20 (recommended)k = 50

Even though the scene is brighted up as k increases, especially at the top of the sofa or at edge of sofa seats, there is also more noise in the renders.

Spatial results

As expected, having more candidates help renders converge more quickly. At the 61th frame, we have these renders of our forest scene.

1 sampled neighbors (radius 30)30 sampled neighbors (radius 30)

Global illumination

We have incorporated global illumination into Restir algorithm. Our approach basically adds indirect lighting to the current spatiotemporal output. We shoot 2 types of rays: Shadow ray and Indirect bounce ray. When we shoot our indirect ray and it hits a surface, we perform a simple lambertian shading at that point. To save cost, we are currently only shooting one shadow ray from each hit.

In initLightPlusTemporal.rt.hlsl, We have a flag gDoIndirectGI which allows us to toggle Indirect illumination. We first sample a random direction for our diffuse interreflected ray by using either cosine hemisphere sampling or uniform hemishphere sampling. We calculate the lambertian term for this randomly selected direction and then shoot out indirect ray to calculate the bounce color. We finally do a monte carlo intergration of the rendering equation for just the indirect component by multiplying the bounce color and lambertian term to the albedo and dividing this by the pdf based on the sampling technique used. We have added a new buffer to hold the output of indirect illumination called "gIndirectOutput" to store the output from indirect illumination and pass it on to the updateReservoirPlusShade pass.

Output:

Global Illum (Pink Room Scene gif )

Converged Images:

With Global IlluminationWithout Global Illumination
Global Illum (Pink Room Scene GI )Global Illum (Pink Room Scene NO GI )
Global Illum (Purple Room Scene GI )Global Illum (Purple Room Scene NO GI )

Global illumination helps with lighting up some scenes and creating color bleeding effect, as seen with the red carpet reflecting light at the bottom of the white sofa in one of the scens above. It can help certain scenes look significantly better as the one below. The Bistro scene lights up very well upon adding global illumination since we are also casting indirect rays in order to accumulate color. Using Direct Lighting alone leaves the scene pretty dark since it is a very huge environment and not all areas receive direct lighting.

With Global IlluminationWithout Global Illumination

However, as expected, the effect of global illumination is more apparent for scenes where objects are close to each other as above and less apparent for scenes where objects are further apart. In the scene below, the brightening and color-bleeding effects are barely or not noticeable at all.

With Global IlluminationWithout Global Illumination
Global Illum (Forest GI )Global Illum (Forest NO GI )

The only apparent difference is a leaf of one pine tree in the scene is brighter

With Global IlluminationWithout Global Illumination
Global Illum (Forest GI )Global Illum (Forest NO GI )

Runtime analysis

The below are results from our forest scene.

As expected ReSTIR has a lower FPS compared with the method of only sampling one random light. This might be due to many buffers used for ReSTIR, so the time accumulated by passing in the buffer data as well as reading from and writing into buffers increase drastically. There are also a lot of branching in various shaders for ReSTIR, which can significantly slow down the method and result in low FPS. Another factor is that we are using more passes than the other method, which also lead to the lower FPS in methods involved with ReSTIR in the graph above.

Due to the inefficiencies mentioned above, the time for ReSTIR to converge are also high. However, there might be a drastic difference when there are a lot more lights in the scene (thousands or millions), which are not displayed here, that show ReSTIR with a better convergence time.

Potential improvements

Complex light handling

Currently, we are only handling static point lights. Having dynamic lights and area or mesh lights might show off more benefits of ReSTIR.

Candidate generation

The paper and presentations also suggests ways to better sample light candidates for the first step - candidates generation per reservoir - by storing emissive triangles based on their power. We have yet to incorporated this into our implementation (since we are not dealing with complex lights here)

Pass reduction

We may be able to reduce at least one pass by refactoring and moving our implementation of global illumination to the last pass, and by using the buffers in a better way as well.

Build and run

Credits and resources