Home

Awesome

NVIDIA REAL-TIME DENOISERS v4.11.2 (NRD)

Build NRD SDK

Title

For quick starting see NRD sample project.

OVERVIEW

NVIDIA Real-Time Denoisers (NRD) is a spatio-temporal API agnostic denoising library. The library has been designed to work with low rpp (ray per pixel) signals. NRD is a fast solution that slightly depends on input signals and environment conditions.

NRD includes the following denoisers:

Performance on RTX 4080 @ 1440p (native resolution, default denoiser settings, NormalEncoding::R10_G10_B10_A2_UNORM):

Supported signal types:

For diffuse and specular signals de-modulated irradiance (i.e. irradiance with "removed" materials) can be used instead of radiance (see "Recommendations and Best Practices" section).

NRD is distributed as a source as well with a “ready-to-use” library (if used in a precompiled form). It can be integrated into any DX12, VULKAN or DX11 engine using two variants:

  1. Native implementation of the NRD API using engine capabilities
  2. Integration via an abstraction layer. In this case, the engine should expose native Graphics API pointers for certain types of objects. The integration layer, provided as a part of SDK, can be used to simplify this kind of integration.

HOW TO BUILD?

CMake options:

NRD_NORMAL_ENCODING and NRD_ROUGHNESS_ENCODING can be defined only once during project deployment. These settings are dumped in NRDEncoding.hlsli file, which needs to be included on the application side prior NRD.hlsli inclusion to deliver encoding settings matching NRD settings. LibraryDesc includes encoding settings too. It can be used to verify that the library meets the application expectations.

Tested platforms:

OSArchitecturesCompilers
WindowsAMD64MSVC, Clang
LinuxAMD64, ARM64GCC, Clang

SDK packaging:

HOW TO UPDATE?

HOW TO REPORT ISSUES?

NRD sample has TESTS section in the bottom of the UI, a new test can be added if needed. The following procedure is recommended:

Additionally, for any information, suggestions or general requests please feel free to contact us at NRD-SDK-Support@nvidia.com

API

Terminology:

Flow:

  1. GetLibraryDesc - contains general NRD library information (supported denoisers, SPIRV binding offsets). This call can be skipped if this information is known in advance (for example, is diffuse denoiser available?), but it can’t be skipped if SPIRV binding offsets are needed for VULKAN
  2. CreateInstance - creates an instance for requested denoisers
  3. GetInstanceDesc - returns descriptions for pipelines, samplers, texture pools, constant buffer and descriptor set. All this stuff is needed during the initialization step
  4. SetCommonSettings - sets common (shared) per frame parameters
  5. SetDenoiserSettings - can be called to change parameters dynamically before applying the denoiser on each new frame / denoiser call
  6. GetComputeDispatches - returns per-dispatch data for the list of denoisers (bound subresources with required state, constant buffer data). Returned memory is owned by the instance and gets overwritten by the next GetComputeDispatches call
  7. DestroyInstance - destroys an instance

NRD doesn't make any graphics API calls. The application is supposed to invoke a set of compute Dispatch calls to actually denoise input signals. Please, refer to NrdIntegration::Denoise() and NrdIntegration::Dispatch() calls in NRDIntegration.hpp file as an example of an integration using low level RHI.

NRD doesn’t have a "resize" functionality. On resolution change the old denoiser needs to be destroyed and a new one needs to be created with new parameters. But NRD supports dynamic resolution scaling via CommonSettings::resolutionScale.

Some textures can be requested as inputs or outputs for a method (see the next section). Required resources are specified near a denoiser declaration inside the Denoiser enum class. Also NRD.hlsli has a comment near each front-end or back-end function, clarifying which resources this function is for.

NON-NOISY INPUTS

Commons inputs for primary hits (if PSR is not used, common use case) or for secondary hits (if PSR is used, valid only for 0-roughness):

All textures should be NaN free at each pixel, even at pixels outside of denoising range.

The illustration below shows expected inputs for primary hits:

Input without PSR

hitDistance = length( B - A ); // hitT for 1st bounce (recommended baseline)

IN_VIEWZ = TransformToViewSpace( A ).z;
IN_NORMAL_ROUGHNESS = GetNormalAndRoughnessAt( A );
IN_MV = GetMotionAt( A );

See NRDDescs.h and NRD.hlsli for more details and descriptions of other inputs and outputs.

NOISY INPUTS

NRD sample is a good start to familiarize yourself with input requirements and best practices, but main requirements can be summarized to:

Radiance:

Hit distance (REBLUR and RELAX):

Distance to occluder (SIGMA):

See NRDDescs.h and NRD.hlsli for more details and descriptions of other inputs and outputs.

IMPROVING OUTPUT QUALITY

The temporal part of NRD naturally suppresses jitter, which is essential for upscaling techniques. If an SH denoiser is in use, a high quality resolve can be applied to the final output to regain back macro details, micro details and per-pixel jittering. As an example, the image below demonstrates the results after and before resolve with active DLSS (quality mode).

Resolve

The resolve process takes place on the application side and has the following modular structure:

Shader code:

// Diffuse
float4 diff = gIn_Diff.SampleLevel( gLinearSampler, pixelUv, 0 );
float4 diff1 = gIn_DiffSh.SampleLevel( gLinearSampler, pixelUv, 0 );
NRD_SG diffSg = REBLUR_BackEnd_UnpackSh( diff, diff1 );

// Specular
float4 spec = gIn_Spec.SampleLevel( gLinearSampler, pixelUv, 0 );
float4 spec1 = gIn_SpecSh.SampleLevel( gLinearSampler, pixelUv, 0 );
NRD_SG specSg = REBLUR_BackEnd_UnpackSh( spec, spec1 );

// ( Optional ) AO / SO ( available only for REBLUR )
diff.w = diffSg.normHitDist;
spec.w = specSg.normHitDist;

if( gResolve )
{
    // ( Optional ) replace "roughness" with "roughnessAA"
    roughness = NRD_SG_ExtractRoughnessAA( specSg );

    // Regain macro-details
    diff.xyz = NRD_SG_ResolveDiffuse( diffSg, N ); // or NRD_SH_ResolveDiffuse( sg, N )
    spec.xyz = NRD_SG_ResolveSpecular( specSg, N, V, roughness );

    // Regain micro-details & jittering // TODO: preload N and Z into SMEM
    float3 Ne = NRD_FrontEnd_UnpackNormalAndRoughness( gIn_Normal_Roughness[ pixelPos + int2( 1, 0 ) ] ).xyz;
    float3 Nw = NRD_FrontEnd_UnpackNormalAndRoughness( gIn_Normal_Roughness[ pixelPos + int2( -1, 0 ) ] ).xyz;
    float3 Nn = NRD_FrontEnd_UnpackNormalAndRoughness( gIn_Normal_Roughness[ pixelPos + int2( 0, 1 ) ] ).xyz;
    float3 Ns = NRD_FrontEnd_UnpackNormalAndRoughness( gIn_Normal_Roughness[ pixelPos + int2( 0, -1 ) ] ).xyz;

    float Ze = gIn_ViewZ[ pixelPos + int2( 1, 0 ) ];
    float Zw = gIn_ViewZ[ pixelPos + int2( -1, 0 ) ];
    float Zn = gIn_ViewZ[ pixelPos + int2( 0, 1 ) ];
    float Zs = gIn_ViewZ[ pixelPos + int2( 0, -1 ) ];

    float2 scale = NRD_SG_ReJitter( diffSg, specSg, Rf0, V, roughness, viewZ, Ze, Zw, Zn, Zs, N, Ne, Nw, Nn, Ns );

    diff.xyz *= scale.x;
    spec.xyz *= scale.y;
}
else
{
    // ( Optional ) Unresolved color matching the non-SH version of the denoiser
    diff.xyz = NRD_SG_ExtractColor( diffSg );
    spec.xyz = NRD_SG_ExtractColor( specSg );
}

Re-jittering math with minorly modified inputs can also be used with RESTIR produced sampling without involving SH denoisers. You only need to get light direction in the current pixel from RESTIR. Despite that RESTIR produces noisy light selections, its low variations can be easily handled by DLSS or other upscaling techs.

VALIDATION LAYER

Validation

If CommonSettings::enableValidation = true REBLUR & RELAX denoisers render debug information into OUT_VALIDATION output. Alpha channel contains layer transparency to allow easy mix with the final image on the application side. Currently the following viewport layout is used on the screen:

0123
4567
891011
12131415

where:

REBLUR specific:

MEMORY REQUIREMENTS

The Persistent column (matches NRD Permanent pool) indicates how much of the Working set is required to be left intact for subsequent frames of the application. This memory stores the history resources consumed by NRD. The Aliasable column (matches NRD Transient pool) shows how much of the Working set may be aliased by textures or other resources used by the application outside of the operating boundaries of NRD.

ResolutionDenoiserWorking set (Mb)Persistent (Mb)Aliasable (Mb)
1080pREBLUR_DIFFUSE84.5642.2542.31
REBLUR_DIFFUSE_OCCLUSION40.3125.3814.94
REBLUR_DIFFUSE_SH135.1959.1276.06
REBLUR_SPECULAR103.6250.7552.88
REBLUR_SPECULAR_OCCLUSION48.8133.8814.94
REBLUR_SPECULAR_SH154.2567.6286.62
REBLUR_DIFFUSE_SPECULAR164.8871.8893.00
REBLUR_DIFFUSE_SPECULAR_OCCLUSION67.9438.1229.81
REBLUR_DIFFUSE_SPECULAR_SH266.12105.62160.50
REBLUR_DIFFUSE_DIRECTIONAL_OCCLUSION84.5642.2542.31
RELAX_DIFFUSE90.8154.8835.94
RELAX_DIFFUSE_SH158.3188.6269.69
RELAX_SPECULAR101.4463.3838.06
RELAX_SPECULAR_SH168.9497.1271.81
RELAX_DIFFUSE_SPECULAR168.9497.1271.81
RELAX_DIFFUSE_SPECULAR_SH303.94164.62139.31
SIGMA_SHADOW31.888.4423.44
SIGMA_SHADOW_TRANSLUCENCY50.818.4442.38
REFERENCE33.7533.750.00
1440pREBLUR_DIFFUSE150.0675.0075.06
REBLUR_DIFFUSE_OCCLUSION71.3145.0026.31
REBLUR_DIFFUSE_SH240.06105.00135.06
REBLUR_SPECULAR183.8190.0093.81
REBLUR_SPECULAR_OCCLUSION86.3160.0026.31
REBLUR_SPECULAR_SH273.81120.00153.81
REBLUR_DIFFUSE_SPECULAR292.56127.50165.06
REBLUR_DIFFUSE_SPECULAR_OCCLUSION120.0667.5052.56
REBLUR_DIFFUSE_SPECULAR_SH472.56187.50285.06
REBLUR_DIFFUSE_DIRECTIONAL_OCCLUSION150.0675.0075.06
RELAX_DIFFUSE161.3197.5063.81
RELAX_DIFFUSE_SH281.31157.50123.81
RELAX_SPECULAR180.06112.5067.56
RELAX_SPECULAR_SH300.06172.50127.56
RELAX_DIFFUSE_SPECULAR300.06172.50127.56
RELAX_DIFFUSE_SPECULAR_SH540.06292.50247.56
SIGMA_SHADOW56.3815.0041.38
SIGMA_SHADOW_TRANSLUCENCY90.1215.0075.12
REFERENCE60.0060.000.00
2160pREBLUR_DIFFUSE318.88159.38159.50
REBLUR_DIFFUSE_OCCLUSION151.5095.6255.88
REBLUR_DIFFUSE_SH510.12223.12287.00
REBLUR_SPECULAR390.56191.25199.31
REBLUR_SPECULAR_OCCLUSION183.38127.5055.88
REBLUR_SPECULAR_SH581.81255.00326.81
REBLUR_DIFFUSE_SPECULAR621.62270.94350.69
REBLUR_DIFFUSE_SPECULAR_OCCLUSION255.06143.44111.62
REBLUR_DIFFUSE_SPECULAR_SH1004.12398.44605.69
REBLUR_DIFFUSE_DIRECTIONAL_OCCLUSION318.88159.38159.50
RELAX_DIFFUSE342.81207.25135.56
RELAX_DIFFUSE_SH597.81334.75263.06
RELAX_SPECULAR382.69239.12143.56
RELAX_SPECULAR_SH637.69366.62271.06
RELAX_DIFFUSE_SPECULAR637.69366.62271.06
RELAX_DIFFUSE_SPECULAR_SH1147.69621.62526.06
SIGMA_SHADOW119.9431.8888.06
SIGMA_SHADOW_TRANSLUCENCY191.5631.88159.69
REFERENCE127.50127.500.00

INTEGRATION VARIANTS

VARIANT 1: Black-box library (using the application-side Render Hardware Interface)

RHI must have the ability to do the following:

VARIANT 2: White-box library (using the application-side Render Hardware Interface)

Logically it's close to the Method 1, but the integration takes place in the full source code (only the NRD project is needed). In this case NRD shaders are handled by the application shader compilation pipeline. The application should still use NRD via NRD API to preserve forward compatibility. This variant suits best for compilation on other platforms (consoles, ARM), unlocks NRD modification on the application side and increases portability.

VARIANT 3: Black-box library (using native API pointers)

If Graphics API's native pointers are retrievable from the RHI, the standard NRD integration layer can be used to greatly simplify the integration. In this case, the application should only wrap up native pointers for the Device, CommandList and some input / output Resources into entities, compatible with an API abstraction layer (NRI), and all work with NRD library will be hidden inside the integration layer:

Engine or App → native objects → NRD integration layer → NRI → NRD

NRI = NVIDIA Rendering Interface - an abstraction layer on top of Graphics APIs: DX11, DX12 and VULKAN. NRI has been designed to provide low overhead access to the Graphics APIs and simplify development of DX12 and VULKAN applications. NRI API has been influenced by VULKAN as the common denominator among these 3 APIs.

NRI and NRD are ready-to-use products. The application must expose native pointers only for Device, Resource and CommandList entities (no SRVs and UAVs - they are not needed, everything will be created internally). Native resource pointers are needed only for the denoiser inputs and outputs (all intermediate textures will be handled internally). Descriptor heap will be changed to an internal one, so the application needs to bind its original descriptor heap after invoking the denoiser.

In rare cases, when the integration via the engine’s RHI is not possible and the integration using native pointers is complicated, a "DoDenoising" call can be added explicitly to the application-side RHI. It helps to avoid increasing code entropy.

The pseudo code below demonstrates how NRD integration and NRI can be used to wrap native Graphics API pointers into NRI objects to establish connection between the application and NRD:

//=======================================================================================================
// INITIALIZATION - DECLARATIONS
//=======================================================================================================

#include "NRI.h"
#include "Extensions/NRIHelper.h"
#include "Extensions/NRIWrapperD3D11.h"
#include "Extensions/NRIWrapperD3D12.h"
#include "Extensions/NRIWrapperVK.h"

#include "NRD.h"
#include "NRDIntegration.hpp"

// bufferedFramesNum (usually 2-3 frames):
//      The application must provide number of buffered frames, it's needed to guarantee that
//      constant data and descriptor sets are not overwritten while being executed on the GPU.
// enableDescriptorCaching:
//      true - enables descriptor caching for the whole lifetime of an NrdIntegration instance
//      false - descriptors are cached only within a single "Denoise" call
NrdIntegration NRD = NrdIntegration(bufferedFramesNum, enableDescriptorCaching, "Name");

struct NriInterface
    : public nri::CoreInterface
    , public nri::HelperInterface
    , public nri::WrapperD3D12Interface
{};
NriInterface NRI;

//=======================================================================================================
// INITIALIZATION - WRAP NATIVE DEVICE
//=======================================================================================================

// Wrap the device
nri::DeviceCreationD3D12Desc deviceDesc = {};
deviceDesc.d3d12Device = ...;
deviceDesc.d3d12GraphicsQueue = ...;
deviceDesc.enableNRIValidation = false;

nri::Device* nriDevice = nullptr;
nri::Result nriResult = nri::nriCreateDeviceFromD3D12Device(deviceDesc, nriDevice);

// Get core functionality
nriResult = nri::nriGetInterface(*nriDevice,
  NRI_INTERFACE(nri::CoreInterface), (nri::CoreInterface*)&NRI);

nriResult = nri::nriGetInterface(*nriDevice,
  NRI_INTERFACE(nri::HelperInterface), (nri::HelperInterface*)&NRI);

// Get appropriate "wrapper" extension (XXX - can be D3D11, D3D12 or VULKAN)
nriResult = nri::nriGetInterface(*nriDevice,
  NRI_INTERFACE(nri::WrapperXXXInterface), (nri::WrapperXXXInterface*)&NRI);

//=======================================================================================================
// INITIALIZATION - INITIALIZE NRD
//=======================================================================================================

const nrd::DenoiserDesc denoiserDescs[] =
{
    // Put neeeded denoisers here, like:
    { identifier1, nrd::Denoiser::XXX },
    { identifier2, nrd::Denoiser::YYY },
};

nrd::InstanceCreationDesc instanceCreationDesc = {};
instanceCreationDesc.denoisers = denoiserDescs;
instanceCreationDesc.denoisersNum = GetCountOf(denoiserDescs);

// NRD itself is flexible and supports any kind of dynamic resolution scaling, but NRD INTEGRATION pre-
// allocates resources with statically defined dimensions. DRS is only supported by adjusting the viewport
// via "CommonSettings::rectSize"
bool result = NRD.Initialize(resourceWidth, resourceHeight, instanceCreationDesc, *nriDevice, NRI, NRI);

//=======================================================================================================
// INITIALIZATION or RENDER - WRAP NATIVE POINTERS
//=======================================================================================================

// Wrap a command buffer
nri::CommandBufferD3D12Desc commandBufferDesc = {};
commandBufferDesc.d3d12CommandList = (ID3D12GraphicsCommandList*)d3d12CommandList;

// Not needed for NRD integration layer, but needed for NRI validation layer
commandBufferDesc.d3d12CommandAllocator = (ID3D12CommandAllocator*)d3d12CommandAllocatorOrJustNonNull;

nri::CommandBuffer* nriCommandBuffer = nullptr;
NRI.CreateCommandBufferD3D12(*nriDevice, commandBufferDesc, nriCommandBuffer);

// Wrap required textures (better do it only once on initialization)
nri::TextureBarrierDesc entryDescs[N] = {};

for (uint32_t i = 0; i < N; i++)
{
    nri::TextureBarrierDesc& entryDesc = entryDescs[i];
    const MyResource& myResource = GetMyResource(i);

    nri::TextureD3D12Desc textureDesc = {};
    textureDesc.d3d12Resource = myResource->GetNativePointer();

    NRI.CreateTextureD3D12(*nriDevice, textureDesc, (nri::Texture*&)entryDesc.texture );

    // You need to specify the current state of the resource here, after denoising NRD can modify
    // this state. Application must continue state tracking from this point.
    // Useful information:
    //    SRV = nri::AccessBits::SHADER_RESOURCE, nri::TextureLayout::SHADER_RESOURCE
    //    UAV = nri::AccessBits::SHADER_RESOURCE_STORAGE, nri::TextureLayout::GENERAL
    entryDesc.after.access = ConvertResourceStateToAccess( myResource->GetCurrentState() );
    entryDesc.after.layout = ConvertResourceStateToLayout( myResource->GetCurrentState() );
}

//=======================================================================================================
// RENDER - DENOISE
//=======================================================================================================

// Must be called once on a frame start
NRD.NewFrame();

// Set common settings
//  - for the first time use defaults
//  - currently NRD supports only the following view space: X - right, Y - top, Z - forward or backward
nrd::CommonSettings commonSettings = {};
PopulateCommonSettings(commonSettings);

NRD.SetCommonSettings(commonSettings);

// Set settings for each method in the NRD instance
nrd::XxxSettings settings1 = {};
PopulateXxxSettings(settings1);

NRD.SetDenoiserSettings(identifier1, &settings1);

nrd::YyySettings settings2 = {};
PopulateYyySettings(settings2);

NRD.SetDenoiserSettings(identifier2, &settings2);

// Fill up the user pool
NrdUserPool userPool = {};
{
    // Set "entryDescs" into the "in-use" slots, applying remapping if necessary
    NrdIntegration_SetResource(userPool, nrd::ResourceType::IN_NORMAL_ROUGHNESS, &entryDescs[0]);
    NrdIntegration_SetResource(userPool, nrd::ResourceType::IN_VIEWZ, &entryDescs[1]);
    ...
};

const nrd::Identifier denoisers[] = {identifier1, identifier2};
NRD.Denoise(denoisers, helper::GetCountOf(denoisers), *nriCommandBuffer, userPool);

// IMPORTANT: NRD integration binds own descriptor pool, don't forget to re-bind back your pool (heap)

//=======================================================================================================
// SHUTDOWN or RENDER - CLEANUP
//=======================================================================================================

// Better do it only once on shutdown
for (uint32_t i = 0; i < N; i++)
    NRI.DestroyTexture(entryDescs[i].texture);

NRI.DestroyCommandBuffer(*nriCommandBuffer);

//=======================================================================================================
// SHUTDOWN - DESTROY
//=======================================================================================================

// Also NRD needs to be recreated on "resize"
NRD.Destroy();

// Release wrapped device
nri::nriDestroyDevice(*nriDevice);

Shader part:

#if 1
    #include "NRDEncoding.hlsli"
#else
    // Or define NRD encoding in Cmake and deliver macro definitions to shader compilation command line
#endif

#include "NRD.hlsli"

// Call corresponding "front end" function to encode data for NRD (NRD.hlsli indicates which function
// needs to be used for a specific input for a specific denoiser). For example:

float4 nrdIn = RELAX_FrontEnd_PackRadianceAndHitDist(radiance, hitDistance);

// Call corresponding "back end" function to decode data produced by NRD. For example:

float4 nrdOut = RELAX_BackEnd_UnpackRadiance(nrdOutEncoded);

RECOMMENDATIONS AND BEST PRACTICES: GREATER TIPS

Denoising is not a panacea or miracle. Denoising works best with ray tracing results produced by a suitable form of importance sampling. Additionally, NRD has its own restrictions. The following suggestions should help to achieve best image quality:

MATERIAL DE-MODULATION (IRRADIANCE → RADIANCE)

NRD has been designed to work with pure radiance coming from a particular direction. This means that data in the form "something / probability" should be avoided if possible because overall entropy of the input signal will be increased (but it doesn't mean that denoising won't work). Additionally, it means that materials needs to be decoupled from the input signal, i.e. irradiance, typically produced by a path tracer, needs to be transformed into radiance, i.e. BRDF should be applied after denoising. This is achieved by using "demodulation" trick:

// Diffuse
Denoising( diffuseRadiance * albedo ) → NRD( diffuseRadiance / albedo ) * albedo

// Specular
float3 preintegratedBRDF = PreintegratedBRDF( Rf0, N, V, roughness )
Denoising( specularRadiance * BRDF ) → NRD( specularRadiance * BRDF / preintegratedBRDF ) * preintegratedBRDF

A good approximation for pre-integrated specular BRDF can be found [here](https://github.com/NVIDIAGameWorks/MathLib/blob/407ecd0d1892d12ee1ec98c3d46cbeed73b79a0d/STL.hlsli#L2147. Pre-integrated specular BRDF can also be referenced as "specular albedo" or "environment BRDF".

COMBINED DENOISING OF DIRECT AND INDIRECT LIGHTING

  1. For specular signal use indirect hitT for both direct and indirect lighting

The reason is that the denoiser uses hitT mostly for calculating motion vectors for reflections. For that purpose, the denoiser expects to see hitT from surfaces that are in the specular reflection lobe. When calculating direct lighting (NEE/RTXDI), we select a light per pixel, and the distance to that light becomes the hitT for both diffuse and specular channels. In many cases, the light is selected for a surface because of its diffuse contribution, not specular, which makes the specular channel contain the hitT of a diffuse light. That confuses the denoiser and breaks reprojection. On the other hand, the indirect specular hitT is always computed by tracing rays in the specular lobe.

  1. For diffuse signal hitT can be further adjusted by mixing hitT from direct and indirect rays to get sharper shadows

Use first bounce hit distance for the indirect in the pseudo-code below:

float hitDistContribution = directDiffuseLuminance / ( directDiffuseLuminance + indirectDiffuseLuminance + EPS );

float maxContribution = 0.5; // 0.65 works good as well
float directHitDistContribution = min(directHitDistContribution, maxContribution); // avoid over-sharpening

hitDist = lerp(indirectDiffuseHitDist, directDiffuseHitDist, directHitTContribution);

INTERACTION WITH PRIMARY SURFACE REPLACEMENTS (PSR)

When denoising reflections in pure mirrors, some advantages can be reached if NRD "sees" the first "non-pure mirror" point after a series of pure mirror bounces (delta events). This point is called Primary Surface Replacement.

Primary Surface Replacement (PSR) can be used with NRD.

Notes, requirements and restrictions:

IMPORTANT: in other words, PSR is perfect for flat mirrors. PSR on curved surfaces works even without respecting curvature, but reprojection artefacts can appear.

In case of PSR NRD disocclusion logic doesn't take curvature at primary hit into account, because data for primary hits is replaced. This can lead to more intense disocclusions on bumpy surfaces due to significant ray divergence. To mitigate this problem 2x-10x larger disocclusionThreshold can be used. This is an applicable solution if the denoiser is used to denoise surfaces with PSR only (glass only, for example). In a general case, when PSR and normal surfaces are mixed on the screen, higher disocclusion thresholds are needed only for pixels with PSR. This can be achieved by using IN_DISOCCLUSION_THRESHOLD_MIX input to smoothly mix baseline disocclusionThreshold into bigger disocclusionThresholdAlternate from CommonSettings. Most likely the increased disocclusion threshold is needed only for pixels with normal details at primary hits (local curvature is not zero).

The illustration below shows expected inputs for secondary hits:

Input with PSR

hitDistance = length( C - B ); // hitT for 2nd bounce, but it's 1st bounce in the reflected world
Bvirtual = A + viewVector * length( B - A );

IN_VIEWZ = TransformToViewSpace( Bvirtual ).z;
IN_NORMAL_ROUGHNESS = GetVirtualSpaceNormalAndRoughnessAt( B );
IN_MV = GetMotionAt( B );

INTERACTION WITH INFs AND NANs

INTERACTION WITH FRAME GENERATION TECHNIQUES

Frame generation (FG) techniques boost FPS by interpolating between 2 last available frames. NRD works better when frame rate increases, because it gets more data per second. It's not the case for FG, because all rendering pipeline underlying passes (like, denoising) continue to work on the original non-boosted framerate. GetMaxAccumulatedFrameNum helper should get a real FPS, not a fake one.

HAIR DENOISING TIPS

NRD tries to preserve jittering at least on geometrical edges, it's essential for upscalers, which are usually applied at the end of the rendering pipeline. It naturally moves the problem of anti-aliasing to the application side. In order, it implies the following obvious suggestions:

Sub-pixel thin geometry of strand-based hair transforms "normals guide" into jittering & flickering pixel mess, i.e. the guide itself becomes noisy. It worsens denoising IQ. At least for NRD better to replace geometry normals in "normals guide" with a vector = normalize( cross( T, B ) ), where:

Hair strands tangent vectors can't be used as "normals guide" for NRD due to BRDF and curvature related calculations, requiring a vector, which can be considered a "normal" vector.

RECOMMENDATIONS AND BEST PRACTICES: LESSER TIPS

[NRD] The NRD API has been designed to support integration into native VULKAN apps. If the RHI you work with is DX11-like, not all provided data will be needed.

[NRD] Read all comments in NRDDescs.h, NRDSettings.h and NRD.hlsli.

[NRD] If you are unsure of which parameters to use - use defaults via {} construction. It helps to improve compatibility with future versions and offers optimal IQ, because default settings are always adjusted by recent algorithmic changes.

[NRD] NRD requires linear roughness and world-space normals. See NRD.hlsli for more details and supported customizations.

[NRD] NRD requires non-jittered matrices.

[NRD] Most denoisers do not write into output pixels outside of CommonSettings::denoisingRange.

[NRD] When upgrading to the latest version keep an eye on ResourceType enumeration. The order of the input slots can be changed or something can be added, you need to adjust the inputs accordingly to match the mapping. Or use NRD integration to simplify the process.

[NRD] All pixels in floating point textures should be INF / NAN free to avoid propagation, because such values are used in weight calculations and accumulation of a weighted sum. Functions XXX_FrontEnd_PackRadianceAndHitDist perform optional NAN / INF clearing of the input signal. There is a boolean to skip these checks.

[NRD] All denoisers work with positive RGB inputs (some denoisers can change color space in front end functions). For better image quality, HDR color inputs need to be in a sane range [0; 250], because the internal pipeline uses FP16 and RELAX tracks second moments of the input signal, i.e. x^2 must fit into FP16 range. If the color input is in a wider range, any form of non-aggressive color compression can be applied (linear scaling, pow-based or log-based methods). REBLUR supports wider HDR ranges, because it doesn't track second moments. Passing pre-exposured colors (i.e. color * exposure) is not recommended, because a significant momentary change in exposure is hard to react to in this case.

[NRD] NRD can track camera motion internally. For the first time pass all MVs set to 0 (you can use CommonSettings::motionVectorScale = {0} for this) and set CommonSettings::isMotionVectorInWorldSpace = true, it will allow you to simplify the initial integration. Enable application-provided MVs after getting denoising working on static objects.

[NRD] Using 2D MVs can lead to massive history reset on moving objects, because 2D motion provides information only about pixel screen position but not about real 3D world position. Consider using 2.5D or 3D MVs instead. 2.5D motion, which is 2D motion with additionally provided viewZ delta (i.e. viewZprev = viewZ + MV.z), is even better, because it has the same benefits as 3D motion, but doesn't suffer from imprecision problems caused by world-space delta rounding to FP16 during MV patching on the NRD side.

[NRD] Firstly, try to get a working reprojection on a diffuse signal for camera rotations only (without camera motion).

[NRD] Diffuse and specular signals must be separated at primary hit (or at secondary hit in case of PSR).

[NRD] Denoising logic is driven by provided hit distances. For indirect lighting denoising passing hit distance for the 1st bounce only is a good baseline. For direct lighting a distance to an occluder or a light source is needed. Primary hit distance must be excluded in any case.

[NRD] Importance sampling is recommended to achieve good results in case of complex lighting environments. Consider using:

[NRD] Any form of a radiance cache (SHARC or NRC) is highly recommended to achieve better signal quality and improve behavior in disocclusions.

[NRD] Additionally the quality of the input signal can be increased by re-using already denoised information from the current or the previous frame.

[NRD] Hit distances should come from an importance sampling method. But if denoising of AO/SO is needed, AO/SO can come from cos-weighted (or VNDF) sampling in a tradeoff of IQ.

[NRD] Low discrepancy sampling (blue noise) helps to get more stable output in 0.5-1 rpp mode. It's a must for REBLUR-based Ambient and Specular Occlusion denoisers and SIGMA.

[NRD] It's recommended to set CommonSettings::accumulationMode to RESET for a single frame, if a history reset is needed. If history buffers are recreated or contain garbage, it's recommended to use CLEAR_AND_RESET for a single frame. CLEAR_AND_RESET is not free because clearing is done in a compute shader. Render target clears on the application side should be prioritized over this solution.

[NRD] If there are areas (besides sky), which don't require denoising (for example, casting a specular ray only if roughness is less than some threshold), providing viewZ > CommonSettings::denoisingRange in IN_VIEWZ texture for such pixels will effectively skip denoising. Additionally, the data in such areas won't contribute to the final result.

[NRD] If there are areas (besides sky), which don't require denoising (for example, skipped diffuse rays for true metals). materialID and materialMask can be used to drive spatial passes.

[NRD] Input signal quality can be improved by enabling pre-pass via setting diffusePrepassBlurRadius and specularPrepassBlurRadius to a non-zero value. Pre-pass is needed more for specular and less for diffuse, because pre-pass outputs optimal hit distance for specular tracking (see the sample for more details).

[NRD] In case of probabilistic diffuse / specular split at the primary hit, hit distance reconstruction pass must be enabled, if exposed in the denoiser (see HitDistanceReconstructionMode).

[NRD] In case of probabilistic diffuse / specular split at the primary hit, pre-pass must be enabled, if exposed in the denoiser (see diffusePrepassBlurRadius and specularPrepassBlurRadius).

[NRD] Maximum number of accumulated frames can be FPS dependent. The following formula can be used on the application side to adjust maxAccumulatedFrameNum, maxFastAccumulatedFrameNum and potentially historyFixFrameNum too:

maxAccumulatedFrameNum = accumulationPeriodInSeconds * FPS

[NRD] Fast history is the input signal, accumulated for a few frames. Fast history helps to minimize lags in the main history, which is accumulated for more frames. The number of accumulated frames in the fast history needs to be carefully tuned to avoid introducing significant bias and dirt. Initial integration should be done with default settings. Bear in mind the following recommendation:

maxAccumulatedFrameNum > maxFastAccumulatedFrameNum > historyFixFrameNum

[NRD] In case of quarter resolution tracing and denoising use pixelPos / 2 as texture coordinates. Using a "rotated grid" approach (when a pixel gets selected from 2x2 footprint one by one) is not recommended because it significantly bumps entropy of non-noisy inputs, leading to more disocclusions. In case of REBLUR it's recommended to increase sigmaScale in antilag settings. "Nearest Z" upsampling works best for upscaling of the denoised output. Code, as well as upsampling function, can be found in NRD sample releases before 3.10.

[NRD] SH denoisers can use more relaxed lobeAngleFraction. It can help to improve stability, while details will be reconstructed back by SG resolve.

[REBLUR] If more performance is needed, consider using enablePerformanceMode = true.

[REBLUR] REBLUR expects hit distances in a normalized form. To avoid mismatching, REBLUR_FrontEnd_GetNormHitDist must be used for normalization. Normalization parameters should be passed into NRD as HitDistanceParameters for internal hit distance denormalization. Some tweaking can be needed here, but in most cases default HitDistanceParameters works well. REBLUR outputs denoised normalized hit distance, which can be used by the application as ambient or specular occlusion (AO & SO) (see unpacking functions from NRD.hlsli).

[REBLUR/RELAX] Antilag parameters need to be carefully tuned. Initial integration should be done with disabled antilag.

[RELAX] RELAX works well with signals produced by RTXDI or very clean high RPP signals. The Sweet Home of RELAX is RTXDI sample. Please, consider getting familiar with this application.

[SIGMA] Using "blue" noise helps to minimize shadow shimmering and flickering. It works best if the pattern has limited number of animated frames (4-8) or it is static on the screen.

[SIGMA] SIGMA can be used for multi-light shadow denoising if applied "per light". SigmaSettings::stabilizationStrength can be set to 0 to disable temporal history. It provides the following benefits:

[SIGMA] In theory SIGMA_TRANSLUCENT_SHADOW can be used as a "single-pass" shadow denoiser for shadows from multiple light sources:

L[i] - unshadowed analytical lighting from a single light source (not noisy)<br/> S[i] - stochastically sampled light visibility for L[i] (noisy)<br/> Σ( L[i] ) - unshadowed analytical lighting, typically a result of tiled lighting (HDR, not in range [0; 1])<br/> Σ( L[i] × S[i] ) - final lighting (what we need to get)

The idea:<br/> L1 × S1 + L2 × S2 + L3 × S3 = ( L1 + L2 + L3 ) × [ ( L1 × S1 + L2 × S2 + L3 × S3 ) / ( L1 + L2 + L3 ) ]

Or:<br/> Σ( L[i] × S[i] ) = Σ( L[i] ) × [ Σ( L[i] × S[i] ) / Σ( L[i] ) ]<br/> Σ( L[i] × S[i] ) / Σ( L[i] ) - normalized weighted sum, i.e. pseudo translucency (LDR, in range [0; 1])

Input data preparation example:

float3 Lsum = 0;
float3 LSsum = 0.0;
float Wsum = 0.0;
float Psum = 0.0;

for( uint i = 0; i < N; i++ )
{
    float3 L = ComputeLighting( i );
    Lsum += L;

    // "distanceToOccluder" should respect rules described in NRD.hlsli in "INPUT PARAMETERS" section
    float distanceToOccluder = SampleShadow( i );
    float shadow = !IsOccluded( distanceToOccluder );
    LSsum += L * shadow;

    // The weight should be zero if a pixel is not in the penumbra, but it is not trivial to compute...
    float weight = ...;
    weight *= Luminance( L );
    Wsum += weight;

    float penumbraRadius = SIGMA_FrontEnd_PackPenumbra( ... ).x;
    Psum += penumbraRadius * weight;
}

float3 translucency = LSsum / max( Lsum, NRD_EPS );
float penumbraRadius = Psum / max( Wsum, NRD_EPS );

After denoising the final result can be computed as:

Σ( L[i] × S[i] ) = Σ( L[i] ) × OUT_SHADOW_TRANSLUCENCY.yzw

Is this a biased solution? If spatial filtering is off - no, because we just reorganized the math equation. If spatial filtering is on - yes, because denoising will be driven by most important light in a given pixel.

This solution is limited and hard to use: