Awesome

TensorRT Node for ComfyUI

This node enables the best performance on NVIDIA RTX™ Graphics Cards  (GPUs) for Stable Diffusion by leveraging NVIDIA TensorRT.

Supports:

• Stable Diffusion 1.5
• Stable Diffusion 2.1
• Stable Diffusion 3.0
• SDXL
• SDXL Turbo
• Stable Video Diffusion
• Stable Video Diffusion-XT 
• AuraFlow
• Flux

Requirements:

• GeForce RTX™ or NVIDIA RTX™ GPU
• For SDXL and SDXL Turbo, a GPU with 12 GB or more VRAM is recommended for best performance due to its size and computational intensity.
• For Stable Video Diffusion (SVD), a GPU with 16 GB or more VRAM is recommended.
• For Stable Video Diffusion-XT (SVD-XT), a GPU with 24 GB or more VRAM is recommended.
• For Flux more than 24GB vram is currently required.

Installing

The recommended way to install these nodes is to use the ComfyUI Manager to easily install them to your ComfyUI instance.

You can also manually install them by git cloning the repo to your ComfyUI/custom_nodes folder and installing the requirements like:

cd custom_nodes
git clone https://github.com/comfyanonymous/ComfyUI_TensorRT
cd ComfyUI_TensorRT
pip install -r requirements.txt

Description

NVIDIA TensorRT allows you to optimize how you run an AI model for your specific NVIDIA RTX GPU, unlocking the highest performance. To do this, we need to generate a TensorRT engine specific to your GPU.

You have the option to build either dynamic or static TensorRT engines:

• Dynamic engines support a range of resolutions and batch sizes, specified by the min and max parameters. Best performance will occur when using the optimal (opt) resolution and batch size, so specify opt parameters for your most commonly used resolution and batch size.

• Static engines only support a single resolution and batch size. These provide the same performance boost as the optimal settings for the dynamic engines.

Note: Most users will prefer dynamic engines, but static engines can be useful if you use a specific resolution + batch size combination most of the time. Static engines also require less VRAM; the wider the dynamic range, the more VRAM will be consumed.

Instructions

You can find different workflows in the workflows folder of this repo. These .json files can be loaded in ComfyUI.

Building A TensorRT Engine From a Checkpoint

1. Add a Load Checkpoint Node

2. Add either a Static Model TensorRT Conversion node or a Dynamic Model TensorRT Conversion node to ComfyUI

3. 

4. Connect the Load Checkpoint Model output to the TensorRT Conversion Node Model input.

5. 

6. 

7. To help identify the converted TensorRT model, provide a meaningful filename prefix, add this filename after “tensorrt/”

8. 

9. Click on Queue Prompt to start building the TensorRT Engines

10. 

The Model Conversion node will be highlighted while the TensorRT Engine is being built.

Additional information about the model conversion process can be seen in the console.

The first time generating an engine for a checkpoint will take awhile. Additional engines generated thereafter for the same checkpoint will be much faster. Generating engines can take anywhere from 3-10 minutes for the image generation models and 10-25 minutes for SVD. SVD-XT is an extremely extensive model - engine build times may take up to an hour.

Accelerated Image Generation Using a TensorRT Engine

TensorRT Engines are loaded using the TensorRT Loader node.

Common Issues/Limitations

ComfyUI TensorRT engines are not yet compatible with ControlNets or LoRAs. Compatibility will be enabled in a future update.

1. Add a TensorRT Loader node
2. Note, if a TensorRT Engine has been created during a ComfyUI session, it will not show up in the TensorRT Loader until the ComfyUI interface has been refreshed (F5 to refresh browser).
3. 
4. Select a TensorRT Engine from the unet_name dropdown
5. Dynamic Engines will use a filename format of:

1. dyn-b-min-max-opt-h-min-max-opt-w-min-max-opt
2. dyn=dynamic, b=batch size, h=height, w=width

6. Static Engine will use a filename format of:

1. stat-b-opt-h-opt-w-opt
2. stat=static, b=batch size, h=height, w=width

7. 
8. The model_type must match the model type of the TensorRT engine.
9. 
10. The CLIP and VAE for the workflow will need to be utilized from the original model checkpoint, the MODEL output from the TensorRT Loader will be connected to the Sampler.