Home

Awesome

<!-- Copyright (c) 2020-2023, NVIDIA CORPORATION & AFFILIATES. All rights reserved. Licensed under the Apache License, Version 2.0 (the "License"); you may not use this file except in compliance with the License. You may obtain a copy of the License at http://www.apache.org/licenses/LICENSE-2.0 Unless required by applicable law or agreed to in writing, software distributed under the License is distributed on an "AS IS" BASIS, WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See the License for the specific language governing permissions and limitations under the License. -->

License

Triton Model Analyzer

[!Warning]

LATEST RELEASE

You are currently on the main branch which tracks under-development progress towards the next release. <br> The latest release of the Triton Model Analyzer is 1.45.0 and is available on branch r24.10.

Triton Model Analyzer is a CLI tool which can help you find a more optimal configuration, on a given piece of hardware, for single, multiple, ensemble, or BLS models running on a Triton Inference Server. Model Analyzer will also generate reports to help you better understand the trade-offs of the different configurations along with their compute and memory requirements. <br><br>

Features

Search Modes

Model Types

Other Features

Examples and Tutorials

Single Model

See the Single Model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on a simple PyTorch model.

Multi Model

See the Multi-model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on two models running concurrently on the same GPU.

Ensemble Model

See the Ensemble Model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on a simple Ensemble model.

BLS Model

See the BLS Model Quick Start for a guide on how to use Model Analyzer to profile, analyze and report on a simple BLS model. <br><br>

Documentation

Terminology

Below are definitions of some commonly used terms in Model Analyzer:

Reporting problems, asking questions

We appreciate any feedback, questions or bug reporting regarding this project. When help with code is needed, follow the process outlined in the Stack Overflow (https://stackoverflow.com/help/mcve) document. Ensure posted examples are: