Awesome

FlexInfer

A flexible Python front-end inference SDK.

Features

Flexible

FlexInfer has a Python front-end, which makes it easy to build a computer vision product prototype.
Efficient

Most of time consuming part of FlexInfer is powered by C++ or CUDA, so FlexInfer is also efficient. If you are really hungry for efficiency and don't mind the trouble of C++, you can refer to CheetahInfer.

License

This project is released under Apache 2.0 license.

Installation

Requirements

Linux
Python 3.6 or higher
TensorRT 7.1.3.4 or higher
PyTorch 1.4.0 or higher
CUDA 10.2 or higher
volksdep 3.2.0 or higher

We have tested the following versions of OS and softwares:

OS: Ubuntu 16.04.6 LTS
Python 3.6.9
TensorRT 7.1.3.4
PyTorch 1.6.0
CUDA: 10.2
volksdep: 3.2.0

Install FlexInfer

If your platform is x86 or x64, you can create a conda virtual environment and activate it.

conda create -n flexinfer python=3.6.9 -y
conda activate flexinfer

Install volksdep following the official instructions
Setup

pip install "git+https://github.com/Media-Smart/flexinfer.git"

Usage

We provide some examples for different tasks.

Throughput benchmark

Device: Jetson AGX Xavier
CUDA: 10.2

<table> <tr> <td colspan="2" align="center" valign="center">Tasks</td> <td align="center" valign="center">framework</td> <td align="center" valign="center">version</td> <td align="center" valign="center">input shape</td> <td align="center" valign="center">data type</td> <td align="center" valign="center">throughput(FPS)</td> <td align="center" valign="center">latency(ms)</td> </tr> <tr> <td rowspan="2" colspan="2" align="center" valign="center">Classification (ResNet18)</td> <td align="center" valign="center">PyTorch</td> <td align="center" valign="center">1.5.0</td> <td align="center" valign="center">(1, 3, 224, 224)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">172</td> <td align="center" valign="center">6.01</td> </tr> <tr> <td align="center" valign="center">TensorRT</td> <td align="center" valign="center">7.1.0.16</td> <td align="center" valign="center">(1, 3, 224, 224)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">754</td> <td align="center" valign="center">1.8</td> </tr> <tr> <td rowspan="2" colspan="2" align="center" valign="center">Segmentation（U-Net）</td> <td align="center" valign="center">PyTorch</td> <td align="center" valign="center">1.5.0</td> <td align="center" valign="center">(1, 3, 513, 513)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">15</td> <td align="center" valign="center">63.27</td> </tr> <tr> <td align="center" valign="center">tensorrt</td> <td align="center" valign="center">7.1.0.16</td> <td align="center" valign="center">(1, 3, 513, 513)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">29</td> <td align="center" valign="center">34.03</td> </tr> <tr> <td rowspan="4" align="center" valign="center">Object Detection</td> <td rowspan="2" align="center" valign="center">RetinaNet-R50</td> <td align="center" valign="center">PyTorch</td> <td align="center" valign="center">1.5.0</td> <td align="center" valign="center">(1, 3, 768, 1280)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">8</td> <td align="center" valign="center">118.79</td> </tr> <tr> <td align="center" valign="center">TensorRT</td> <td align="center" valign="center">7.1.0.16</td> <td align="center" valign="center">(1, 3, 768, 1280)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">15</td> <td align="center" valign="center">68.10</td> </tr> <tr> <td rowspan="2" align="center" valign="center">TinaFace-R50-FPN-BN</td> <td align="center" valign="center">PyTorch</td> <td align="center" valign="center">1.5.0</td> <td align="center" valign="center">(1, 3, 768, 1280)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">3</td> <td align="center" valign="center">273.60</td> </tr> <tr> <td align="center" valign="center">TensorRT</td> <td align="center" valign="center">7.1.0.16</td> <td align="center" valign="center">(1, 3, 768, 1280)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">6</td> <td align="center" valign="center">159.70</td> </tr> <tr> <td rowspan="2" colspan="2" align="center" valign="center">Scene Text Recognition (ResNet-CTC)</td> <td align="center" valign="center">PyTorch</td> <td align="center" valign="center">1.5.0</td> <td align="center" valign="center">(1, 1, 32, 100)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">113</td> <td align="center" valign="center">10.75</td> </tr> <tr> <td align="center" valign="center">TensorRT</td> <td align="center" valign="center">7.1.0.16</td> <td align="center" valign="center">(1, 1, 32, 100)</td> <td align="center" valign="center">FP16</td> <td align="center" valign="center">308</td> <td align="center" valign="center">3.55</td> </tr> </table>

Media-Smart toolboxes

We provide some toolboxes of different tasks for training, testing and deploying.

Classification (vedacls)
Segmentation (vedaseg)
Object Detection (vedadet)
Scene Text Recognition (vedastr)

Contact

This repository is currently maintained by Yuxin Zou (@Yuxin Zou), Jun Sun(@ChaseMonsterAway), Hongxiang Cai (@hxcai) and Yichao Xiong (@mileistone).