Home

Awesome

DeepSort YOLOv4 based object tracking

The repository contains the implementation of DeepSort object tracking based on YOLOv4 detections. Detector inference class is implemented in several frameworks like TensorFlow, TensorFlow Lite, TensorRT, OpenCV, and OpenVINO to benchmark methods and use the best one for edge-tailored solutions.

<p align="center"><img src="data/helpers/ulica_03.gif"\></p>

How to use

python3 object_tracker.py -f trt -m <PATH TO MODEL> -s <MODEL INPUT SIZE> -n <PATH TO FILE WITH CLASS NAMES> -v <PATH TO INPUT VIDEO> --dont_show True

NOTE

If one wants to run inference one NVIDIA devices with TensoRT support needs to uncomment TrtYOLO detector import in detectors/__init__.py. The same issue is for utilizing OpenvinoYOLO class on Intel hardware.

Command line arguments

Usage: object_tracker.py [OPTIONS]

Options:
  -f, --framework TEXT      Inference framework: {tf, tflite, trt, opencv,
                            openvino}
  -m, --model_path TEXT     Path to detection model
  -n, --yolo_names TEXT     Path to YOLO class names file
  -s, --size INTEGER        Model input size
  -t, --tiny BOOLEAN        If YOLO tiny architecture
  -t, --model_type TEXT     yolov3 or yolov4
  -v, --video_path TEXT     Path to input video
  -o, --output TEXT         Path to output, inferenced video
  --output_format TEXT      Codec used in VideoWriter when saving video to
                            file
  --iou FLOAT               IoU threshold
  --score_threshold FLOAT   Confidence score threshold
  --opencv_dnn_target TEXT  Precision of OpenCV DNN model
  --device TEXT             OpenVINO inference device, available: {MYRIAD,
                            CPU, GPU}
  --dont_show BOOLEAN       Do not show video output
  --info BOOLEAN            Show detailed info of tracked objects
  --count BOOLEAN           Count objects being tracked on screen
  --help                    Show this message and exit.

Preparation of the algorithm

YOLOv4 object detector was trained with the use of Darknet framework and on the VisDrone datasets. Images were captured from UAV's perspective. Each independent object belongs to one of the 11 categories. Small and occluded parts of images where were many instances were marked as an ignored region (12 category). The YOLOv4 configuration file is in data/darknet/yolov4_visdrone.cfg, classes file is in data/classes/visdrone.names and calculated sizes of anchor boxes are in data/anchors/visdrone_anchors.txt.

The optimization and quantization process was performed with use of TensorRT for NVIDIA Jetson Xavier NX and OpenVINO for Intel Neural Compute Stick 2.

Conversion steps are described in following files:

Performance tests

The benchmark tests were performed on NVIDIA Jetson Xavier NX and Intel Neural Compute Stick 2. Jetson Xavier NX was in mode 2 (sudo nvpmodel -m 0) and clocks, fan were set to the maximum frequency with sudo jetson clocks --fan command. To evaluate Intel INCS 2 Raspberry Pi 4B was used. The evaluation results are listed below.

The full TensorFlow library was utilized for Raspberry Pi 4B. The build instruction for TensorFlow 2.2.0 is available here.

NVIDIA Jetson Xavier NX

YOLOv4-608 modelaverage time per frame [s]average detection time [s]average tracker time [s]
TensorFlow FP321.54750.33650.7518
TensorRT FP321.19770.17870.6487
TensorRT FP161.05450.05160.6583
TensorRT INT80.62410.03010.3622
TensorFlow Lite FP3222.716521.42310.8031
TensorFlow Lite FP1617.121615.96080.7268

Intel Neural Compute Stick 2 with Raspberry Pi 4B

YOLOv4-608 modelaverage time per frame [s]average detection time [s]average tracker time [s]
OpenVINO FP32 USB35.52141.27080.1709
OpenVINO FP16 USB35.44091.26090.1738
OpenVINO FP32 USB25.83451.56610.1705
OpenVINO FP16 USB25.67971.55600.1723

Raspberry Pi 4B

YOLOv4-608 modelaverage time per frame [s]average detection time [s]average tracker time [s]
TensorFlow Lite FP3219.963314.76230.2095
TensorFlow Lite FP1619.480314.45800.2058
OpenCV DNN TARGET CPU37.192331.18720.2491

Power consumption tests

During the performance evaluation, the energy efficiency of benchmarked edge devices was checked. The power consumption of Intel Neural Compute Stick 2 and Raspberry Pi 4B was measured with the use of a USB multifunction tester as is shown in the figure below.

Intel Neural Compute Stick 2 with Raspberry Pi 4B

<p align="center"><img src="data/helpers/ncs_inference.jpg"\></p>
StateAverage Voltage [V]Average Current [A]Average Power [W]
idle5.0970.1210.616
model loaded to plugins5.0420.2321.169
inference5.0210.4052.033

Raspberry Pi 4B

<p align="center"><img src="data/helpers/rpi_ncs_inference.jpg"\></p>
StateAverage Voltage [V]Average Current [A]Average Power [W]
idle5.0740.6453.272
model loaded to plugins5.0600.9785.102
inference5.0611.1375.748

NVIDIA Jetson Xavier NX

Jetson Xavier NX energy usage was inspected with the use of the jetson-stats package which is intended for monitoring and control NVIDIA Jetson devices. The results of the performed tests are presented in the table below.

StateAverage Voltage [V]Average Current [A]Average Power [W]
idle190.13712.605
Jetson clocks, fan max frequency190.18393.495
inference190.37707.163

Use cases

The possible use cases of multiple object tracking are:

References

Many thanks for great job for: