Awesome

ros2_openvino_toolkit

Table of Contents

• ➤ Overview
  • ROS2 Version Supported
  • Inference Features Supported
• ➤ Prerequisite
• ➤ Introduction
  • Design Architecture
  • Logic Flow
• ➤ Supported Features
  • Multiple Input Components
  • Inference Implementations
  • ROS Interfaces and Outputs
  • Demo Result Snapshots
• ➤ Installation & Launching
  • Deploy in Local Environment
  • Deploy in Docker
• ➤ Reference
• ➤ FAQ
• ➤ Feedback
• ➤ More Information

Overview

ROS2 Version Supported

Inference Features Supported

• Object Detection
• Face Detection
• Age Gender Recognition
• Emotion Recognition
• Head Pose Estimation
• Object Segmentation
• Person Re-Identification
• Vehicle Attribute Detection
• Vehicle License Plate Detection

Prerequisite

Introduction

Design Architecture

From the view of hirarchical architecture design, the package is divided into different functional components, as shown in below picture.

• Intel® OpenVINO™ toolkit provides a ROS-adapted runtime framework of neural network which quickly deploys applications and solutions for vision inference. By leveraging Intel® OpenVINO™ toolkit and corresponding libraries, this ROS2 runtime framework extends workloads across Intel® hardware (including accelerators) and maximizes performance.
  • Increase deep learning workload performance up to 19x1 with computer vision accelerators from Intel.
  • Unleash convolutional neural network (CNN)-based deep learning inference using a common API.
  • Speed development using optimized OpenCV* and OpenVX* functions. See more from here for Intel OpenVINO™ introduction.

• ROS OpenVINO Runtime Framework is the main body of this repo. It provides key logic implementation for pipeline lifecycle management, resource management and ROS system adapter, which extends Intel OpenVINO toolkit and libraries. Furthermore, this runtime framework provides ways to simplify launching, configuration, data analysis and re-use.

• Diversal Input resources are data resources to be infered and analyzed with the OpenVINO framework.
• ROS interfaces and outputs currently include Topic and service. Natively, RViz output and CV image window output are also supported by refactoring topic message and inferrence results.

• Optimized Models provided by Model Optimizer component of Intel® OpenVINO™ toolkit. Imports trained models from various frameworks (Caffe*, Tensorflow*, MxNet*, ONNX*, Kaldi*) and converts them to a unified intermediate representation file. It also optimizes topologies through node merging, horizontal fusion, eliminating batch normalization, and quantization. It also supports graph freeze and graph summarize along with dynamic input freezing.

Logic Flow

From the view of logic implementation, the package introduces the definitions of parameter manager, pipeline and pipeline manager. The following picture depicts how these entities co-work together when the corresponding program is launched.

Once a corresponding program is launched with a specified .yaml config file passed in the .launch file or via commandline, parameter manager analyzes the configurations about pipeline and the whole framework, then shares the parsed configuration information with pipeline procedure. A pipeline instance is created by following the configuration info and is added into pipeline manager for lifecycle control and inference action triggering.

The contents in .yaml config file should be well structured and follow the supported rules and entity names. Please see yaml configuration guidance for how to create or edit the config files.

Pipeline fulfills the whole data handling process: initiliazing Input Component for image data gathering and formating; building up the structured inference network and passing the formatted data through the inference network; transfering the inference results and handling output, etc.

Pipeline manager manages all the created pipelines according to the inference requests or external demands (say, system exception, resource limitation, or end user's operation). Because of co-working with resource management and being aware of the whole framework, it covers the ability of performance optimization by sharing system resource between pipelines and reducing the burden of data copy.

Supported Features

Multiple Input Components

Currently, the package supports several input resources for acquiring image data. The following tables are listed:

Inference Implementations

Currently, the corresponding relation of supported inference features, models used and yaml configurations are listed as follows:

ROS interfaces and outputs

The inference results can be output in several types. One or more types can be enabled for any inference pipeline.

Topic

Specific topic(s) can be generated and published according to the given inference functionalities.

Service

Several ROS2 Services are created, expecting to be used in client/server mode, especially when synchronously getting inference results for a given image frame or when managing inference pipeline's lifecycle.</br>

RViz

RViz display is also supported by the composited topic of original image frame with inference result. To show in RViz tool, add an image marker with the composited topic: /ros2_openvino_toolkit/image_rviz(sensor_msgs::Image)

Image Window

OpenCV based image window is natively supported by the package. To enable window, Image Window output should be added into the output choices in .yaml config file. Refer to the config file guidance for more information about checking/adding this feature in your launching.

Demo Result Snapshots

For the snapshot of demo results, refer to the following picture.

• Face detection input from standard camera

• Object detection input from realsense camera

• Object segmentation input from video

• Person reidentification input from standard camera

Installation and Launching

Deploy in Local Environment

• Refer to the quick start document for getting_started_with_ros2 for detailed installation & lauching instructions.
• Refer to the quick start document for yaml configuration guidance for detailed configuration guidance.

Deploy in Docker

• Refer to the docker instruction for docker_instructions for detailed information about building docker image and launching.
• Refer to the quick start document for yaml configuration guidance for detailed configuration guidance.

Reference

• Open_model_zoo: Refer to the OpenVINO document for open_model_zoo for detailed model structure and demo samples.
• OpenVINO api 2.0: Refer to the OpenVINO document for OpenVINO_api_2.0 for latest api 2.0 transition guide.

FAQ

• How to get the IR file for yolov5?
• How to build OpenVINO by source?
• How to build RealSense by source?
• What is the basic command of Docker CLI?
• What is the canonical C++ API for interacting with ROS?

Feedback

• Report questions, issues and suggestions, using: issue.

More Information

• ROS2 OpenVINO discription written in Chinese: https://mp.weixin.qq.com/s/BgG3RGauv5pmHzV_hkVAdw

Any security issue should be reported using process at https://01.org/security