Awesome
Isaac ROS Object Detection
NVIDIA-accelerated, deep learned model support for object detection including DetectNet.
<div align="center"><img alt="original image" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example.png/" width="300px"/> <img alt="bounding box predictions using DetectNet" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example_bbox.png/" width="300px"/></div>Overview
Isaac ROS Object Detection contains ROS 2 packages to perform object
detection.
isaac_ros_rtdetr
, isaac_ros_detectnet
, and isaac_ros_yolov8
each provide a method for spatial
classification using bounding boxes with an input image. Classification
is performed by a GPU-accelerated model of the appropriate architecture:
isaac_ros_rtdetr
: RT-DETR modelsisaac_ros_detectnet
: DetectNet modelsisaac_ros_yolov8
: YOLOv8 models
The output prediction can be used by perception functions to understand the presence and spatial location of an object in an image.
<div align="center"><a class="reference internal image-reference" href="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_nodegraph.png/"><img alt="image" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_nodegraph.png/" width="800px"/></a></div>Each Isaac ROS Object Detection package is used in a graph of nodes to provide a bounding box detection array with object classes from an input image. A trained model of the appropriate architecture is required to produce the detection array.
Input images may need to be cropped and resized to maintain the aspect ratio and match the
input resolution of the specific object detection model; image resolution may be reduced to
improve DNN inference performance, which typically scales directly with
the number of pixels in the image. isaac_ros_dnn_image_encoder
provides DNN encoder utilities to process the input image into Tensors for the
object detection models.
Prediction results are decoded in model-specific ways,
often involving clustering and thresholding to group multiple detections
on the same object and reduce spurious detections.
Output is provided as a detection array with object classes.
DNNs have a minimum number of pixels that need to be visible on the object to provide a classification prediction. If a person cannot see the object in the image, it’s unlikely the DNN will. Reducing input resolution to reduce compute may reduce what is detected in the image. For example, a 1920x1080 image containing a distant person occupying 1k pixels (64x16) would have 0.25K pixels (32x8) when downscaled by 1/2 in both X and Y. The DNN may detect the person with the original input image, which provides 1K pixels for the person, and fail to detect the same person in the downscaled resolution, which only provides 0.25K pixels for the person.
<div align="center"><a class="reference internal image-reference" href="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example_bboxseg.png/"><img alt="image" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_object_detection/isaac_ros_object_detection_example_bboxseg.png/" width="800px"/></a></div>Object detection classifies a rectangle of pixels as containing an object, whereas image segmentation provides more information and uses more compute to produce a classification per pixel. Object detection is used to know if, and where in a 2D image, the object exists. If a 3D spacial understanding or size of an object in pixels is required, use image segmentation.
Isaac ROS NITROS Acceleration
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
Performance
Sample Graph<br/><br/> | Input Size<br/><br/> | AGX Orin<br/><br/> | Orin NX<br/><br/> | Orin Nano 8GB<br/><br/> | x86_64 w/ RTX 4090<br/><br/> |
---|---|---|---|---|---|
RT-DETR Object Detection Graph<br/><br/><br/>SyntheticaDETR<br/><br/> | 720p<br/><br/><br/><br/> | 56.5 fps<br/><br/><br/>30 ms @ 30Hz<br/><br/> | 33.8 fps<br/><br/><br/>39 ms @ 30Hz<br/><br/> | 24.1 fps<br/><br/><br/>53 ms @ 30Hz<br/><br/> | 490 fps<br/><br/><br/>7.1 ms @ 30Hz<br/><br/> |
DetectNet Object Detection Graph<br/><br/><br/><br/> | 544p<br/><br/><br/><br/> | 70.5 fps<br/><br/><br/>26 ms @ 30Hz<br/><br/> | 30.1 fps<br/><br/><br/>46 ms @ 30Hz<br/><br/> | 22.9 fps<br/><br/><br/>57 ms @ 30Hz<br/><br/> | 254 fps<br/><br/><br/>11 ms @ 30Hz<br/><br/> |
Documentation
Please visit the Isaac ROS Documentation to learn how to use this repository.
Packages
Latest
Update 2024-12-10: Update to be compatible with JetPack 6.1