Awesome
Isaac ROS Image Pipeline
NVIDIA-accelerated Image Pipeline.
<div align="center"><img src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_pipeline/100_right.jpg/" width="300px"/> <img src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_pipeline/300_right_hallway2_rect.png/" width="300px"/></div> <div align="center"><img src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_pipeline/300_right_hallway2_gray_rect.png/" width="300px"/></div>Overview
Isaac ROS Image Pipeline is a metapackage of functionality for image processing. Camera output often needs pre-processing to meet the input requirements of multiple different perception functions. This can include cropping, resizing, mirroring, correcting for lens distortion, and color space conversion. For stereo cameras, additional processing is required to produce disparity between left + right images and a point cloud for depth perception.
This package is accelerated using the GPU and specialized hardware
engines for image computation, replacing the CPU-based
image_pipeline metapackage.
Considerable effort has been made to ensure that replacing
image_pipeline with isaac_ros_image_pipeline on a Jetson or GPU
is as painless a transition as possible.
<div align="center"><a class="reference internal image-reference" href="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_image_pipeline_nodegraph.png/"><img alt="image" src="https://media.githubusercontent.com/media/NVIDIA-ISAAC-ROS/.github/main/resources/isaac_ros_docs/repositories_and_packages/isaac_ros_image_pipeline/isaac_ros_image_pipeline_nodegraph.png/" width="800px"/></a></div>[!Note] Some image pre-processing functions use specialized hardware engines, which offload the GPU to make more compute available for other tasks.
Rectify corrects for lens distortion from the received camera sensor
message. The rectified image is resized to the input resolution for
disparity, using a crop before resizing to maintain image aspect ratio.
The image is color space converted to YUV from RGB using the luma
channel (the Y in YUV) to compute disparity using
SGM. This
common graph of nodes can be performed without the CPU processing a
single pixel using isaac_ros_image_pipeline; in comparison, using
image_pipeline, the CPU would process each pixel ~3 times.
The Isaac ROS Image Pipeline metapackage offloads the CPU from common image processing tasks so it can perform robotics functions best suited for the CPU.
Isaac ROS NITROS Acceleration
This package is powered by NVIDIA Isaac Transport for ROS (NITROS), which leverages type adaptation and negotiation to optimize message formats and dramatically accelerate communication between participating nodes.
Performance
| Sample Graph<br/><br/> | Input Size<br/><br/> | AGX Orin<br/><br/> | Orin NX<br/><br/> | Orin Nano 8GB<br/><br/> | x86_64 w/ RTX 4060 Ti<br/><br/> | x86_64 w/ RTX 4090<br/><br/> |
|---|---|---|---|---|---|---|
| Rectify Node<br/><br/><br/><br/> | 1080p<br/><br/><br/><br/> | 983 fps<br/><br/><br/>2.5 ms @ 30Hz<br/><br/> | 569 fps<br/><br/><br/>3.5 ms @ 30Hz<br/><br/> | 394 fps<br/><br/><br/>5.2 ms @ 30Hz<br/><br/> | 2500 fps<br/><br/><br/>0.88 ms @ 30Hz<br/><br/> | 2500 fps<br/><br/><br/>0.66 ms @ 30Hz<br/><br/> |
| Stereo Disparity Node<br/><br/><br/><br/> | 1080p<br/><br/><br/><br/> | 168 fps<br/><br/><br/>7.5 ms @ 30Hz<br/><br/> | 75.4 fps<br/><br/><br/>15 ms @ 30Hz<br/><br/> | 51.5 fps<br/><br/><br/>22 ms @ 30Hz<br/><br/> | 350 fps<br/><br/><br/>3.4 ms @ 30Hz<br/><br/> | 814 fps<br/><br/><br/>1.8 ms @ 30Hz<br/><br/> |
| Stereo Disparity Graph<br/><br/><br/><br/> | 1080p<br/><br/><br/><br/> | 157 fps<br/><br/><br/>12 ms @ 30Hz<br/><br/> | 72.2 fps<br/><br/><br/>20 ms @ 30Hz<br/><br/> | 49.6 fps<br/><br/><br/>28 ms @ 30Hz<br/><br/> | 349 fps<br/><br/><br/>2.5 ms @ 30Hz<br/><br/> | 791 fps<br/><br/><br/>2.7 ms @ 30Hz<br/><br/> |
Documentation
Please visit the Isaac ROS Documentation to learn how to use this repository.
Packages
isaac_ros_depth_image_procisaac_ros_image_pipelineisaac_ros_image_procisaac_ros_stereo_image_proc
Latest
Update 2024-09-26: Update for ZED compatibility