Awesome
ROS2 Vision Inference
This repo contains a unified multi-threading inferencing nodes for monocular 3D object detection, depth prediction and semantic segmentation.
You could checkout the ROS1 version of each inference package:
Update 1: We have make a ROS1 version of this repo. Please use it with git checkout ros1
.
Update 2: We have added support for Metric3D, which is a state-of-the-art model for depth with scale. It can predict depth with reliable scale while generalize very well on various scenarios. We produce a reliable onnx version of the model, that could runs with TensorRT even on Jetson machines. We also extents its output to point clouds, facilitating robotic applications.
In this repo, we fully re-structure the code and messages formats for ROS2 (humble), and integrate multi-thread inferencing for three vision tasks.
- Currently all pretrained models are trained using the visionfactory repo. Thus focusing on out-door autonomous driving scenarios. But it is ok to plugin ONNX models that satisfiy the interface. Published models description:
Model | Type | Link | Description |
---|---|---|---|
monodepth_res101_384_1280.onnx | MonoDepth | link | FSNet, res101 backbone, model input shape (384x1280) trained on KITTI/KITTI360/nuscenes |
metric_3d .onnx | MonoDepth | link | Metric3Dv2, ViT backbone, supervised depth contains full pipeline from depth image to point cloud. |
bisenetv1.onnx | Segmentation | link | BiSeNetV1, model input shape (512x768) trained on remapped KITTI360/ApolloScene/CityScapes/BDD100k/a2d2 |
mono3d_yolox_576_768.onnx | Mono3D Detection | link | YoloX-m MonoFlex, model input (576x768) trained on KITTI/nuscenes/ONCE/bdd100k/cityscapes |
dla34_deform_576_768.onnx | Mono3D Detection | link | DLA34 Deformable Upsample MonoFlex, model input (576x768) trained on KITTI/nuscenes/ONCE/bdd100k/cityscapes |
dla34_deform_384_1280.onnx | Mono3D Detection | link | DLA34 Deformable Upsample MonoFlex, model input (384x1280) trained on KITTI/nuscenes/ONCE/bdd100k/cityscapes |
Getting Started
This repo relies on ROS2 and onnxruntime:
If you want to use ROS1, checkout to ROS1 branch with
git checkout ros1
. We tested the ROS1 code in ROS noetic Ubuntu 20.04 (we need python3 so we suggest we at least run at Ubuntu 20.04). The branchros1
is a standard ROS1 package built withcatkin_make
and run withroslaunch
pip3 install onnxruntime-gpu
For Orin, find the pip wheel in https://elinux.org/Jetson_Zoo with version number > 1.3 is ok
Under the workspace directory, find the launch file and change topic names and onnx checkpoint paths then build the workspace (you can choose to either launch any of the three tasks in the launch file)
colcon build --synlink-install
source install/setup.bash
ros2 launch ros2_vision_inference detect_seg_depth.launch.xml
Notice that as a known issue from ros2 python package. The launch/rviz files are copied but not symlinked when we run "colcon build", so whenever we modify the launch file, we need to rebuild the package; whenever we want to modify the rviz file, we need to save it explicitly in the src folder.
colcon build --symlink-install --packages-select=ros2_vision_inference # rebuilding only ros2_vision_inference
Interface
Subscribed Topics
/image_raw (sensor_msgs/Image)
/camera_info (sensor_msgs/CameraInfo)
Publishing Topics
/depth_image (sensor_msgs/Image): Depth image of float type.
/seg_image (sensor_msgs/Image): RGB-colorized segmentation images.
/mono3d/bbox (visualization_msgs/MarkerArray): 3D bounding boxes outputs.
/point_cloud (sensor_msgs/PointCloud2): projecting /seg_image with /depth_image into a point cloud.
ONNX Model Interface
MonoDepth ONNX: def forward(self, normalized_images[1, 3, H, W], P[1, 3, 4]):->float[1, 1, H, W]
Segmentation ONNX def forward(self, normalized_images[1, 3, H, W]):->long[1, H, W]
Mono3D ONNX: def forward(self, normalized_images[1, 3, H, W], P[1, 3, 4]):->scores[N], bboxes[N,12], cls_indexes[N]
Metric3D ONNX: def forward(self, unnormalized_images[1, 3, 616, 1064], P[1, 3, 4], P_inv[1, 4, 4], T[1, 4, 4], mask[1, 616, 1064]):->float[1, 1, H, W], float[HW, 6], bool[HW, 6]
Classes definitions are from the visionfactory repo.
Data, Domain
Reshape Scheme
We resize the input image to the length/width, and pad zero on the others. We also modify the camera intrinsic accordingly before feeding into the onnx model. The output will be de-resized to the original shape. Currently published models are all trained with various input images, so the node should work naturally with different image sources.
Expected Data Domain
This is related to the training data of the onnx models. The published models now mainly work on autonomous driving/road scenes. Most of the data for the published segmentation models only use the front-facing camera (Detection / MonoDepth are trained with various camera views).