Home

Awesome

中文版本

CeleX5-ROS

The ROS packages for CeleX™ CeleX5-MIPI Dynamic Vision Sensor.

This repository provides several functional packages and examples of CeleX5-MIPI Event-based Camera under ROS, including:

The code in this repository mainly follows the Google C++ Style.

Introduction

Overview

Build and Run

Known Issues

Q&A

Introduction

<img src="assets/242069421.jpg" height="300" alt="CeleX5-MIPI"/>

CeleX™ is a family of smart image sensor specially designed for machine vision. Each pixel in CeleX™ sensor can individually monitor the relative change in light intensity and report an event if it reaches a certain threshold. Asynchronous row and column arbitration circuits process the pixel requests and make sure only one request is granted at a time in fairly manner when they received multiple simultaneous requests. The output of the sensor is not a frame, but a stream of asynchronous digital events. The speed of the sensor is not limited by any traditional concept such as exposure time and frame rate. It can detect fast motion which is traditionally captured by expensive, high speed cameras running at thousands of frames per second, but with drastic reduced amount of data.

CeleX-5 is a multifunctional smart image sensor with 1Mega-pixels(1280*800) and some additional features integrating on-chip (such as on-chip optical-flow). The sensor supports several different output formats: pure binary address-events, address-events with either pixel intensity information or timing information. In addition, the readout scheme of the sensor could either be asynchronous data stream or synchronous full frames. Different combinations of the output format and readout scheme lead to great flexibility of this sensor, which supports 6 separate operation modes in total (but one of them doesn't provide corresponding interface in SDK). To further meet the requirements of different applications, the sensor could also be configured into a loop-mode, in which it could automatically switch among three separate modes.

ROS is a very popular experimental platform, which provides rich development interfaces and resources. But in the official open source repository of CeleX ™, the examples under ROS are very incomplete:

  1. The official SDK version has been updated to v2.0, however the version supported by its ROS-Sample remains at v1.6.
  2. The official ROS-Sample is just a simple example. It only outputs an image in a working mode. It does not provide a comprehensive and convenient parameter configuration function and interface.

Therefore, in order to make more convenient use of the resources provided by ROS, I developed the content in this repository.

Overview

This repository provides several functional packages and examples of CeleX5-MIPI Event-based Camera under ROS, including:

It is recommended that you read the concept section of the CeleX_SDK_Getting_Started_Guide and CeleX5_SDK_Reference documentation provided by CeleX carefully before using it, and have a general understanding of basic terminology.

Build and Run

  1. Build

    mkdir -p ~/celex_ws/src
cd ~/celex_ws/src
git clone git@github.com:kehanXue/CeleX5-ROS.git
git submodule update --init --recursive
# Or with http: `git clone https://github.com/kehanXue/CeleX5-ROS.git`
cd ..
rosdep install -y --from-paths src --ignore-src --rosdistro $ROS_DISTRO
catkin_make # Or use `catkin build`

    If the building process reports the errors of which dependent libraries are missing, install them.

  2. Create the udev rules to solve the permission denied problem when read/write from a usb device:

    cd ~/celex_ws/src/CeleX5-ROS/celex5_ros/scripts
chmod +x ./create_udev_rules.sh
./create_udev_rules.sh

  3. Run celex5_ros. More details

    First plug the sensor into your computer.

    source ~/celex_ws/devel/setup.bash # Or source setup.zsh when you use zsh
roslaunch celex5_ros celex5_ros_node.launch
# In a new Terminal
rosrun rqt_reconfigure rqt_reconfigure # Open rqt_reconfigure to config

    If the program runs without error, you have run successfully. Use rostopic list to see published topics, and use rivz or image_view to subscribe to corresponding image topics to see published images. Whether to publish data to an image topic is determined by both the whether display parameter configuration and the sensor current working mode.

    $ rostopic list
/celex5_mipi/display/accumulated_img/camera_info
/celex5_mipi/display/accumulated_img/raw_image
/celex5_mipi/display/binary_img/camera_info
/celex5_mipi/display/binary_img/raw_image
/celex5_mipi/display/count_img/camera_info
/celex5_mipi/display/count_img/raw_image
/celex5_mipi/display/denoised_binary_img/camera_info
/celex5_mipi/display/denoised_binary_img/raw_image
/celex5_mipi/display/full_frame_img/camera_info
/celex5_mipi/display/full_frame_img/raw_image
/celex5_mipi/display/gray_img/camera_info
/celex5_mipi/display/gray_img/raw_image
/celex5_mipi/display/in_pixel_img/camera_info
/celex5_mipi/display/in_pixel_img/raw_image
/celex5_mipi/display/optical_flow_direction_img/camera_info
/celex5_mipi/display/optical_flow_direction_img/raw_image
/celex5_mipi/display/optical_flow_img/camera_info
/celex5_mipi/display/optical_flow_img/raw_image
/celex5_mipi/display/optical_flow_speed_img/camera_info
/celex5_mipi/display/optical_flow_speed_img/raw_image
/celex5_mipi/display/parameter_descriptions
/celex5_mipi/display/parameter_updates
/celex5_mipi/display/superimposed_img/camera_info
/celex5_mipi/display/superimposed_img/raw_image
/celex5_mipi/events
/celex5_mipi/imu_data
/celex5_mipi/polarity_img/camera_info
/celex5_mipi/polarity_img/raw_image
/celex5_mipi/sensor/parameter_descriptions
/celex5_mipi/sensor/parameter_updates
/rosout
/rosout_agg

    Topic of raw events data: /celex5_mipi/events

    Topic of imu data: /celex5_mipi/imu_data . Note that the ROS message type of imu topic is different from the standard IMU message type (sensor_msgs/Imu) in ROS due to the way of obtaining the raw imu data of CeleX5.

    View an image via image_view:

    rosrun image_view image_view image:=/celex5_mipi/display/binary_img/raw_image

  4. Run celex5_calibration. More details

    A series of methods and tools for camera parameter calibration based on events data are provided.

    • Pattern generator (blinking chessboard). More details

      Run:

      rosrun celex5_ros pattern_generator_node
# In a new Terminal
rosrun rqt_reconfigure rqt_reconfigure #  Open rqt_reconfigure to config

    • Intrinsics calibration based on camera_calibration toolkit in ROS. (if you want to calibrate the external parameters of other cameras, this tool in ROS only supports the same resolution...)

      Install:

      sudo apt install ros-$ROS_DISTRO-camera-calibration

      The detailed tutorials: link

    • Based on Kalibr toolkit, which supports both intrinsics calibration and extrinsics calibration with another (traditional or event-based) cameras, in different resolutions. Event data based method.

      The detailed tutorials: link

      Also provides a series of tools to collect calibration data of CeleX5-MIPI Camera.

    • Time stamp calibration with another camera.

      TODO, existing problems. link

Known Issues

celex5_ros

celex5_calibration

Q&A