Home

Awesome

<h1>WiDEVIEW: An UltraWideBand and Vision Dataset for Deciphering Pedestrian-Vehicle Interactions</h1> <p align="center"> <a href="https://www.tamu.edu/"><img src="images/tamu_logo.png" alt="Texas A&M University" height="90px" width="450px"></a>&emsp;&emsp;&emsp;&emsp;<a href="https://www.hyundai.com/worldwide/en/"><img src="images/Hyundai_logo.png" alt="Hyundai Motor Company" height="90px" width="270px"></a></p> <p align="center"> Jia Huang<sup>1</sup>, Alvika Gautam<sup>1</sup>, Junghun Choi<sup>2</sup> and Srikanth Saripalli<sup>1</sup><br> 1. <a href="https://www.tamu.edu/">Texas A&M University; </a>&emsp;2. <a href="https://www.hyundai.com/worldwide/en/">Hyundai Motor Company</a><br> </p>

Overview

Recording Platform

Sensor Setup

Sensor Setup Illustration

Folder structure

<pre> UWB_Dataset ├── UWB-gcart-camera-calibration --directory for the three front cameras calibration by using Kalibr ├── benchmarks --directory for object tracking benchmark by using Yolov5 and DeepSORT ├── decawave_ros --directory for TREK1000 UWB ros package containing ros node and messages ├── lidar2camera --directory for lidar to middle camera calibration by using SensorsCalibration toolbox ├── sensor_syn.py --script for sensor data approximate synchronization for future sensor fusion </pre>

Annotated Data:

The dataset contains 30,936 frames which come from the front three cameras with 10,312 frames each. Every 5-th frame was annotated manually and carefully in Computer Vision Annotation Tool (CVAT) for the front middle camera frames while the ones in between are interpolated automatically without adjustment. There are a total of 8,368 bounding boxes manually annotated for pedestrians and cyclists along with six tag IDs for those who carried tags, and 33,239 bounding boxes are interpolated in CVAT. These annotations are exported in MOT 1.1 format for the following multi-object tracking task evaluation for images. The annotations for the left and right cameras frames are in progress and will be updated on the Github gradually.

Annotations and Annotation tasks (include original videos) Download:

Annotations include raw images in the img1 folder, annotation tasks include the raw videos named output.mp4 used for annotation in CVAT.

Annotation folder structure

<pre> rosbag_filename(subfolder_middle_annotation).zip ├── gt ├── gt.txt └── labels.txt ├── img1 ├── 000001.PNG ├── 000002.PNG └── ...... </pre>

Example of rosbag_filename(subfolder_middle_annotation).zip : 2023-02-04-16-04-35_0(1_middle_annotation)

Annotation task folder structure

<pre> rosbag_filename(subfolder_middle_task).zip ├── annotations.json ├── task.json ├── data ├── index.json ├── manifest.jsonl └── output.mp4 </pre>

Example of rosbag_filename(subfolder_middle_task).zip : 2023-02-04-16-04-35_0(1_middle_task)

Sequence 1_1: 2023-02-04-16-04-35_0.bag: (Annotation) (Task)

Sequence 1_2: 2023-02-04-16-04-35_0.bag: (Annotation) (Task)

Sequence 2: 2023-02-10-15-23-44_0.bag: (Annotation) (Task)

Sequence 3_1: 2023-02-10-15-26-14_1.bag:(Annotation) (Task)

Sequence 3_2: 2023-02-10-15-26-14_1.bag:(Annotation) (Task)

Sequence 4: 2023-02-10-15-28-44_2.bag: (Annotation) (Task)

Sequence 5_1: 2023-02-10-15-35-56_0.bag: (Annotation) (Task)

Sequence 5_2: 2023-02-10-15-35-56_0.bag: (Annotation) (Task)

Sequence 6: 2023-02-10-15-38-30_0.bag: (Annotation) (Task)

Sequence 7: 2023-02-12-13-59-09_0.bag: (Annotation) (Task)

Sequence 8: 2023-02-12-14-13-00_0.bag: (Annotation) (Task)

Sequence 9: 2023-02-12-14-14-40_0.bag: (Annotation) (Task)

Sequence 10:2023-02-12-14-20-14_0.bag: (Annotation) (Task)

Sequence 11:2023-02-12-14-24-09_0.bag: (Annotation) (Task)

Sequence 12:2023-02-12-14-25-15_0.bag: (Annotation) (Task)

Sequence 13:2023-02-12-14-31-29_0.bag: (Annotation) (Task)

Sequence 14:2023-02-17-15-40-11_0.bag: (Annotation) (Task)

Sequence 15:2023-02-17-15-42-27_0.bag: (Annotation) (Task)

Sequence 16:2023-02-17-15-44-23_0.bag: (Annotation) (Task)

Sequence 17:2023-02-17-16-00-23_0.bag: (Annotation) (Task)

Sequence 18_1:2023-02-17-16-53-27_0.bag: (Annotation) (Task)

Sequence 18_2:2023-02-17-16-53-27_0.bag: (Annotation) (Task)

Sequence 19:2023-02-17-16-54-38_0.bag: (Annotation) (Task)

Sequence 20:2023-02-17-17-01-15_0.bag: (Annotation) (Task)

Sequence 21:2023-02-17-17-03-49_0.bag: (Annotation) (Task)

Calibration:

Multi-cameras Intrinsic and Extrinsic Calibration

The sensor suite cameras (left, middle and right on the autonomous shuttle roof rack) have a 30$^{\circ}$ overlapping field of view between adjacent cameras. The intrinsic and extrinsic camera parameters are estimated by the multiple camera calibration tool in the Kalibr toolbox. A 6 x 6 aprilgrid target with spacing of 0.03m is used. We utilize a pinhole projection model for our cameras, where a three-dimensional scene is projected onto an image plane through a perspective transform. The calibration details can be found in the folder UWB-gcart-camera-calibration.

Lidar to Middle Camera Calibration

To prepare data for LiDAR and middle camera calibration, the data across both sensors is first matched by finding the pairs with the closest time stamps between two sensors. Then the point cloud and image pairs are calibrated in SensorsCalibration toolbox to facilitate the sensor fusion process. The calibration details can be found in the folder lidar2camera.

Benchmarks

2D Object Detection and Tracking

CaseHOTAMOTAMOTPIDF1FPFNID Sw.RecallPrecisionDetsGT DetsIDsGT IDs
141.57742.04672.49654.95891441100672969.45173.2363416536027491231
245.87344.20478.17357.2931607203839871.87476.4268157246425231

Benchmark Reproduction

To reproduce the results, please refer to here

ROS Bag Raw Data

Data included in raw ROS bagfiles:

Topic NameMessage TpyeMessage Descriptison
/Decawavedecawave_ros/uwb_distanceDistances between Anchors and Tags
/left/image_rawsensor_msgs/ImageImages from the left camera
/middle/image_rawsensor_msgs/ImageImages from the middele camera
/right/image_rawsensor_msgs/ImageImages from the right camera
/vectornav/fixsensor_msgs/NavSatFix
/vectornav/gpsvn300/gpsGPS data from VectorNav-VN300
/vectornav/imuvn300/sensorsImu data from VectorNav-VN300
/vectornav/insvn300/insINS data from VectorNav-VN300
/velodyne_pointssensor_msgs/PointCloud2PointCloud produced by the Velodyne Lidar

ROS Bag Download

The ROS bag information can be found on the google drive link. The following are the google drive links for the ROS Bag files.

Sequence 1: 2023-02-04-16-04-35_0.bag: (19.91GB)

Sequence 2: 2023-02-10-15-23-44_0.bag: (19.94GB)

Sequence 3: 2023-02-10-15-26-14_1.bag: (20.07GB)

Sequence 4: 2023-02-10-15-28-44_2.bag: (18.42GB)

Sequence 5: 2023-02-10-15-35-56_0.bag: (7.29GB)

Sequence 6: 2023-02-10-15-38-30_0.bag: (8.04GB)

Sequence 7: 2023-02-12-13-59-09_0.bag: (6.66GB)

Sequence 8: 2023-02-12-14-13-00_0.bag: (5.97GB)

Sequence 9: 2023-02-12-14-14-40_0.bag: (15.71GB)

Sequence 10:2023-02-12-14-20-14_0.bag: (5.69GB)

Sequence 11:2023-02-12-14-24-09_0.bag: (9.16GB)

Sequence 12:2023-02-12-14-25-15_0.bag: (8.66GB)

Sequence 13:2023-02-12-14-31-29_0.bag: (5.59GB)

Sequence 14:2023-02-17-15-40-11_0.bag: (4.26GB)

Sequence 15:2023-02-17-15-42-27_0.bag: (7.53GB)

Sequence 16:2023-02-17-15-44-23_0.bag: (16.47GB)

Sequence 17:2023-02-17-16-00-23_0.bag: (7.04GB)

Sequence 18:2023-02-17-16-53-27_0.bag: (3.93GB)

Sequence 19:2023-02-17-16-54-38_0.bag: (6.57GB)

Sequence 20:2023-02-17-17-01-15_0.bag: (6.57GB)

Sequence 21:2023-02-17-17-03-49_0.bag: (4.43GB)

Working in progress

Citation

@misc{huang2023wideview,
      title={WiDEVIEW: An UltraWideBand and Vision Dataset for Deciphering Pedestrian-Vehicle Interactions}, 
      author={Jia Huang and Alvika Gautam and Junghun Choi and Srikanth Saripalli},
      year={2023},
      eprint={2309.16057},
      archivePrefix={arXiv},
      primaryClass={cs.RO}
}

License

All datasets and code on this page are copyright by us and published under the GNU General Public License v2.0.