Awesome
<div align="center"> <h1>PreSight</h1> <h3>[ECCV 2024] PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors</h3> <img src="./resources/main_teaser_detail.jpg" width="1050px"> </div>Introduction
This repository is an official implementation of PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors.
News
-
[2024/07/20]: :tada: We have released the code of PreSight!
-
[2024/07/09]: :confetti_ball: Our paper has been accepted by the The 18th European Conference on Computer Vision (ECCV 2024)! Our code will be release this month. Stay tuned!
Main Results
Online HD Mapping
Model | Metric | w. Prior | Ped Crossing | Divider | Boundary | All | Runtime (FPS) |
---|---|---|---|---|---|---|---|
StreamMapNet | AP | × | 10.19 | 11.26 | 11.87 | 11.10 | 22.4 |
StreamMapNet | AP | ✓ | 21.11 | 23.73 | 32.31 | 25.72 (+14.62) | 21.9 |
MapTR | AP | × | 4.97 | 8.20 | 9.83 | 7.67 | 25.2 |
MapTR | AP | ✓ | 16.18 | 19.04 | 34.14 | 23.12 (+15.45) | 23.2 |
BEVFormer | IoU | × | 14.90 | 29.88 | 32.74 | 25.84 | 15.5 |
BEVFormer | IoU | ✓ | 16.37 | 34.82 | 51.66 | 34.28 (+8.44) | 14.3 |
Occupancy
Method | w. Priors | mIoU | Dynamic | Static | others | barrier | bicycle | bus | car | constr. vehicle | motorcycle | pedestrian | traffic cone | truck | drive surface | other flat | sidewalk | terrain | manmade | vegetation | Runtime (FPS) |
---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|---|
BEVDet | × | 29.3 | 24.4 | 38.2 | 1.5 | 42.4 | 11.0 | 43.0 | 47.1 | 19.1 | 23.3 | 23.4 | 19.5 | 37.8 | 72.9 | 11.6 | 30.9 | 48.6 | 32.7 | 32.5 | 5.1 |
BEVDet | ✓ | 33.7 (+4.4) | 24.4 | 50.5 (+12.3) | 1.2 | 40.1 | 14.8 | 42.1 | 48.3 | 15.7 | 26.4 | 24.4 | 18.7 | 37.2 | 81.8 | 15.2 | 40.3 | 60.5 | 50.4 | 54.9 | 4.9 |
FB-Occ | × | 30.0 | 25.1 | 39.2 | 9.2 | 37.2 | 21.8 | 41.6 | 43.4 | 15.8 | 27.3 | 25.4 | 23.8 | 30.3 | 74.7 | 17.3 | 33.0 | 50.6 | 28.2 | 31.1 | 9.1 |
FB-Occ | ✓ | 34.3 (+4.3) | 25.4 | 50.7 (+11.5) | 9.3 | 38.3 | 21.0 | 40.3 | 45.0 | 15.9 | 29.9 | 26.0 | 23.8 | 30.2 | 82.3 | 18.5 | 39.1 | 61.2 | 48.0 | 54.7 | 8.6 |
Getting Started
To get started, please follow the instructions below step-by-step.
Pretrained Weights
Extracted Priors
Boston-Seaport | Singapore-Onenorth | Singapore-Queenstown | Singapore-Hollandvillage | |
---|---|---|---|---|
Google Drive | Download | Download | Download | Download |
Perception Models
Vectorized Online Mapping | Occupancy Prediction | |
---|---|---|
Google Drive | Download | Download |
TODO
- Add scripts to inference per-image monocular depth using Depth-Anything to enable training NeRFs with monocular-depth loss. Monocular-depth loss improves visualization quality but do not improve downstream perception metrics.
Acknowledgement
This project builds upon the outstanding work of several open-source projects. We extend our sincere thanks to the following codebases:
Citation
If you find our work useful in your research, please consider citing:
@article{yuan2024presight,
title={PreSight: Enhancing Autonomous Vehicle Perception with City-Scale NeRF Priors},
author={Yuan, Tianyuan and Mao, Yucheng and Yang, Jiawei and Liu, Yicheng and Wang, Yue and Zhao, Hang},
journal={arXiv preprint arXiv:2403.09079},
year={2024}
}