Awesome
RoMe: Towards Large Scale Road Surface Reconstruction via Mesh Representation
Paper,YouTube, Bilibili
Ruohong Mei, Wei Sui, Jiaxin Zhang, Xue Qin, Gang Wang, Tao Peng, Cong Yang
<p align="center"> <img src="assets/rome_structure.png" width="70%"/> </p> RoMe can also be applied for driving drowsiness reduction based on road surface element matching and alarming (Integration of Cabin and Driving). More details: https://fatigueview.github.io/nuScenes
In configs/local_nusc.yaml
- base_dir: Put official nuScenes here, e.g. {base_dir}/v1.0-trainval
- image_dir: Put segmentation results here. We use Mask2Former to segment source images. Folder structure is like {image_dir}/{sweeps/samples}/seg_CAM_FRONT. We provide processed data on google drive, including all semantic images used in this paper. "Scene-1" and "Scene-2" of the paper contain
scene-0063, scene-0064, scene-0200, scene-0283
andscene-0109, scene-0508, scene-0523, scene-0821
respectively.
KITTI Odom
In configs/local_kitti.yaml
- base_dir: Put official kitti odometry dataset here, e.g. {base_dir}/sequences
- image_dir: Put segmentation results here. Also we use Mask2Former and folder structure is like {image_dir}/seg_sequences. We provide processed data on google drive, including all semantic images and poses we used.
Quick Start
Environment
torch==1.10.2+cu111
torchvision==0.11.3+cu111
torchaudio==0.10.2+cu111
pytorch3d==0.6.1
pymeshlab==2021.10
scipy opencv-py thon tqdm wandb python3.8
For wandb usage, please visit here.
Train a scene from nuScenes
-
Modify
configs/local_nusc.yaml
- change wandb configs
- change
base_dir
andimage_dir
according to your folder - change
clip_list
to train one scene or multiple scenes.
-
Modify wandb url and api_key in
run_local.sh
and then runsh run_local.sh
to start training.
Train a scene from KITTI
-
Modify
configs/local_kitti.yaml
- change wandb configs
- change
base_dir
andimage_dir
according to your folder - change
sequence
to choose which sequence to train. - modify
choose_point
andbev_x/y_length
to choose which sub area to train.
-
Modify wandb url and api_key in
run_local.sh
and then runsh run_local.sh
to start training.
Evaluation
- Modify
configs/nusc_eval.yaml
- change
model_path
andpose_path
where your trained models saved. - Make sure other training parameters are same as your configs when training.
- This is a simple evaluation script and only support
batch_size: 1
- change
Future work
- This is the first version of RoMe, and it is hard to reconstruct steep slopes.
- Using SfM(structure from motion) or MVS(multi-view stereo) points and lidar points will give strong supervision.
- Monodepth estimation would be useful like monosdf.
- We are trying to use other mesh render methods like nvdiffrec.
Citation
@misc{mei2023rome,
title={RoMe: Towards Large Scale Road Surface Reconstruction via Mesh Representation},
author={Ruohong Mei and Wei Sui and Jiaxin Zhang and Xue Qin and Gang Wang and Tao Peng and Cong Yang},
year={2023},
eprint={2306.11368},
archivePrefix={arXiv},
primaryClass={cs.CV}
}