Home

Awesome

CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoor Object Detection from Multi-view Images

This repository is an official implementation of CN-RMA.

Results

datasetmAP@0.25mAP@0.5config
ScanNet58.636.8config
ARKit67.656.5config

Configuration, data processing and running the entire project is complicated. We provide all the detection results, visualization results and checkpoints of the validation set of the two datasets at Tsinghua Cloud. Since our preparing and training procedure is complicated, you can directly download our results for ScanNet and ARKitScenes, or directly use our pre-trained weights for ScanNet and ARKitScenes to validate.

Prepare

How to Run

To evaluate our method on ScanNet, you can download the final checkpoint, set the 'work_dir' of projects/configs/mvsdetection/ray_marching_scannet.py to your desired path, and run:

bash dist_test.sh projects/configs/mvsdetection/ray_marching_scannet.py {scannet_best.pth} 4

Similarly, to evaluate on ARKitScenes, you should download the final checkpoint, set the 'work_dir' of projects/configs/mvsdetection/ray_marching_arkit.py to your desired path, and run:

bash dist_test.sh projects/configs/mvsdetection/ray_marching_arkit.py {arkit_best.pth} 4

After this, you should do nms post-processing to the data by running:

python ./post_process/nms_bbox.py --result_path {your_work_dir}/results

The pc_det_nms do not always work very well, if it fails, just run it again and again....

You can then evaluate the results by running

./post_process/evaluate_bbox.py --dataset {arkit/scannet} --data_path {your_arkit_or_scannet_source_path} --result_path {your_work_dir}/results

And you can visualize the results by running

./post_process/visualize_results.py --dataset {arkit/scannet} --data_path {your_arkit_or_scannet_source_path} --save_path {your_work_dir}/results

if the nms fails, you can discover many bounding boxes very close to each other on the visualized results, then you can run the nms again.

Training the network from scratch is complicated. If you want to train the network from scratch, please follow train_val.md

Citation

If you find this project useful for your research, please consider citing:

@InProceedings{Shen_2024_CVPR,
    author    = {Shen, Guanlin and Huang, Jingwei and Hu, Zhihua and Wang, Bin},
    title     = {CN-RMA: Combined Network with Ray Marching Aggregation for 3D Indoor Object Detection from Multi-view Images},
    booktitle = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    month     = {June},
    year      = {2024},
    pages     = {21326-21335}
}

Contact

If you have any questions, feel free to open an issue or contact us at shenguanlin1999@163.com