Awesome
<div align=center><img src=".github/efg_logo.jpeg" width="400" ><div align=left> <br/>An Efficient, Flexible, and General deep learning framework that retains minimal. Users can use EFG to explore any research topics following project templates.
What's New
- 2023.08.22 Code release of ICCV2023 paper: TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses.
- 2023.04.13 Support COCO Panoptic Segmentation with Mask2Former.
- 2023.03.30 Support Pytorch 2.0.
- 2023.03.21 Code release of CVPR2023 Highlight paper: ConQueR: Query Contrast Voxel-DETR for 3D Object Detection.
- 2023.03.21 Code release of EFG codebase, with support for 2D object detection (MS COCO dataset) and 3D object detection (Waymo and nuScenes dataset).
0. Benchmarking
<div align=center><img src=".github/efg_benchmark.png" width="800" ><div align=left>1. Installation
1.1 Prerequisites
- gcc 5 (c++11 or newer)
- python >= 3.6
- cuda >= 10.1
- pytorch >= 1.6
# spconv
spconv_cu11{X} (set X according to your cuda version)
# waymo_open_dataset
## python 3.6
waymo-open-dataset-tf-2-1-0==1.2.0
## python 3.7, 3.8
waymo-open-dataset-tf-2-4-0==1.3.1
1.2 Build from source
git clone https://github.com/poodarchu/EFG.git
cd EFG
pip install -v -e .
# set logging path to save model checkpoints, training logs, etc.
echo "export EFG_CACHE_DIR=/path/to/your/logs/dir" >> ~/.bashrc
2. Data
2.1 Data Preparation - Waymo
# download waymo dataset v1.2.0 (or v1.3.2, etc)
gsutil -m cp -r \
"gs://waymo_open_dataset_v_1_2_0_individual_files/testing" \
"gs://waymo_open_dataset_v_1_2_0_individual_files/training" \
"gs://waymo_open_dataset_v_1_2_0_individual_files/validation" \
.
# extract frames from tfrecord to pkl
CUDA_VISIBLE_DEVICES=-1 python cli/data_preparation/waymo/waymo_converter.py --record_path "/path/to/waymo/training/*.tfrecord" --root_path "/path/to/waymo/train/"
CUDA_VISIBLE_DEVICES=-1 python cli/data_preparation/waymo/waymo_converter.py --record_path "/path/to/waymo/validation/*.tfrecord" --root_path "/path/to/waymo/val/"
# create softlink to datasets
cd /path/to/EFG/datasets; ln -s /path/to/waymo/dataset/root waymo; cd ..
# create data summary and gt database from extracted frames
python cli/data_preparation/waymo/create_data.py --root-path datasets/waymo --split train --nsweeps 1
python cli/data_preparation/waymo/create_data.py --root-path datasets/waymo --split val --nsweeps 1
2.2 Data Preparation - nuScenes
# nuScenes
dataset/nuscenes
├── can_bus
├── lidarseg
├── maps
├── occupancy
│ ├── annotations.json
│ └── gts
├── panoptic
├── samples
├── sweeps
├── v1.0-mini
├── v1.0-test
└── v1.0-trainval
# create softlink to datasets
cd /path/to/EFG/datasets; ln -s /path/to/nuscenes/dataset/root nuscenes; cd ..
# suppose that here we use nuScenes/samples images, put gts and annotations.json under nuScenes/occupancy
python cli/data_preparation/nuscenes/create_data.py --root-path datasets/nuscenes --version v1.0-trainval --nsweeps 11 --occ --seg
3. Get Started
3.1 Training & Evaluation
# cd playground/path/to/experiment/directory
efg_run --num-gpus x # default 1
efg_run --num-gpus x task [train | val | test]
efg_run --num-gpus x --resume
efg_run --num-gpus x dataloader.num_workers 0 # dynamically change options in config.yaml
Models will be evaluated automatically at the end of training. Or,
efg_run --num-gpus x task val
4. Model ZOO
All models are trained and evaluated on 8 x NVIDIA A100 GPUs.
Waymo Open Dataset - 3D Object Detection (val - mAPH/L2)
Methods | Frames | Schedule | VEHICLE | PEDESTRIAN | CYCLIST |
---|---|---|---|---|---|
CenterPoint | 1 | 36 | 66.9/66.4 | 68.2/62.9 | 69.0/67.9 |
CenterPoint | 4 | 36 | 70.0/69.5 | 72.8/69.7 | 72.6/71.8 |
Voxel-DETR | 1 | 6 | 67.6/67.1 | 69.5/63.0 | 69.0/67.8 |
ConQueR | 1 | 6 | 68.7/68.2 | 70.9/64.7 | 71.4/70.1 |
nuScenes - 3D Object Detection (val)
Methods | Schedule | mAP | NDS | Logs |
---|---|---|---|---|
CenterPoint | 20 | 59.0 | 66.7 |
5. Call for contributions
EFG is currently in a relatively preliminary stage, and we still have a lot of work to do, if you are interested in contributing, you can email me at poodarchu@gmail.com.
6. Citation
@article{chen2023trajectoryformer,
title={TrajectoryFormer: 3D Object Tracking Transformer with Predictive Trajectory Hypotheses},
author={Chen, Xuesong and Shi, Shaoshuai and Zhang, Chao and Zhu, Benjin and Wang, Qiang and Cheung, Ka Chun and See, Simon and Li, Hongsheng},
journal={arXiv preprint arXiv:2306.05888},
year={2023}
}
@inproceedings{zhu2023conquer,
title={Conquer: Query contrast voxel-detr for 3d object detection},
author={Zhu, Benjin and Wang, Zhe and Shi, Shaoshuai and Xu, Hang and Hong, Lanqing and Li, Hongsheng},
booktitle={Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition},
pages={9296--9305},
year={2023}
}
@misc{zhu2023efg,
title={EFG: An Efficient, Flexible, and General deep learning framework that retains minimal},
author={EFG Contributors},
howpublished = {\url{https://github.com/poodarchu/efg}},
year={2023}
}