Home

Awesome

FIERY

This is the PyTorch implementation for inference and training of the future prediction bird's-eye view network as described in:

FIERY: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras

Anthony Hu, Zak Murez, Nikhil Mohan, Sofía Dudas, Jeffrey Hawke, ‪Vijay Badrinarayanan, Roberto Cipolla and Alex Kendall

ICCV 2021 (Oral)<br/> Blog post

<p align="center"> <img src="https://github.com/wayveai/fiery/releases/download/v1.0/predictions.gif" alt="FIERY future prediction"> <br/> <sub><em>Multimodal future predictions by our bird’s-eye view network.<br/> Top two rows: RGB camera inputs. The predicted future trajectories and segmentations are projected to the ground plane in the images.<br/> Bottom row: future instance prediction in bird’s-eye view in a 100m×100m capture size around the ego-vehicle, which is indicated by a black rectangle in the center. </em></sub> </p>

If you find our work useful, please consider citing:

@inproceedings{fiery2021,
  title     = {{FIERY}: Future Instance Segmentation in Bird's-Eye view from Surround Monocular Cameras},
  author    = {Anthony Hu and Zak Murez and Nikhil Mohan and Sofía Dudas and 
               Jeffrey Hawke and Vijay Badrinarayanan and Roberto Cipolla and Alex Kendall},
  booktitle = {Proceedings of the International Conference on Computer Vision ({ICCV})},
  year = {2021}
}

⚙ Setup

🏄 Prediction

Visualisation

In a colab notebook: Open In Colab

Or locally:

Evaluation

🔥 Pre-trained models

All the configs are in the folder fiery/configs

Config and weightsDatasetPast contextFuture horizonBEV sizeIoUVPQ
baseline.ymlNuScenes1.0s2.0s100mx100m (50cm res.)36.729.9
lyft/baseline.ymlLyft0.8s2.0s100mx100m (50cm res.)36.329.2
literature/static_pon_setting.ymlNuScenes0.0s0.0s100mx50m (25cm res.)37.7-
literature/pon_setting.ymlNuScenes1.0s0.0s100mx50m (25cm res.)39.9-
literature/static_lss_setting.ymlNuScenes0.0s0.0s100mx100m (50cm res.)35.8-
literature/lift_splat_setting.ymlNuScenes1.0s0.0s100mx100m (50cm res.)38.2-
literature/fishing_setting.ymlNuScenes1.0s2.0s32.0mx19.2m (10cm res.)57.6-

🏊 Training

To train the model from scratch on NuScenes:

This will train the model on 4 GPUs, each with a batch of size 3. To train on single GPU add the flag GPUS 1, and to change the batch size use the flag BATCHSIZE ${DESIRED_BATCHSIZE}.

🙌 Credits

Big thanks to Giulio D'Ippolito (@gdippolito) for the technical help on the gpu servers, Piotr Sokólski (@pyetras) for the panoptic metric implementation, and to Hannes Liik (@hannesliik) for the awesome future trajectory visualisation on the ground plane.