Awesome

<p align="center">FlowLens: Seeing Beyond the FoV via Flow-guided Clip-Recurrent Transformer

Update

• 2022.11.19 Init repository.
• 2022.11.21 Release the arXiv version with supplementary materials.
• 2023.04.04 :fire: Our code is publicly available.
• 2023.04.04 :fire: Release pretrained models.
• 2023.04.04 :fire: Release KITTI360-EX dataset.

TODO List

• Code release.
• KITTI360-EX release.
• Towards higher performance with extra small costs.

Abstract

Limited by hardware cost and system size, camera's Field-of-View (FoV) is not always satisfactory. However, from a spatio-temporal perspective, information beyond the camera’s physical FoV is off-the-shelf and can actually be obtained ''for free'' from past video streams. In this paper, we propose a novel task termed Beyond-FoV Estimation, aiming to exploit past visual cues and bidirectional break through the physical FoV of a camera. We put forward a FlowLens architecture to expand the FoV by achieving feature propagation explicitly by optical flow and implicitly by a novel clip-recurrent transformer, which has two appealing features: 1) FlowLens comprises a newly proposed Clip-Recurrent Hub with 3D-Decoupled Cross Attention (DDCA) to progressively process global information accumulated in the temporal dimension. 2) A multi-branch Mix Fusion Feed Forward Network (MixF3N) is integrated to enhance the spatially-precise flow of local features. To foster training and evaluation, we establish KITTI360-EX, a dataset for outer- and inner FoV expansion. Extensive experiments on both video inpainting and beyond-FoV estimation tasks show that FlowLens achieves state-of-the-art performance.

Demos

Dependencies

This repo has been tested in the following environment:

torch == 1.10.2
cuda == 11.3
mmflow == 0.5.2

Usage

To train FlowLens(-S), use:

python train.py --config configs/KITTI360EX-I_FlowLens_small_re.json

To eval on KITTI360-EX, run:

python evaluate.py \
--model flowlens \
--cfg_path configs/KITTI360EX-I_FlowLens_small_re.json \
--ckpt release_model/FlowLens-S_re_Out_500000.pth --fov fov5

Turn on --reverse for test time augmentation (TTA).

Trun on --save_results to save your output.

Pretrained Models

The pretrained model can be found there:

https://share.weiyun.com/6G6QEdaa

KITTI360-EX for Beyond-FoV Estimation

The preprocessed KITTI360-EX can be downloaded from here:

https://share.weiyun.com/BReRdDiP

Results

KITTI360EX-InnerSphere

KITTI360EX-OuterPinhole

Note that when using the ''Video Inpainting'' logic for output, the model is allowed to use more reference frames from the future, and each local frame is estimated at least twice, thus higher accuracy can be obtained while result in slower inference speed, and it is not realistic for real-world deployment.

Citation

If you find our paper or repo useful, please consider citing our paper:

@article{shi2022flowlens,
title={FlowLens: Seeing Beyond the FoV via Flow-guided Clip-Recurrent Transformer},
author={Shi, Hao and Jiang, Qi and Yang, Kailun and Yin, Xiaoting and Wang, Kaiwei},
journal={arXiv preprint arXiv:2211.11293},
year={2022}
}

Acknowledgement

This project would not have been possible without the following outstanding repositories:

STTN, MMFlow

Devs

Hao Shi

Contact

Feel free to contact me if you have additional questions or have interests in collaboration. Please drop me an email at haoshi@zju.edu.cn. =)