Home

Awesome

Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation

teaser

Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation (CVPR 2022)<br/> Naofumi Akimoto, Yuhi Matsuo, Yoshimitsu Aoki<br/>

arXiv | BibTeX | Project Page | Supp Video

Requirements

A suitable conda environment named omnidreamer can be created and activated with:

conda env create -f environment.yaml
conda activate omnidreamer

Trained models

Please send us an email. We will send you the URL for downloading. You may distribute the trained models to others, but please do not reveal the URL.

Running trained models

Inference examples

Sampling for a comparison against 360IC

CUDA_VISIBLE_DEVICES=0 python sampling.py \
--config_path logs/2021-07-27T05-57-41_sun360_basic_transformer/configs/2021-07-27T05-57-41-project.yaml \
--ckpt_path logs/2021-07-27T05-57-41_sun360_basic_transformer/checkpoints/last.ckpt \
--config_path_2 logs/2021-07-27T10-49-57_sun360_refine_net/configs/2021-07-27T10-49-57-project.yaml \
--ckpt_path_2 logs/2021-07-27T10-49-57_sun360_refine_net/checkpoints/last.ckpt \
--outdir outputs/test

Sampling for a comparison against SIG-SS

CUDA_VISIBLE_DEVICES=0 python sampling.py \
--config_path logs/2021-07-27T05-57-41_sun360_basic_transformer/configs/2021-07-27T05-57-41-project.yaml \
--ckpt_path logs/2021-07-27T05-57-41_sun360_basic_transformer/checkpoints/last.ckpt \
--config_path_2 logs/2021-07-27T10-49-57_sun360_refine_net/configs/2021-07-27T10-49-57-project.yaml \
--ckpt_path_2 logs/2021-07-27T10-49-57_sun360_refine_net/checkpoints/last.ckpt \
--mask_path assets/90binarymask.png \
--outdir outputs/test

Sampling for a comparison against EnvMapNet

CUDA_VISIBLE_DEVICES=0 python sampling.py \
--config_path logs/2021-08-12T03-27-04_sun360_basic_transformer/configs/2021-08-12T03-27-04-project.yaml \
--ckpt_path logs/2021-08-12T03-27-04_sun360_basic_transformer/checkpoints/last.ckpt \
--config_path_2 logs/2021-08-12T03-42-53_sun360_refine_net/configs/2021-08-12T03-42-53-project.yaml \
--ckpt_path_2 logs/2021-08-12T03-42-53_sun360_refine_net/checkpoints/last.ckpt \
--mask_path assets/90binarymask.png \
--outdir outputs/test

Training

We train the four networks separately (VQGAN_1, VQGAN_2, Transformer, and AdjustmentNet).
Order of training

Sample Commands

CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 python main.py --base configs/sun360_comp_vqgan.yaml -t True --gpus 0,1
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 python main.py --base configs/sun360_comp_vqgan.yaml -t True --gpus 0,1
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 python main.py --base configs/sun360_basic_transformer.yaml -t True --gpus 0,1
CUDA_DEVICE_ORDER=PCI_BUS_ID CUDA_VISIBLE_DEVICES=0,1 python main.py --base configs/sun360_refine_net.yaml -t True --gpus 0,1

Development environment

License

This repo is built on top of VQGAN. See the license.

BibTeX

@inproceedings{akimoto2022diverse,
    author    = {Akimoto, Naofumi and Matsuo, Yuhi and Aoki, Yoshimitsu},
    title     = {Diverse Plausible 360-Degree Image Outpainting for Efficient 3DCG Background Creation},
    booktitle   = {Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
    year      = {2022},
}