Home

Awesome

Visitors License CC BY-NC-SA 4.0 Python 3.6 Packagist Last Commit Maintenance Contributing Ask Me Anything !

Contents

PanoGAN

Cross-View Panorama Image Synthesis
Songsong Wu*, Hao Tang*, Xiaoyuan Jing, Haifeng Zhao, Jianjun Qian, Nicu Sebe, and Yan Yan.<br> (* equal contributions) <br> IEEE Transactions on Multimedia, 2022. <br>

Framework

<img src='./imgs/panogan_architecture.jpg' width=800> <br>

Generator

<img src='./imgs/Generator_Feedback.jpg' width=800> <br>

Discriminator

<img src='./imgs/Discrimination.jpg' width=800> <br>

Comparison Results on the CVUSA Dataset

<img src='./imgs/cvusa_sota_comparison.jpg' width=800> <br>

Comparison Results on the OP Dataset

<img src='./imgs/op_sota_comparison.jpg' width=800> <br>

License

All rights reserved. Licensed under the CC BY-NC-SA 4.0 (Attribution-NonCommercial-ShareAlike 4.0 International)

The code is released for academic research use only. For commercial use, please contact sswuai@gmail.com.

Installation

Clone this repo.

git clone https://github.com/sswuai/PanoGAN.git
cd PanoGAN/

This code requires PyTorch 1.0.0 and python 3.6.9+. Please install dependencies by

pip install -r requirements.txt (for pip users)

or

./scripts/conda_deps.sh (for Conda users)

To reproduce the results reported in the paper, you need to run experiments with at least 1 NVIDIA 2080Ti GPU.

Dataset Preparation

Please download the dataset CVUSA from google drive: Training sample set for ablation study, Test sample set.

Please download the dataset OP from google drive: Training sample set, Test sample set.

Generating Images Using Pretrained Model

PanoGAN model tranined on CVUSA dataset with all training samples

  1. Download the model trained on CVUSA dataset from PanoGAN model on CVUSA.
  2. Unzip the file to checkpoints/
  3. Change several parameters in test_panogan_cvusa.sh.
  4. Run sh test_panogan_cvusa.sh for testing.

PanoGAN model tranined on OP dataset

  1. Download the model trained on OP dataset from PanoGAN model on OP.
  2. Unzip the file to checkpoints/
  3. Change several parameters in test_panogan_op.sh.
  4. Run sh test_panogan_op.sh for testing.

Train and Test New Models

CVUSA dataset

  1. Go to the scripts folder.
  2. Change several parameters in train_panogan_cvusa.sh (or train_panogan_cvusa_ablation_study.sh).
  3. Run sh train_panogan_cvusa.sh (or sh train_panogan_cvusa_ablation_study.sh) for training.
  4. Change several parameters in test_panogan_cvusa.sh.
  5. Run sh test_panogan_cvusa.sh for testing.

OP dataset

  1. Go to the scripts folder.
  2. Change several parameters in train_panogan_op.sh.
  3. Run sh train_panogan_op.sh for training.
  4. Change several parameters in test_panogan_op.sh.
  5. Run sh test_panogan_op.sh for testing.

Evaluation

We adopt Prediction Accuracy, Inception Score, KL Score, SSIM, PSNR, and SD for evaluation of all the involved methods. Please refer to Evaluation for more details.

Acknowledgments

This source code is inspired by both Pix2Pix and SelectionGAN.

Related Projects

CrossMLP | SelectionGAN | LGGAN | DAGAN | DPGAN | Guided-I2I-Translation-Papers

Citation

@article{wu2022cross,
  title={Cross-View Panorama Image Synthesis},
  author={Wu, Songsong and Tang, Hao and Jing, Xiao-Yuan and Zhao, Haifeng and Qian, Jianjun and Sebe, Nicu and Yan, Yan},
  journal={IEEE Transactions on Multimedia (TMM)},
  year={2022}
}

Contributions

If you have any questions/comments/bug reports, feel free to open a github issue or pull a request or e-mail to the author Songsong Wu (sswuai@gmail.com).

Collaborations

I'm always interested in meeting new people and hearing about potential collaborations. If you'd like to work together or get in contact with me, please email bjdxtanghao@gmail.com. Some of our projects are listed here.