Awesome

Transformers for Panoramic Semantic Segmentation (Trans4PASS & Trans4PASS+)

Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation, CVPR 2022, [PDF].

Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation, arxiv preprint, [PDF].

Update

• [08/2022], a new panoramic semantic segmentation benchmark SynPASS is released.

• [08/2022], Trans4PASS+ is released.

Environments

conda create -n trans4pass python=3.8
conda activate trans4pass
cd ~/path/to/trans4pass 
conda install pytorch==1.8.0 torchvision==0.9.0 torchaudio==0.8.0 cudatoolkit=11.1 -c pytorch -c conda-forge
pip install mmcv-full==1.3.9 -f https://download.openmmlab.com/mmcv/dist/cu111/torch1.8.0/index.html
pip install -r requirements.txt
python setup.py develop --user
# Optional: install apex follow: https://github.com/NVIDIA/apex

SynPASS dataset

SynPASS dataset contains 9080 panoramic images (1024x2048) and 22 categories.

The scenes include cloudy, foggy, rainy, sunny, and day-/night-time conditions.

The SynPASS dataset is now availabel at GoogleDrive.

SynPASS statistic information:

Data Preparation

Prepare datasets:

• Cityscapes
• DensePASS
• SynPASS
• Stanford2D3D
• Structured3D

datasets/
├── cityscapes
│   ├── gtFine
│   └── leftImg8bit
├── Stanford2D3D
│   ├── area_1
│   ├── area_2
│   ├── area_3
│   ├── area_4
│   ├── area_5a
│   ├── area_5b
│   └── area_6
├── Structured3D
│   ├── scene_00000
│   ├── ...
│   └── scene_00199
├── SynPASS
│   ├── img
│   │   ├── cloud
│   │   ├── fog
│   │   ├── rain
│   │   └── sun
│   └── semantic
│       ├── cloud
│       ├── fog
│       ├── rain
│       └── sun
├── DensePASS
│   ├── gtFine
│   └── leftImg8bit

Prepare pretrained weights, which can be found in the public repository of SegFormer.

pretrained/
├── mit_b1.pth
└── mit_b2.pth

Network Define

The code of Network pipeline is in segmentron/models/trans4pass.py.

The code of backbone is in segmentron/models/backbones/trans4pass.py.

The code of DMLP decoder is in segmentron/modules/dmlp.py.

The code of DMLPv2 decoder is in segmentron/modules/dmlp2.py.

Train

For example, to use 4 1080Ti GPUs to run the experiments:

# Trans4PASS
python -m torch.distributed.launch --nproc_per_node=4 tools/train_cs.py --config-file configs/cityscapes/trans4pass_tiny_512x512.yaml
python -m torch.distributed.launch --nproc_per_node=4 tools/train_s2d3d.py --config-file configs/stanford2d3d/trans4pass_tiny_1080x1080.yaml
# Trans4PASS+, please modify the version at segmentron/models/trans4pass.py
python -m torch.distributed.launch --nproc_per_node=4 tools/train_cs.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml
python -m torch.distributed.launch --nproc_per_node=4 tools/train_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
python -m torch.distributed.launch --nproc_per_node=4 tools/train_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml

Test

Download the models from GoogleDrive and save in ./workdirs folder as:

./workdirs
├── cityscapes
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
├── cityscapes13
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
├── stanford2d3d
│   ├── trans4pass_plus_small_1080x1080
│   ├── trans4pass_plus_tiny_1080x1080
│   ├── trans4pass_small_1080x1080
│   └── trans4pass_tiny_1080x1080
├── stanford2d3d8
│   ├── trans4pass_plus_small_1080x1080
│   ├── trans4pass_plus_tiny_1080x1080
│   ├── trans4pass_small_1080x1080
│   └── trans4pass_tiny_1080x1080
├── stanford2d3d_pan
│   ├── trans4pass_plus_small_1080x1080
│   ├── trans4pass_small_1080x1080
│   └── trans4pass_tiny_1080x1080
├── structured3d8
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
├── synpass
│   ├── trans4pass_plus_small_512x512
│   ├── trans4pass_plus_tiny_512x512
│   ├── trans4pass_small_512x512
│   └── trans4pass_tiny_512x512
└── synpass13
    ├── trans4pass_plus_small_512x512
    ├── trans4pass_plus_tiny_512x512
    ├── trans4pass_small_512x512
    └── trans4pass_tiny_512x512

Some test examples:

# Trans4PASS
python tools/eval.py --config-file configs/cityscapes/trans4pass_tiny_512x512.yaml
python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_tiny_1080x1080.yaml
# Trans4PASS+
python tools/eval.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml
python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml
python tools/eval_dp13.py --config-file configs/synpass13/trans4pass_plus_tiny_512x512.yaml
python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml

Trans4PASS models CS -> DP:

from Cityscapes to DensePASS

python tools/eval.py --config-file configs/cityscapes/trans4pass_plus_tiny_512x512.yaml

Trans4PASS models SPin -> SPan:

from Stanford2D3D_pinhole to Stanford2D3D_panoramic

python tools/eval_s2d3d.py --config-file configs/stanford2d3d/trans4pass_plus_tiny_1080x1080.yaml

Trans4PASS models on SPan:

supervised trained on Stanford2D3D_panoramic

# modify fold id at segmentron/data/dataloader/stanford2d3d_pan.py
# modify TEST_MODEL_PATH at configs/stanford2d3d_pan/trans4pass_plus_small_1080x1080.yaml
python tools/eval_s2d3d.py --config-file configs/stanford2d3d_pan/trans4pass_plus_tiny_1080x1080.yaml

Note: for the Trans4PASS versions (not Trans4PASS+), please modify the respective DMLP version, check here.

Trans4PASS models on SP:

supervised trained on SynPASS

python tools/eval_sp.py --config-file configs/synpass/trans4pass_plus_tiny_512x512.yaml

Trans4PASS models Pin2Pan vs. Syn2Real settings:

• (1) Indoor Pin2Pan: SPin8 -> SPan8
• (2) Indoor Syn2Real: S3D8 -> SPan8
• (3) Outdoor Pin2Pan: CS13 -> DP13
• (4) Outdoor Syn2Real: SP13 -> DP13

python tools/eval_dp13.py --config-file configs/synpass13/trans4pass_plus_tiny_512x512.yaml
python tools/eval_dp13.py --config-file configs/cityscaps13/trans4pass_plus_tiny_512x512.yaml
python tools/eval_s2d3d8.py --config-file configs/structured3d8/trans4pass_plus_tiny_512x512.yaml
python tools/eval_s2d3d8.py --config-file configs/stanford2d3d8/trans4pass_plus_tiny_512x512.yaml

Domain Adaptation

Train

After the model is trained on the source domain, the model can be further trained by the adversarial method for warm-up in the target domain.

cd adaptations
python train_warm.py

Then, use the warm up model to generate the pseudo label of the target damain.

python gen_pseudo_label.py

The proposed MPA method can be jointly used for perform domain adaptation.

# (optional) python train_ssl.py
python train_mpa.py

Domain adaptation with Trans4PASS+, for example, adapting model trained from pinhole to panoramic, i.e., from Cityscapes13 to DensePASS13 (CS13 -> DP13):

python train_warm_out_p2p.py
python train_mpa_out_p2p.py

Test

Download the models from GoogleDrive and save in adaptations/snapshots folder as:

adaptations/snapshots/
├── CS2DensePASS_Trans4PASS_v1_MPA
│   └── BestCS2DensePASS_G.pth
├── CS2DensePASS_Trans4PASS_v2_MPA
│   └── BestCS2DensePASS_G.pth
├── CS132CS132DP13_Trans4PASS_plus_v2_MPA
│   └── BestCS132DP13_G.pth
├── CS2DP_Trans4PASS_plus_v1_MPA
│   └── BestCS2DensePASS_G.pth
└── CS2DP_Trans4PASS_plus_v2_MPA
    └── BestCS2DensePASS_G.pth

Change the RESTORE_FROM in evaluate.py file. Or change the scales in evaluate.py for multi-scale evaluation.

# Trans4PASS
cd adaptations
python evaluate.py
# Trans4PASS+, CS132DP13
python evaluate_out13.py

References

We appreciate the previous open-source works.

• Segmentron
• SegFormer
• mmsegmentation

License

This repository is under the Apache-2.0 license. For commercial use, please contact with the authors.

Citations

If you are interested in this work, please cite the following works:

Trans4PASS+ [PDF]

@article{zhang2022behind,
  title={Behind Every Domain There is a Shift: Adapting Distortion-aware Vision Transformers for Panoramic Semantic Segmentation},
  author={Zhang, Jiaming and Yang, Kailun and Shi, Hao and Rei{\ss}, Simon and Peng, Kunyu and Ma, Chaoxiang and Fu, Haodong and Wang, Kaiwei and Stiefelhagen, Rainer},
  journal={arXiv preprint arXiv:2207.11860},
  year={2022}
}

Trans4PASS [PDF]

@inproceedings{zhang2022bending,
  title={Bending Reality: Distortion-aware Transformers for Adapting to Panoramic Semantic Segmentation},
  author={Zhang, Jiaming and Yang, Kailun and Ma, Chaoxiang and Rei{\ss}, Simon and Peng, Kunyu and Stiefelhagen, Rainer},
  booktitle={2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR)},
  pages={16917--16927},
  year={2022}
}