Home

Awesome

Detecting the Unexpected via Image Resynthesis

Krzysztof Lis, Krishna Nakka, Pascal Fua, Mathieu Salzmann ICCV 2019

[article] [poster]

<img src="doc/unexpected_pipeline_diagram_embed.svg" alt="pipeline" width="100%" />

Installation

git clone https://github.com/cvlab-epfl/detecting-the-unexpected --recursive

Install libraries:

pip install numpy opencv-python-headless matplotlib h5py natsort imageio torch torchvision scipy tqdm tensorboard future ipython

To run notebook files:

pip install jupyterlab

Trained weights

Download weight files [2 GiB] and place them in detecting-the-unexpected/exp (or another location specified with env variable DIR_EXPERIMENTS).

Directory structure

detecting-the-unexpected

Running the pipeline

Please see the notebook Exec_Joint_Pipeline.ipynb:

# specify input dataset, for example a directory with images
from src.datasets.dataset import DatasetImageDir
dset = DatasetImageDir(dir_root='data/joint_pipeline_example')
dset.discover()

# load the networks
from src.a05_differences.E1_article_evaluation import DiscrepancyJointPipeline
joint_pipeline = DiscrepancyJointPipeline()
joint_pipeline.init_semseg()
joint_pipeline.init_gan()
joint_pipeline.init_discrepancy()

# run and show results in notebook
joint_pipeline.run_on_dset(dset, b_show=True)

The notebook Exec_Evaluations.ipynb can be used to the steps separately saving intermediate results.

Dataset processing

The Lost and Found and Cityscapes datasets were used in 1024x512 resolution, while the original downloads are 2048x1024. The conversion can be performed with the scripts in src/datasets/conversion_tools.

The script needs a webp encoder and imagemagick. On Windows the ImageMagick binary is called magick instead of convert, so there is a slightly different version of the scripts for this OS.

Lost and Found

Cityscapes

Discrepancy Network

Synthetic discrepancy dataset

The synthetic discrepancy dataset used in our experiments can be downloaded here: [052X_synthetic_discrepancy__fakeSwapFgd.7z (208MB)]. Please place it at data/discrepancy_dataset/cityscapes/051X_semGT__fakeSwapFgd__genNoSty.

To generate your own version of the synthetic discrepancy dataset, please follow the instructions in Discrepancy_GenerateDataset.ipynb.

Training

The Experiment class (src/pipeline/experiment.py) is used to train the networks (extract from src/a05_differences/experiments.py):

# First we add a new class for our experiment
class Exp0552_NewDiscrepancyTraining(Exp0521_SwapFgd_ImgAndLabelsVsGen_semGT):
	cfg = add_experiment(Exp0521_SwapFgd_ImgAndLabelsVsGen_semGT.cfg,
		name = '0552_NewDiscrepancyTraining',
		gen_name = '051X_semGT__fakeSwapFgd__genNoSty',
	)

# Execute the training process
# from src.a05_differences.experiments import MyExperimentVariant
Exp0552_NewDiscrepancyTraining.training_procedure()

Weights will be written to $DIR_EXP/0552_NewDiscrepancyTraining Checkpoints are saved every epoch:

The directory will also contain:

The loss is written to tensorboard can can be displayed in the following way:

	tensorboard --logdir $DIR_EXP/0552_NewDiscrepancyTraining

Training scripts for variants of the discrepancy network:

Contact

I am working to provide more examples and automated scripts.

For any additional information or requests, please contact Krzysztof Lis.