Home

Awesome

Plugin-Networks

Plugin-Network framework is a code that enables adding partial evidence (conditional information) to already trained neural networks. More details are available in the paper.

Setting up the environment

Currently, we provide a working example on split1 of SUN397 dataset. Please run download_environment.sh. This script will download trained base model and dataset. In the result you should get workspace directory with the following structure:

sun397/
├── base_models
│   └── sun397_base_model_split01.pth.tar
├── hierarchy.csv
├── images -> /home/user/Datasets/SUN397/
│   ├── a
│   ├── b
│   ├── c
│   ├── ClassName.txt
│   ├── d
│   ├── e
│   ├── f
│   ├── g
│   ├── h
│   ├── i
│   ├── j
│   ├── k
│   ├── l
│   ├── m
│   ├── n
│   ├── o
│   ├── p
│   ├── r
│   ├── README.txt
│   ├── s
│   ├── t
│   ├── u
│   ├── v
│   ├── w
│   └── y
└── partitions
    ├── ClassName.txt
    ├── Testing_01.txt
    ├── Testing_02.txt
    ├── Testing_03.txt
    ├── Testing_04.txt
    ├── Testing_05.txt
    ├── Testing_06.txt
    ├── Testing_07.txt
    ├── Testing_08.txt
    ├── Testing_09.txt
    ├── Testing_10.txt
    ├── Training_01.txt
    ├── Training_02.txt
    ├── Training_03.txt
    ├── Training_04.txt
    ├── Training_05.txt
    ├── Training_06.txt
    ├── Training_07.txt
    ├── Training_08.txt
    ├── Training_09.txt
    └── Training_10.txt

Running the experiment

The best idea is to add the code directory to PYTHONPATH eg. export PYTHONPATH=plugin-networks:$PYTHONPATH, as well as update PATH: export PATH=plugin-networks/pluginnet:$PATH.

Now in workspace/sun397 type train.py ../../code_release/confs/SUN397/conf_pe.json output This runs the training script. The results will be saved in the output dir. Also runs directory will be created with TensorBoard output.

The experiment uses conf_pe.json as a configuration file. The structure of the file is explained in the next section.

Config JSON file

Each experiment is described bu JSON file. The example file is located in plugin-networks/confs/SUN397/conf_pe.json. The file has the following structure:

{
   "task": "sun397_pe",
   "dataset_root": "images",
   "split_file_train": "partitions/Training_01.txt",
   "split_file_test": "partitions/Testing_01.txt",
   "hierarchy_file": "hierarchy.csv",
   "base_model_file": "base_models/sun397_base_model_split01.pth.tar",
   "criterion": "cross_entropy",
   "epochs": 15,
   "lr": 0.001,
   "lr_decay": 0.1,
   "lr_step": 5,
   "momentum": 0.9,
   "clip_gradient": -1,
   "weight_decay": 5e-4,
   "plugins": {
       "conv1": [3, 256, 256, 256, 64],
       "conv2": [3, 256, 256, 256, 192],
       "conv3": [3, 256, 256, 256, 384],
       "conv4": [3, 256, 256, 256, 256],
       "conv5": [3, 256, 256, 256, 256],
      "linear1": [3, 256, 256, 256, 4096],
      "linear2": [3, 256, 256, 256, 4096],
      "linear3": [3, 256, 256, 256, 397]
   },
   "tag": "linear3_p10_split01",
   "seed": 0
}

Below the JSON file keys are explained:

How to use Plugin Networks framework with custom architecture

If you have your pretrained model with your architecture that is not yet supported by the current version of code. You can easily extend it.

  1. Use function build_plugins from pluginnet/common/model.py. It takes a plugin definition dictionary as an input as described in the above section. The output is a list of nn.Modules which are plugin networks.

  2. Use function operator_factory from pluginnet/common/model.py to create fusion operator.

  3. Modify your base network in forward method that plugin network input is passed to the network:

if layer_name in self.plugins_dict.keys():
    plugin_layer = self.plugins_dict[layer_name]
    plugin_output = plugin_layer(partial_evidence)

And then fuse base network layer output with plugin network output:

def forward(self, x_in):
    ...
    x = layer_output
    x = self.operator(x, plugin_output)
    ...
    return x

The implemented example is available in pluginnet/sun397/partial_evidence.py class AlexNetPartialEvidence.

Citation

If you found this paper and code useful please cite:

@InProceedings{Koperski_2020_WACV,
author = {Koperski, Michal and Konopczynski, Tomasz and Nowak, Rafal and Semberecki, Piotr and Trzcinski, Tomasz},
title = {Plugin Networks for Inference under Partial Evidence},
booktitle = {The IEEE Winter Conference on Applications of Computer Vision (WACV)},
month = {March},
year = {2020}
}