Home

Awesome

<div align="center"> <h2>ECCV 2022</h2> <h1>Ultra-high-resolution unpaired stain transformation via Kernelized Instance Normalization</h1>

Ming-Yang Ho*, Ā  Min-Sheng Wu, Ā  Che-Ming Wu Ā 

aetherAI, Taipei, Taiwan<br>

(* corresponding author)

[Paper] [Project Page]

</div>

šŸŽ‰ News

We have invented Dense Normalization (DN) and published in ECCV 2024. DN is better than Kernelized Instance Normalization. Please don't forget to check our latest DN here.

KIN Anime

Release

We have released v0 that can reproduce experiments mentioned in our paper or be used in your application and study.

Environment

Quick instruction

A simple example is provided for you here. Or you can jump to the next section to train a model for your own dataset. All the steps here will help you train a model with CUT framework.

Prepare dataset

Take a look at the config.yaml

The whole pipeline is heavily dependent on the config.yaml. Please take a look at the ./data/example/config.yaml first to understand what are necessary during training and testing process. You can easily train your own model with your own dataset by modifiying the config.yaml.

Preprocessing

  1. It is recommended to manually crop a center part from HE.jpg and ER.jpg first as the main contents are surrounded by a lot of unnecessary blank region, which will increase the training time but make the distribution hard to be learned.
  2. Assume these two images are cropped and place at ./data/example/HE_cropped.jpg and ./data/example/ER_cropped.jpg.
  3. Execute the following script to crop patches for training and testing.
python3 crop_pipeline.py -c ./data/example/config.yaml

Training

  1. Train the model
python3 train.py -c ./data/example/config.yaml
  1. Wait for the model training

Inference

As the testing data have been cropped during the first step, we can skip this step here.

python3 transfer.py -c config_example.yaml --skip_cropping

The output will be in the ./experiments/example_CUT/test/HE_cropped/ folder. The following is an example of output file structure.

experiments/
ā””ā”€ā”€ example_CUT
    ā”œā”€ā”€ test
    ā”‚   ā””ā”€ā”€ HE_cropped
    ā”‚       ā”œā”€ā”€ combined_in_30.png
    ā”‚       ā”œā”€ā”€ combined_kin_30_constant_5.png
    ā”‚       ā”œā”€ā”€ combined_tin_30.png
    ā”‚       ā”œā”€ā”€ in
    ā”‚       ā”‚   ā””ā”€ā”€ 30
    ā”‚       ā”œā”€ā”€ kin
    ā”‚       ā”‚   ā””ā”€ā”€ 30
    ā”‚       ā”‚       ā””ā”€ā”€ constant_5
    ā”‚       ā””ā”€ā”€ tin
    ā”‚           ā””ā”€ā”€ 30
    ā””ā”€ā”€ train

Train your own model with your own dataset

  1. Create a folder in ./data/
  2. Put a config.yaml in ./data/$your_folder/
  3. Modify config.yaml
  4. Prepare images (domain X) and images in (domain Y) in ./data/$your_folder/.
  5. Crop those images into patches.
python3 crop_pipeline.py -c ./data/$your_folder/config.yaml
python3 crop.py -i ./data/$your_folder/$image_a -o ./data/$your_folder/trainX/ --thumbnail_output ./data/$your_folder/trainX/
python3 crop.py -i ./data/$your_folder/$image_b -o ./data/$your_folder/trainX/ --thumbnail_output ./data/$your_folder/trainX/
...
python3 crop.py -i ./data/$your_folder/$test_a -o ./data/$your_folder/$test_a/ --stride 512 --thumbnail_output ./data/example/$test_a/
python3 crop.py -i ./data/$your_folder/$test_b -o ./data/$your_folder/$test_b/ --stride 512 --thumbnail_output ./data/example/$test_b/
...
  1. Modify TRAINING_SETTING section in ./data/$your_folder/config.yaml, especially the TRAIN_DIR_X and TRAIN_DIR_Y.
  2. Train the model
python3 train.py -c ./data/$your_folder/config.yaml
  1. Inference
python3 transfer.py -c ./data/$your_folder/config.yaml --skip_cropping
python3 transfer.py -c ./data/$your_folder/config.yaml --skip_cropping

Inference options

Besides kernelized instance normalization, thumbnail instance normalization and instance normalization are also provided.

Kernelized instance normalization

You can adjust NORMALIZATION.PADDING, NORMALIZATION.KERNEL_TYPE, and NORMALIZATION.KERNEL_SIZE for inference.

INFERENCE_SETTING:
  ...
  NORMALIZATION:
      TYPE: "kin"
      PADDING: 1
      KERNEL_TYPE: "constant"  # constant or gaussian
      KERNEL_SIZE: 3

Thumbnail instance normalization

Please provide the path of the THUMBNAIL.

INFERENCE_SETTING:
  ...
  NORMALIZATION:
      TYPE: "tin"
  THUMBNAIL: "./data/example/testX/thumbnail.png"

Instance normalization

Specification of NORMALIZATION.TYPE: in is enough.

INFERENCE_SETTING:
  ...
  NORMALIZATION:
      TYPE: "in"

Training framework options

Besides CUT, LSeSim and CycleGAN are also provided. For each experiment, you should rename EXPERIMENT_NAME to avoid overwritting.

CUT

Has been described above.

MODEL_NAME: "CUT"

CycleGAN

Specify in the config.yaml.

MODEL_NAME: "cycleGAN"

LSeSim

Please use F-LSeSim, which is subtly modifed from the offical implementation.

  1. Training and testing data can be prepared in the current ./data/ where you might have already created during training other models. Duplicated work is not required.
  2. Modify your config.yaml. Please set Augment to True for L-LSeSim or False for F-LSeSim.
MODEL_NAME: "LSeSim"
...
TRAINING_SETTING:
  Augment: True #LSeSim
  1. Modify data path in the config.yaml, including EXPERIMENT_ROOT_PATH, TRAINING_SETTING::TRAIN_ROOT, TRAINING_SETTING::TRAIN_DIR_X, TRAINING_SETTING::TRAIN_DIR_Y, INFERENCE_SETTING::TEST_X, INFERENCE_SETTING::TEST_DIR_X, and INFERENCE_SETTING::THUMBNAIL.<br> It is recommended to use absolute path to avoid any modification when change to different frameworks.
# Example 1
EXPERIMENT_ROOT_PATH: "./experiments/" 
# Change to
EXPERIMENT_ROOT_PATH: "../experiments/" 

# Example 2
TRAINING_SETTING:
  TRAIN_ROOT: "./data/example/"
# Change to
TRAINING_SETTING:
  TRAIN_ROOT: "../data/example/"
  1. Move to ./F-LSeSim.
cd ./F-LSeSim
  1. Run script
./scripts/train_sc.sh $path_to_yaml
./scripts/train_sc.sh ./../data/example/config.yaml
  1. Inference
./scripts/transfer_sc.sh $path_to_yaml
./scripts/transfer_sc.sh ./../data/example/config.yaml

Metrics

1. Human evaluation study

We open-source the web server for human evaluation study. Researchers can easily modify the config to conduct their human evaluation study.

2. Comparison between two distributions (WITH REFERENCE)

Given two folders pathA and pathB that store the original and generated images within the same domain, following metrics will be calculated.

python3 metric_images_with_ref.py --path-A $pathA --path-B $pathB

If images are stored in multiple folders, please concatenate those paths with delimiters of ,.

python3 metric_images_with_ref.py --path-A $pathA1,$pathA2,... --path-B $pathB1,$pathB2,...

3. For two whole images (WITH REFERENCE)

python3 metric_whole_image_with_ref.py --image_A_path $path_to_ref_image --image_B_path $path_to_compared_image

4. For single whole image (NO REFERENCE)

Please refer to the implementation of NIQE and PIQE calcuations in this repo.

python3 metric_whole_image_no_ref.py --path $image_path

Prove of concept

Script has been provided to visualize the relationship between thumbnail's features and patches' features, which shows that the concept using the same mean and variance calcuated from the thumbnail is incorrect and patches nearby each other share similar features. concept concept Please specify the image that would be tested in the inference part of config.yaml. Then:

python3 appendix/proof_of_concept.py -c $path_to_config_file

Generated images would be saved in ./proof_of_concept/

Acknowledgement

We thank Chao-Yuan Yeh, the CEO of aetherAI, for pro- viding computing resources, which enabled this study to be performed, and Cheng-Kun Yang for his revision suggestions.</br> Besides our novel kernelized instance normalizatio module, we use CycleGAN, Contrastive Unpaired Translation (CUT) as our backbone, and LSeSim. For the CUT model, please refer to the official implementation here. This code is a simplified version revised from wilbertcaine's implementation.