Awesome
<div align="center"> <h2>ECCV 2022</h2> <h1>Ultra-high-resolution unpaired stain transformation via Kernelized Instance Normalization</h1>Ming-Yang Ho*, Ā Min-Sheng Wu, Ā Che-Ming Wu Ā
aetherAI, Taipei, Taiwan<br>
(* corresponding author)
[Paper
] [Project Page
]
š News
We have invented Dense Normalization (DN) and published in ECCV 2024. DN is better than Kernelized Instance Normalization. Please don't forget to check our latest DN here.
Release
We have released v0 that can reproduce experiments mentioned in our paper or be used in your application and study.
Environment
- Python 3.8.6
- All the required packages are listed in the
requirements.txt
.
Quick instruction
A simple example is provided for you here. Or you can jump to the next section to train a model for your own dataset. All the steps here will help you train a model with CUT framework.
Prepare dataset
- ANHIR
- A public dataset ANHIR is used in this project. Please first download it from the offical website and put
ANHIR2019/dataset_medium/breast_1/scale-20pc/HE.jpg
andANHIR2019/dataset_medium/breast_1/scale-20pc/ER.jpg
indata/example/
folder. We would like to transferHE (domain X)
toER (domain Y)
.
- A public dataset ANHIR is used in this project. Please first download it from the offical website and put
- Kyoto-summer2autumn
- Due to the lack of dataset consists of high-resolution, we collected and released a dataset named Kyoto-summer2autumn to facilitate further studies. Please download this dataset via the following link.
Take a look at the config.yaml
The whole pipeline is heavily dependent on the config.yaml
. Please take a look at the ./data/example/config.yaml
first to understand what are necessary during training and testing process. You can easily train your own model with your own dataset by modifiying the config.yaml
.
Preprocessing
- It is recommended to manually crop a center part from
HE.jpg
andER.jpg
first as the main contents are surrounded by a lot of unnecessary blank region, which will increase the training time but make the distribution hard to be learned. - Assume these two images are cropped and place at
./data/example/HE_cropped.jpg
and./data/example/ER_cropped.jpg
. - Execute the following script to crop patches for training and testing.
python3 crop_pipeline.py -c ./data/example/config.yaml
Training
- Train the model
python3 train.py -c ./data/example/config.yaml
- Wait for the model training
- Some transfered examples will be generated during training. Please check the
./experiments/example_CUT/train/
folder.
Inference
As the testing data have been cropped during the first step, we can skip this step here.
python3 transfer.py -c config_example.yaml --skip_cropping
The output will be in the ./experiments/example_CUT/test/HE_cropped/
folder.
The following is an example of output file structure.
experiments/
āāā example_CUT
āāā test
ā āāā HE_cropped
ā āāā combined_in_30.png
ā āāā combined_kin_30_constant_5.png
ā āāā combined_tin_30.png
ā āāā in
ā ā āāā 30
ā āāā kin
ā ā āāā 30
ā ā āāā constant_5
ā āāā tin
ā āāā 30
āāā train
Train your own model with your own dataset
- Create a folder in
./data/
- Put a
config.yaml
in./data/$your_folder/
- Modify
config.yaml
- Prepare images (domain X) and images in (domain Y) in
./data/$your_folder/
. - Crop those images into patches.
- If there is only one image in each domain
python3 crop_pipeline.py -c ./data/$your_folder/config.yaml
- Multiple images belong to one domain: you should use
crop.py
to crop each image and save those patches in the same folder (trainX
,trainY
)
python3 crop.py -i ./data/$your_folder/$image_a -o ./data/$your_folder/trainX/ --thumbnail_output ./data/$your_folder/trainX/
python3 crop.py -i ./data/$your_folder/$image_b -o ./data/$your_folder/trainX/ --thumbnail_output ./data/$your_folder/trainX/
...
- Multiple images belong to one domain: for the testing data, it is recommended to seperate patches belong to different image in different folder.
python3 crop.py -i ./data/$your_folder/$test_a -o ./data/$your_folder/$test_a/ --stride 512 --thumbnail_output ./data/example/$test_a/
python3 crop.py -i ./data/$your_folder/$test_b -o ./data/$your_folder/$test_b/ --stride 512 --thumbnail_output ./data/example/$test_b/
...
- Modify
TRAINING_SETTING
section in./data/$your_folder/config.yaml
, especially theTRAIN_DIR_X
andTRAIN_DIR_Y
. - Train the model
python3 train.py -c ./data/$your_folder/config.yaml
- Inference
- If you have only one image requires inference: modify
INFERENCE_SETTING
section in./data/$your_folder/config.yaml
, especially theTEST_X
andTEST_DIR_X
. Then,
python3 transfer.py -c ./data/$your_folder/config.yaml --skip_cropping
- If you have only many images requires inference: assume you have finishing cropping each testing image in separated folder. Please modify
TEST_X
andTEST_DIR_X
in theINFERENCE_SETTING
section and execute the following script for each image.
python3 transfer.py -c ./data/$your_folder/config.yaml --skip_cropping
Inference options
Besides kernelized instance normalization
, thumbnail instance normalization
and instance normalization
are also provided.
Kernelized instance normalization
You can adjust NORMALIZATION.PADDING
, NORMALIZATION.KERNEL_TYPE
, and NORMALIZATION.KERNEL_SIZE
for inference.
INFERENCE_SETTING:
...
NORMALIZATION:
TYPE: "kin"
PADDING: 1
KERNEL_TYPE: "constant" # constant or gaussian
KERNEL_SIZE: 3
Thumbnail instance normalization
Please provide the path of the THUMBNAIL
.
INFERENCE_SETTING:
...
NORMALIZATION:
TYPE: "tin"
THUMBNAIL: "./data/example/testX/thumbnail.png"
Instance normalization
Specification of NORMALIZATION.TYPE: in
is enough.
INFERENCE_SETTING:
...
NORMALIZATION:
TYPE: "in"
Training framework options
Besides CUT
, LSeSim
and CycleGAN
are also provided. For each experiment, you should rename EXPERIMENT_NAME
to avoid overwritting.
CUT
Has been described above.
MODEL_NAME: "CUT"
CycleGAN
Specify in the config.yaml
.
MODEL_NAME: "cycleGAN"
LSeSim
Please use F-LSeSim
, which is subtly modifed from the offical implementation.
- Training and testing data can be prepared in the current
./data/
where you might have already created during training other models. Duplicated work is not required. - Modify your
config.yaml
. Please setAugment
toTrue
forL-LSeSim
orFalse
forF-LSeSim
.
MODEL_NAME: "LSeSim"
...
TRAINING_SETTING:
Augment: True #LSeSim
- Modify data path in the
config.yaml
, includingEXPERIMENT_ROOT_PATH
,TRAINING_SETTING::TRAIN_ROOT
,TRAINING_SETTING::TRAIN_DIR_X
,TRAINING_SETTING::TRAIN_DIR_Y
,INFERENCE_SETTING::TEST_X
,INFERENCE_SETTING::TEST_DIR_X
, andINFERENCE_SETTING::THUMBNAIL
.<br> It is recommended to use absolute path to avoid any modification when change to different frameworks.
# Example 1
EXPERIMENT_ROOT_PATH: "./experiments/"
# Change to
EXPERIMENT_ROOT_PATH: "../experiments/"
# Example 2
TRAINING_SETTING:
TRAIN_ROOT: "./data/example/"
# Change to
TRAINING_SETTING:
TRAIN_ROOT: "../data/example/"
- Move to
./F-LSeSim
.
cd ./F-LSeSim
- Run script
./scripts/train_sc.sh $path_to_yaml
- e.g.
./scripts/train_sc.sh ./../data/example/config.yaml
- Then the model weights and generated samples will be in
./F-LSeSim/checkpoints/$EXPERIMENT_NAME
- Inference
./scripts/transfer_sc.sh $path_to_yaml
- e.g.
./scripts/transfer_sc.sh ./../data/example/config.yaml
- The generated images will be in
./experiments/$EXPERIMENT_NAME
Metrics
1. Human evaluation study
We open-source the web server for human evaluation study. Researchers can easily modify the config to conduct their human evaluation study.
2. Comparison between two distributions (WITH REFERENCE)
Given two folders pathA
and pathB
that store the original and generated images within the same domain, following metrics will be calculated.
- FID
python3 metric_images_with_ref.py --path-A $pathA --path-B $pathB
If images are stored in multiple folders, please concatenate those paths with delimiters of ,
.
python3 metric_images_with_ref.py --path-A $pathA1,$pathA2,... --path-B $pathB1,$pathB2,...
3. For two whole images (WITH REFERENCE)
- Histogram correlation
python3 metric_whole_image_with_ref.py --image_A_path $path_to_ref_image --image_B_path $path_to_compared_image
4. For single whole image (NO REFERENCE)
Please refer to the implementation of NIQE
and PIQE
calcuations in this repo.
- Sobel gradient
- NIQE
- PIQE
python3 metric_whole_image_no_ref.py --path $image_path
Prove of concept
Script has been provided to visualize the relationship between thumbnail's features and patches' features, which shows that the concept using the same mean and variance calcuated from the thumbnail is incorrect and patches nearby each other share similar features.
Please specify the image that would be tested in the inference
part of config.yaml
. Then:
python3 appendix/proof_of_concept.py -c $path_to_config_file
Generated images would be saved in ./proof_of_concept/
Acknowledgement
We thank Chao-Yuan Yeh, the CEO of aetherAI, for pro- viding computing resources, which enabled this study to be performed, and Cheng-Kun Yang for his revision suggestions.</br> Besides our novel kernelized instance normalizatio module, we use CycleGAN, Contrastive Unpaired Translation (CUT) as our backbone, and LSeSim. For the CUT model, please refer to the official implementation here. This code is a simplified version revised from wilbertcaine's implementation.