Awesome

CXR-CLIP

This is an official Pytorch Implementation of "CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training" [arxiv]

Environment setup

We have experimented the implementation on the following enviornment.

Pytorch 1.12
CUDA 11

pip install -r requirements.txt

Prepare dataset

Datasets we used are as follows:

Dataset	Download	Comment
MIMIC-CXR	Link	official split
CheXpert	Link	official split for train and val, and `chexpert_5x200` from GLoRIA for test
ChestX-ray14	Link	not used for test
VinDr-CXR	Link	official split for test, and random split for train and val
RSNA-Pneumonia	Link	same split as GLoRIA
SIIM-Pneumothorax	Link	same split as GLoRIA
OpenI	Link	all frontal images are used for evaluation

For more details, please refer to data preparation.

Pre-trained model checkpoint

We trained Resnet50 and SwinTiny models with three dataset compositions.
MIMIC-CXR (M), MIMIC-CXR + CheXpert (M,C), MIMIC-CXR + CheXpert + ChestX-ray14 (M,C,C14)

model / dataset	M	M,C	M,C,C14
ResNet50	Link	Link	Link
SwinTiny	Link	Link	Link

Pre-Train model

command line

single gpu

python train.py {--config-name default_config}

multi gpu

torchrun --nproc_per_node=4 --nnodes=1 --node_rank=0 --master_addr=127.0.0.1 --master_port=45678 train.py {--config-name default_config}

Evaluation

Zero-shot Evaluation

Zero-shot classification
- perform zero-shot and image-text retrieval evaluation on ( vindr_cxr, rsna_pneumonia, siim_pneumothorax, chexpert5x200, mimic_cxr, openi )
```
python evaluate_clip.py test.checkpoint=${CKPT_PATH/model-best.tar}
```

Fine-tuned Classifier (linear probing)

on rsna_pneumonia

# train
python finetune.py --config-name finetune_10 hydra.run.dir=${SAVE_DIR} data_train=rsna_pneumonia data_valid=rsna_pneumonia model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 10%
python finetune.py hydra.run.dir=${SAVE_DIR} data_train=rsna_pneumonia data_valid=rsna_pneumonia model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 100%
# evaluate
python evaluate_finetune.py data_test=rsna_pneumonia test.checkpoint=${FINETUNED_CKPT_PATH/model-best.tar}

on siim_pneumothorax

# train
python finetune.py --config-name finetune_10 hydra.run.dir=${SAVE_DIR} data_train=siim_pneumothorax data_valid=siim_pneumothorax model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 10%
python finetune.py hydra.run.dir=${SAVE_DIR} data_train=siim_pneumothorax data_valid=siim_pneumothorax model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 100%
# evaluate
python evaluate_finetune.py data_test=siim_pneumothorax test.checkpoint=${FINETUNED_CKPT_PATH/model-best.tar}

on vindr_cxr

# train
python finetune.py --config-name finetune_10 hydra.run.dir=${SAVE_DIR} data_train=vindr_cxr data_valid=vindr_cxr model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 10%
python finetune.py hydra.run.dir=${SAVE_DIR} data_train=vindr_cxr data_valid=vindr_cxr model.load_backbone_weights=${CKPT_PATH/model-best.tar} # 100%
# evaluate
python evaluate_finetune.py data_test=vindr_cxr test.checkpoint=${FINETUNED_CKPT_PATH/model-best.tar}

Citation

@incollection{You_2023,
	doi = {10.1007/978-3-031-43895-0_10},
	url = {https://doi.org/10.1007%2F978-3-031-43895-0_10},
	year = 2023,
	publisher = {Springer Nature Switzerland},
	pages = {101--111},
	author = {Kihyun You and Jawook Gu and Jiyeon Ham and Beomhee Park and Jiho Kim and Eun K. Hong and Woonhyuk Baek and Byungseok Roh},
	title="CXR-CLIP: Toward Large Scale Chest X-ray Language-Image Pre-training",
	booktitle="Medical Image Computing and Computer Assisted Intervention -- MICCAI 2023",
}

License

Contact for Issues

Kihyun You, ukihyun@kakaobrain.com
Jawook Gu, jawook.gu@kakaobrain.com