Awesome
<link href="./docs/style.css" rel="stylesheet"/>About the generation of synthetic laparascopic images using diffusion-based models
View research on Github Page
<div class="row"></div>This repository is the code base used for our research. Please follow the guide:
Clone Repository
git clone https://github.com/SimeonAllmendinger/SyntheticImageGeneration.git
cd SyntheticImageGeneration
Virtual Environment
To set up a virtual environment, follow these steps:
- Create a virtual environment with python version 3.9:
virtualenv venv -p $(which python3.9)
- Activate the virtual environment:
source venv/bin/activate
- Install the required packages:
pip install --no-cache-dir -r requirements.txt
Download Model Weights
To test the generation of laparoscopic images with the Elucidated Imagen model, please do the following:
cd src/assets/
gdown --folder https://drive.google.com/drive/folders/1np4BON_jbQ1-15nVdgMCP1VKSbKS3h2M
gdown --folder https://drive.google.com/drive/folders/1BNdUmmqN18K4_lH0BMk0bwRkiy8Sv6D-
gdown --folder https://drive.google.com/drive/folders/1Y0yQmP3THRzP8UFlAyMFHYUymTUu7ZUu
cd ../../
Testing
To test the generation of laparoscopic images with the pre-trained Elucidated Imagen model, please do the following:
python3 src/components/test.py --model=ElucidatedImagen --text='grasper grasp gallbladder in callot triangle dissection' --cond_scale=3
You can apply the Imagen and Elucidated Imagen model, various conditiong scales and a suitable text prompt according to your desire! Feel free to try everything out. (The sampling of the Elucidated Imagen model also works well on a machine without GPU).
The hyperparameter configurations of the diffusion-based models are contained in the config file respectively (Model Config Folder). Their weights can be found in the table:
Model | Training Dataset | Link |
---|---|---|
Dall-e2 Prior | CholecT45 | Dalle2_Prior_T45 |
Dall-e2 Decoder | CholecT45 | Dalle2_Decoder_T45 |
Imagen | CholecT45 | Imagen_T45 |
Imagen | CholecT45 + CholecSeg8k | Imagen_T45_Seg8k |
Elucidated Imagen | CholecT45 | ElucidatedImagen_T45 |
Elucidated Imagen | CholecT45 + CholecSeg8k | ElucidatedImagen_T45_Seg8k |
Results
Before running the code for training, tuning and extensive testing purposes, please create a directory to store the results:
mkdir results
cd results
mkdir rendevouz
mkdir testing
mkdir training
mkdir TSNE
mkdir tuning
Data
git LFS
Install git LFS with homebrew: https://brew.sh/index_de
brew install git-lfs
git lfs install
git lfs track "*.pt"
git add .gitattributes
Download
To download the required datasets (CholecT45, CholecSeg8k, CholecT50, Cholec80), follow these steps:
- Create a directory to store the data:
cd
cd SyntheticImageGeneration
mkdir data
cd data
- Download the datasets into this directory after successful registration:
- Cholec80: https://docs.google.com/forms/d/1GwZFM3-GhEduBs1d5QzbfFksKmS1OqXZAz8keYi-wKI
- CholecT45: https://forms.gle/jTdPJnZCmSe2Daw7A
- CholecSeg8k: https://www.kaggle.com/datasets/newslab/cholecseg8k/download?datasetVersionNumber=11
- CholecT50: https://forms.gle/GbMj8TwNoNpMUJuv9
Preparation
To enable dashboards please copy your configs of neptune.ai and wandb.ai in the according .yaml file:
cd
cd SyntheticImageGeneration/configs/visualization/
touch config_neptune.yaml
touch config_wandb.yaml
- Neptune.ai (https://neptune.ai): Insert your acceess configs in the file config_neptune.yaml
project: "your-project-name"
api_token: "your-api-token"
- Weights&Biases: Insert your access configs in the file config_neptune.yaml
project: "your-project-name"
api_key: "your-api-key"
To prepare the data for the experiments, run the following script:
cd SyntheticImageGeneration
./scripts/run_data_preparation.sh
Now, you are prepared to explore the code base in full extense!
Rendezvouz (GitHub)
In the following, we provide trained rendezvous model weights from the 3-fold cross-validation for various proportions of generated samples:
Model | %2 samples | %5 samples | %10 samples | %20 samples | %25 samples |
---|---|---|---|---|---|
I5-RDV | Weights | Weights | Weights | Weights | Weights |
EI5-RDV | Weights | Weights | Weights | Weights | Weights |
Acknowledgements
We acknowledge support by the state of Baden-Württemberg through bwHPC.