Home

Awesome

XrayPULSE

<!-- **Here are some ideas to get you started:** šŸ™‹ā€ā™€ļø A short introduction - what is your organization all about? šŸŒˆ Contribution guidelines - how can the community get involved? šŸ‘©ā€šŸ’» Useful resources - where can the community find your docs? Is there anything else the community should know? šŸæ Fun facts - what does your team eat for breakfast? šŸ§™ Remember, you can do mighty things with the power of [Markdown](https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) --> <!-- Insert the project banner here --> <div align="center"> <a href="https://"><img width="1000px" height="auto" src="./banner.png"></a> </div>
<!-- Select some of the point info, feel free to delete -->

Key Features

This repository provides the official implementation of XrayPULSE:

Key feature bulletin points here

Details

Our model is based on PULSE. We utilize MedCLIP as our medical visual encoder and Q-former (BLIP2) following a simple linear transformation as the adapter to inject the image to PULSE. For aligning the frozen visual encoder and the LLM by the adapter, we generate Chinese-version Xray-Report paired data from free-text radiology reports of two datasets (MIMIC-CXR and OpenI) with the help of chatGPT. To facilitate research in biomedical multimodal learning, we will release the data to the public.

<!-- Insert a pipeline of your algorithm here if got one --> <div align="center"> <a href="https://"><img width="1000px" height="auto" src="./framework.png"></a> </div>

Get Started

Installation

git clone https://github.com/openmedlab/XrayPULSE.git
cd XrayPULSE

Environment

conda env create -f env.yml
conda activate xraypulse

Prepare the pretrained weights

You can find the pretrained model weights.

The weights of PULSE would be in a single folder in a structure similar to the following:

pulse_weights
ā”œā”€ā”€ config.json
ā”œā”€ā”€ generation_config.json
ā”œā”€ā”€ tokenizer.json
ā”œā”€ā”€ tokenizer_config.json
ā”œā”€ā”€ special_tokens_map.json 
ā”œā”€ā”€ pytorch_model.bin.index.json
ā”œā”€ā”€ pytorch_model-00001-of-00002.bin
ā”œā”€ā”€ pytorch_model-00002-of-00002.bin 

Then, set the path of pulse_weights to "bloom_model" in the model config file "xraypulse/configs/models/xraypulse.yaml"

And add the path of the pretrained checkpoint in "demo_configs/xraypulse_demo.yaml".

Run Demo

bash run_demo.sh

šŸ™ Acknowledgement

This project is built upon the gaint sholders of XrayGPT. Great thanks to it!

We used medical aware image encoder from MedCLIP.

The model architecture of XrayGPT follows BLIP2.

šŸ›”ļø License

This project is under the CC-BY-NC 4.0 license. See LICENSE for details.