Awesome
XrayPULSE
<!-- **Here are some ideas to get you started:** šāāļø A short introduction - what is your organization all about? š Contribution guidelines - how can the community get involved? š©āš» Useful resources - where can the community find your docs? Is there anything else the community should know? šæ Fun facts - what does your team eat for breakfast? š§ Remember, you can do mighty things with the power of [Markdown](https://docs.github.com/github/writing-on-github/getting-started-with-writing-and-formatting-on-github/basic-writing-and-formatting-syntax) --> <!-- Insert the project banner here --> <div align="center"> <a href="https://"><img width="1000px" height="auto" src="./banner.png"></a> </div><!-- Select some of the point info, feel free to delete -->
Key Features
This repository provides the official implementation of XrayPULSE:
Key feature bulletin points here
- An attempt to extend PULSE to a biomedical multimodal conversational assistant.
- XrayPULSE is fintuned on Xray-Report paired datasets in Chinese
Details
Our model is based on PULSE. We utilize MedCLIP as our medical visual encoder and Q-former (BLIP2) following a simple linear transformation as the adapter to inject the image to PULSE. For aligning the frozen visual encoder and the LLM by the adapter, we generate Chinese-version Xray-Report paired data from free-text radiology reports of two datasets (MIMIC-CXR and OpenI) with the help of chatGPT. To facilitate research in biomedical multimodal learning, we will release the data to the public.
<!-- Insert a pipeline of your algorithm here if got one --> <div align="center"> <a href="https://"><img width="1000px" height="auto" src="./framework.png"></a> </div>Get Started
Installation
git clone https://github.com/openmedlab/XrayPULSE.git
cd XrayPULSE
Environment
conda env create -f env.yml
conda activate xraypulse
Prepare the pretrained weights
You can find the pretrained model weights.
The weights of PULSE would be in a single folder in a structure similar to the following:
pulse_weights
āāā config.json
āāā generation_config.json
āāā tokenizer.json
āāā tokenizer_config.json
āāā special_tokens_map.json
āāā pytorch_model.bin.index.json
āāā pytorch_model-00001-of-00002.bin
āāā pytorch_model-00002-of-00002.bin
Then, set the path of pulse_weights to "bloom_model" in the model config file "xraypulse/configs/models/xraypulse.yaml"
And add the path of the pretrained checkpoint in "demo_configs/xraypulse_demo.yaml".
Run Demo
bash run_demo.sh
š Acknowledgement
This project is built upon the gaint sholders of XrayGPT. Great thanks to it!
We used medical aware image encoder from MedCLIP.
The model architecture of XrayGPT follows BLIP2.
š”ļø License
This project is under the CC-BY-NC 4.0 license. See LICENSE for details.