Awesome

Pre-trained Multimodal Large Language Model Enhances Dermatological Diagnosis using SkinGPT-4

Juexiao Zhou, Xiaonan He, Liyuan Sun, Jiannan Xu, Xiuying Chen, Yuetan Chu, Longxi Zhou, Xingyu Liao, Bin Zhang, Shawn Afvari, Xin Gao

King Abdullah University of Science and Technology, KAUST

<a href='s41467-024-50043-3.pdf'><img src='https://img.shields.io/badge/Paper-PDF-red'></a>

Installation

conda env create -f environment.yml
conda activate skingpt4_llama2
conda install -c conda-forge mamba=1.4.7
conda install pytorch==2.0.0 torchvision==0.15.0 torchaudio==2.0.0 pytorch-cuda=11.8 -c pytorch -c nvidia

Download our trained weights

Our previous trained weights for skin disease diagnosis with only base dataset and Llama2 could be downloaded at skingpt4_llama2_13bchat_base_pretrain_stage2.pth. Then modify line 10 at SkinGPT-4-llama2/eval_configs/skingpt4_eval_llama2_13bchat.yaml to be the path of SkinGPT-4 weight.

Our previous trained weights for skin disease diagnosis with only step-1 dataset and Vicuna could be downloaded at skingpt4_vicuna_v1.pth. Then modify line 11 at SkinGPT-4-llama2/eval_configs/skingpt4_eval_vicuna.yaml to be the path of SkinGPT-4 weight.

Please note:

• The released trained model above cannot be used for skin disease diagnosis, they can only be used for testing code.

• The latest model trained with both public skin disease datasets and the proprietary skin disease dataset based on falcon-40b-instruct (deprecated) and llama-2-13b-chat-hf (code published only) are not publicly available currently due to privacy issues.

• Please feel free to keep in touch with xin.gao@kaust.edu.sa and juexiao.zhou@kaust.edu.sa for potential collaboration.

Prepare weight for LLMs

Llama2 Version

git clone https://huggingface.co/meta-llama/Llama-2-13b-chat-hf

Then modify line 16 at SkinGPT-4-llama2/skingpt4/configs/models/skingpt4_llama2_13bchat.yaml to be the path of Llama-2-13b-chat-hf.

Vicuna Version

# download Vicuna’s **delta** weight
git lfs install
git clone https://huggingface.co/lmsys/vicuna-13b-delta-v0

# get llama-13b model
git clone https://huggingface.co/huggyllama/llama-13b

pip install git+https://github.com/lm-sys/FastChat.git@v0.1.10
pip install transformers==4.28.0

python -m fastchat.model.apply_delta --base ./llama-13b --target ./vicuna --delta ./vicuna-13b-delta-v0

Then modify line 16 at SkinGPT-4-llama2/skingpt4/configs/models/skingpt4_vicuna.yaml to be the path of vicuna.

Launching Demo Locally

Llama2 Version

python demo.py --cfg-path eval_configs/skingpt4_eval_llama2_13bchat.yaml  --gpu-id 0

Vicuna Version

python demo.py --cfg-path eval_configs/skingpt4_eval_vicuna.yaml  --gpu-id 0

Illustraion of SkinGPT-4

Examples of Skin disease diagnosis

Clinical Evaluation

Acknowledgement

• MiniGPT-4 This repo is developped on MiniGPT-4, an awesome repo for vision-language chatbot!
• Lavis
• Vicuna
• Falcon
• Llama 2

Citation

Our paper has been accepted by Nature Communications.

If you find SkinGPT-4 to be helpful in your research or applications, please cite SkinGPT-4 using this BibTeX:

@article{zhou2024pre,
  title={Pre-trained multimodal large language model enhances dermatological diagnosis using SkinGPT-4},
  author={Zhou, Juexiao and He, Xiaonan and Sun, Liyuan and Xu, Jiannan and Chen, Xiuying and Chu, Yuetan and Zhou, Longxi and Liao, Xingyu and Zhang, Bin and Afvari, Shawn and others},
  journal={Nature Communications},
  volume={15},
  number={1},
  pages={5649},
  year={2024},
  publisher={Nature Publishing Group UK London}
}