Awesome

[COMMSENG'24, TMI'24] Interactive Computer-Aided Diagnosis using LLMs

This repo includes official implementations of ChatCAD and ChatCAD+

Paper

<b>[Nature COMMSENG] ChatCAD: Interactive Computer-Aided Diagnosis on Medical Image using Large Language Models</b> by Sheng Wang, Zihao Zhao, Xi Ouyang, Tianming Liu, Qian Wang, Dinggang Shen

<p align="justify"> (a) Overview of our proposed strategy. The image is processed by various networks to generate diverse outputs, which are then transformed into text descriptions. The descriptions, served as a link between visual and linguistic information, are combined as inputs to a large language model (LLM). With its ability to reason and its knowledge of the medical field, the LLM can provide a condensed report. (b) Interactive explanations and medical advice from ChatCAD.

<b>[IEEE TMI] ChatCAD+: Towards a Reliable and Universal Interactive CAD using LLMs</b> by Zihao Zhao*, Sheng Wang*, Jinchen Gu*, Yitao Zhu*, Lanzhuju Mei, Zixu Zhuang, Zhiming Cui, Qian Wang, Dinggang Shen

<p align="justify"> Overview of our proposed ChatCAD+ system. (a) For patients seeking a diagnosis, ChatCAD+ generates reliable medical reports based on the input medical image(s) by referring to local report database. (b) Additionally, for any inquiry from patients, ChatCAD+ retrieves related knowledge from online database and lets large language model generate reliable response.

Introduction

This repository provides the official implementation of some components of ChatCAD+:<br/>

• Modality identification <a src="https://colab.research.google.com/assets/colab-badge.svg" href="https://colab.research.google.com/drive/1mbBgkoyk4n_qAJasY5_cOAqg7I5WP1H7?usp=sharing"> <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open in Colab"> </a>
• Chinese version Interactive CAD of Chest X-rays
• LLM-based knowledge retrieval
• An easy-deploy local web ui based on Gradio

Resources

• We would like to thank Merck Manual Professional who make all these medical knowledge public, we sorted their website for easier usage: here
• A BART-based model that has the capability to translate chest X-ray reports into Chinese well [link]

Usage

weights&others

• R2GenCMN: r2gcmn_mimic-cxr.pth and annotation.json
• PCAM weights: JFchexpert.pth
• Place annotation.json under ./r2g/ and pre-trained weights under ./weights/
• For template retrieval system, please download MIMIC-CXR reports from official website and organize them into a dictionary, save as report_en_dict.json under the ./

You can either find them from original repository or download from Google Drive

Deploy local web ui

• pip install -r requirements.txt
• implement web.py and load your openai api-key

Citation

If you find our work useful, please consider giving a star ⭐ and citation.

@article{wang2024interactive,
  title={Interactive computer-aided diagnosis on medical image using large language models},
  author={Wang, Sheng and Zhao, Zihao and Ouyang, Xi and Liu, Tianming and Wang, Qian and Shen, Dinggang},
  journal={Communications Engineering},
  volume={3},
  number={1},
  pages={133},
  year={2024},
  publisher={Nature Publishing Group UK London}
}

@article{zhao2024chatcad+,
  title={Chatcad+: Towards a universal and reliable interactive cad using llms},
  author={Zhao, Zihao and Wang, Sheng and Gu, Jinchen and Zhu, Yitao and Mei, Lanzhuju and Zhuang, Zixu and Cui, Zhiming and Wang, Qian and Shen, Dinggang},
  journal={IEEE Transactions on Medical Imaging},
  year={2024},
  publisher={IEEE}
}

Acknowledgment

Our implementation (including coming version) is based on the following codebases. We gratefully thank the authors for their wonderful works.

R2GenCMN, PCAM, CSNet.