Awesome

ChatPsychiatrist

Introducing our project, the ChatPsychiatrist—a transformative solution in the field of mental health support. We aim to address the delays in traditional psychiatric and counseling services by providing early-stage assistance and effective interventions. Utilizing LLMs, the ChatPsychiatrist swiftly identifies individual issues and offers tailored treatment recommendations. With an adaptive communication style, it becomes a powerful and free tool to provide personalized mental health support to users in need.

This repo open-sources the Instruct-tuned LLaMA-7B model that has been fine-tuned with counseling domian instruction data. To construct our 8K size instruct-tuning dataset, we collected real-world counseling dialogue examples and employed GPT-4 as an extractor and filter. In addition, we have introduced a comprehensive set of metrics, specifically tailored to the LLM+Counseling domain, by incorporating counseling domain evaluation criteria. These metrics enable the assessment of performance in generating language content that involves multi-dimensional counseling skills. The brief pipeline is shown below:

Quick Start

⚙️ Install

Our repo mainly constructed based on FastChat. Please follow the install instructions. Or you can install from source by

git clone https://github.com/EmoCareAI/ChatPsychiatrist.git
cd ChatPsychiatrist
pip3 install -r requirements.txt

⏬ Model Download

We provide the weights hold on Huggingface. You can download the model weights using Huggingface Client Library. Or use the following wget script:

mkdir -P PATH_TO_WEIGHTS_DIR && cd PATH_TO_WEIGHTS_DIR
user='EmoCareAI'
repo='ChatPsychiatrist'
curl "https://huggingface.co/${user}/${repo}/tree/main" | grep -o 'href="[^"]*"' | cut -d'"' -f2 | grep "^/${user}/${repo}/blob/main/" | sed "s|^|https://huggingface.co|; s|/blob/|/resolve/|g" > files.txt && wget -nc -i files.txt && rm files.txt
cd ..

🚀 Inference

Inference with CLI

See details refer to FastChat CLI Inference. In short, you can run the following command below around 14GB of GPU memory.

# Single GPU inference
python3 -m fastchat.serve.cli --model-path PATH_TO_WEIGHTS_DIR
# Multi GPUs inference
python3 -m fastchat.serve.cli --model-path PATH_TO_WEIGHTS_DIR --num-gpus N

Serving with Web GUI

1. Launch the controller, which manages the distributed workers

python3 -m fastchat.serve.controller

2. Launch the model worker.

python3 -m fastchat.serve.model_worker --model-path PATH_TO_WEIGHTS_DIR

3. Launch the Gradio web server

python3 -m fastchat.serve.gradio_web_server

See details refer to FastChat Web GUI Serving

Data

Data Source

The data used for this project comes from ~260 real conversations in counseling recordings (in English). The transcripts of these recordings were used as the primary source for building the training and testing datasets. These conversations cover a variety of topics, including emotion, family, relationship, career development, academic stress, etc. This figure shows the coverage:

Data Processing

To build the dataset, we took the following steps:

1. Transcription: The counseling recordings were transcribed to obtain the raw textual data.

2. Data Cleaning & Information Extraction: The transcripts were cleaned to remove any irrelevant or sensitive information, ensuring that the data used for training and testing maintains privacy and ethical standards. Since single round conversations in real counseling recordings may contain limited information, we segmented transcripts into short conversations for further query-answer pair generation.

3. Query-Answer Pair Generation: With the extracted short conversations, we then used the GPT-4 API to build the query-answer data. Each short conversation clip (i.e., each of ten round) text was used as the context, and GPT4 was prompted to generate the corresponding query and answer. In total, 8,187 query-answer pairs were produced.

   GPT4 generated a summary of the important information for each conversation. These summaries provided a higher level of context and detail that helps the model to understand and generate meaningful responses.

4. Testing Data: Another 100 quert-answer pairs were created using the same process as the training set. Topics covered include: Addiction, Anger, Anxiety, Bereavement, Culture, Depression, Eating_Disorders, Family, Fear, Illness, LGBTQ, Pain, Perfectionism and Stress.

Data Examples

The following table contains four samples from different topics, with the user providing the query and the psychiatrist providing the feedback.

Evaluation

Here is the comparison between the performance of our chatbot and other large language models. The evaluation is based on seven counseling skills and overall performance, and our chatbot demonstrates a better performance in most of these categories.

Counselling Bench

We have designed 229 queries specifically to evaluate the performance of chatbots in counseling scenarios. The current benchmark comprises a challenging set of single-turn open-ended questions, providing a rigorous evaluation of chat assistance capabilities.

Please check the subset santitised part counseliing bench questions for reference. You can evaluate using the following command:

python3 -m fastchat.llm.judge.gen_model_answer \
  --model-id chatpsychiatrist \
  --model-path PATH_TO_WEIGHTS_DIR \
  --bench-name counselling_bench 

Metrics

The evaluation metrics are derived from the following categories:

1. Information: Measures the ability to provide accurate and relevant information.
2. Direct Guidance: Evaluates the chatbot's capability to offer clear instructions and guidance.
3. Approval & Reassurance: Assesses the chatbot's capacity to provide emotional support and encouragement.
4. Restatement, Reflection & Listening: Rates how effectively the chatbot can rephrase, reflect on user inputs, and exhibit active listening skills.
5. Interpretation: Measures how well the chatbot can analyze and interpret situations or user inputs.
6. Self-disclosure: Quantifies the chatbot's ability to share relevant information about itself.
7. Obtain Relevant Information: Evaluates the capability to ask appropriate questions to gather necessary details.

See Evaluation Definetion for details on the metrics' definition and example phrases.

Comparison with Other Large Language Models

Single Turn Evaluation

In the figure below, we compare the performance of our LLM chatbot with other models (evaluated by GPT-4). The scores in each category represent the effectiveness of the chatbot in that particular aspect.

As seen in the table, our LLM chatbot outperforms other models in most categories, demonstrating an overall higher performance. This result highlights the effectiveness of our chatbot in providing users with an efficient and engaging conversational experience.

Please note that the scores mentioned in the table represent relative performance and are for comparison purposes only. Individual model performance may vary depending on the specific use case and scenario.

Acknowledgement

This project has referred the following open-source projects. We would like to express our gratitude to the developers and researchers involved in those projects.

• lmsys FastChat https://github.com/lm-sys/FastChat/tree/main
• Facebook LLaMA: https://github.com/facebookresearch/llama

Citation

If you use the data or code from this project, please declare the reference:

@misc{ChatPsychiatrist2023,
    author={EmoCareAI},
    title={ChatPsychiatrist},
    year={2023},
    publisher={GitHub},
    journal={GitHub repository},
    howpublished={\url{https://github.com/EmoCareAI/ChatPsychiatrist}},
}

@misc{zheng2023judging,
      title={Judging LLM-as-a-judge with MT-Bench and Chatbot Arena},
      author={Lianmin Zheng and Wei-Lin Chiang and Ying Sheng and Siyuan Zhuang and Zhanghao Wu and Yonghao Zhuang and Zi Lin and Zhuohan Li and Dacheng Li and Eric. P Xing and Hao Zhang and Joseph E. Gonzalez and Ion Stoica},
      year={2023},
      eprint={2306.05685},
      archivePrefix={arXiv},
      primaryClass={cs.CL}
}