Awesome

OpenResearcher: Unleashing AI for Accelerated Scientific Research

This is the official repository for OpenResearcher.

🔥News

• [2024/10/06] Our paper has been accepted by EMNLP Demo Track 2024! 🎉

📋 Table of Contents

• Introduction
• Performance
• Get started
  • Setup
  • Supported models
    • Using Api
    • Using Opensource LLMs
  • Process Data to embeddings
  • Usage
• Citation

📝 Introduction

🏆 Performance

We release the benchmarking results on various RAG-related systems as a leaderboard.

We used human experts to evaluate the responses from various RAG systems. If one answer was significantly better than another, it was judged as a win for the former and a lose for the latter. If the two answers were similar, it was considered a tie.

GPT-4 Preference Results compared with Perplexity AI outcome.

🚀 Get Started

🛠️ Setup <a name="setup"></a>

Install necessary packages:

To begin using OpenResearcher, you need to install the required dependencies. You can do this by running the following command:

git clone https://github.com/GAIR-NLP/OpenResearcher.git 
conda create -n openresearcher python=3.10 
conda activate openresearcher
cd OpenResearcher
pip install -r requirements.txt

Install Qdrant vector search engine:

First, download the latest Qdrant image from Dockerhub:

docker pull qdrant/qdrant

Then, run the service:

docker run -p 6333:6333 -p 6334:6334 \
    -v $(pwd)/qdrant_storage:/qdrant/storage:z \
    qdrant/qdrant

For more Qdrant installation details, you can follow this link.

Install Elasticsearch:

You can follow this link to install Elasticsearch with docker.

🤖 Supported models

OpenResearcher currently supports API models from OpenAI, Deepseek, and Aliyun, as well as most huggingface models supported by vllm.

Using API:

Modify the API and base URL values in the config.py file located in the root directory to use large language model service platforms that support the OpenAI interface

For example, if you use Deepseek as an API provider, and then modify the following value in config.py::

...
openai_api_base_url = "https://api.deepseek.com/v1"
openai_api_key = "api key here"
...

Using Opensource LLMs:

Please use vllm to set up the API server for open-source LLMs. For example, use the following command to deploy a Llama 3 70B hosted on HuggingFace:

python -m vllm.entrypoints.openai.api_server \
  --model meta-llama/Meta-Llama-3-70B-Instruct \
  --tensor-parallel-size 8 \
  --dtype auto \
  --api-key sk-dummy \
  --gpu-memory-utilization 0.9 \
  --port 5000

Then we can initialize the chat-llm with config.py:

...
openai_api_base_url = "http://localhost:5000/v1"
openai_api_key = "sk-dummy"
...

Enable Web search:

We currently support Bing Search in OpenResearcher. Modify the following value in config.py:

...
bing_search_key = "api key here"
bing_search_end_point = "https://api.bing.microsoft.com/"
...

📊 Process Data to embeddings

Indexing and Saving in Qdrant

1. Download arXiv data (html file) and metadata into the /data

​ arXiv data refers to https://info.arxiv.org/help/bulk_data/index.html

​ Metadata refers to https://www.kaggle.com/datasets/Cornell-University/arxiv

The directory of datais formatted as follows:

   - data/
     - 2401/  # pub date   
       - 2401.00001/  # paper id    
         - doc.html   # paper content 
       - 2401.00002/
         - doc.html
     - 2402/
    ...
     -arxiv-metadata-oai-snapshot.jsonl   # metadata        

2. Parse the html data to Qdrant vector

CUDA_VISIBLE_DEVICES=0 python -um connector.html_parsing --target_dir /path/to/target/directory --start_index 0 --end_index -1 \
--meta_data_path /path/to/metadata/file

Parameter explanation:

​ target_dir: process the 'target_dir' papers

​ start_index,end_index: papers in directory from 'start_index' to 'end_index' will be processed

​ meta_data_path: metadata saved path

3. Parse the paper's metadata to Elastic search

CUDA_VISIBLE_DEVICES=0 python -um connector.meta_elastic --meta_data_path /path/to/metadata/file \
--chunk_size 512 --embed_batch_size 32

Parameter explanation:

​ meta_data_path: metadata saved path

​ chunk_size: The chunk length of the text

​ embed_batch_size: vectorized batch size, you can adjust this parameter according to the size of the GPU memory

📘 Usage

Run the RAG application

First, run the Qdrant retriever server:

python -um utils.async_qdrant_retriever

Then run the Elastic Search retriever server:

python -um utils.async_elasticsearch_retriever

Then you can run the OpenResearcher system by following the command:

 CUDA_VISIBLE_DEVICES=0 streamlit run ui_app.py

📚 Citation

If this work is helpful, please kindly cite as:

@article{zheng2024openresearcher,
  title={OpenResearcher: Unleashing AI for Accelerated Scientific Research},
  author={Zheng, Yuxiang and Sun, Shichao and Qiu, Lin and Ru, Dongyu and Jiayang, Cheng and Li, Xuefeng and Lin, Jifan and Wang, Binjie and Luo, Yun and Pan, Renjie and others},
  journal={arXiv preprint arXiv:2408.06941},
  year={2024}
}