Home

Awesome

Auto-evaluator :brain: :memo:

Context

Document Question-Answering is a popular LLM use-case. LangChain makes it easy to assemble LLM components (e.g., models and retrievers) into chains that support question-answering: input documents are split into chunks and stored in a retriever, relevant chunks are retrieved given a user question and passed to an LLM for synthesis into an answer.

Challenge

The quality of QA systems can vary considerably; for example, we have seen cases of hallucination and poor answer quality due specific parameter settings. But, it is not always obvious to (1) evaluate the answer quality in a systematic way and (2) use this evaluation to guide improved QA chain settings (e.g., chunk size) or components (e.g., model or retriever choice).

App overview

This app aims to address the above limitations. Recent work from Anthropic has used model-written evaluation sets. OpenAI and others have shown that model-graded evaluation is an effective way to evaluate models. This app combines both of these ideas into a single workspace, auto-generating a QA test set and auto-grading the result of the specified QA chain.

image

Usage

The app can be used in two ways:

image

image

Building the document retrieval:

Test set generation:

LLM question-answering:

Model-graded evaluation:

(1) The app will evaluate the relevance of the retrieved documents relative to the question.

(2) The app will evaluate the similarity of the LLM generated answer relative to ground truth answer.

Experimental results:

image

User inputs

The left panel of the app (shown in red in the above image) has several user-configurable parameters.

Number of eval questions - This is the number of question-answer pairs to auto-generate for the given inputs documents. As mentioned above, question-answer pair auto-generation will use Langchain's QAGenerationChain with prompt specified here.

Chunk size - Number of characters per chunk when the input documents are split. This can impact answer quality. Retrievers often use text embedding similarity to select chunks related to the question. If the chunks are too large, each chunk may contain more information unrelated to the question, which may degrade the summarized answer quality. If chunks are too small, important context may be left out of the retrieved chunks.

Overlap - The overlap in characters between chunks.

Embedding - The method used to embed chunks.

Retriever - The method used to retrieve chunks that are relevant to the user question. The default vector database used for similarity search is FAISS, but support for others is a welcome addition. You can also try other methods, such as SVM or TF-IDF.

Number of chunks to retrieve - Number of chunks retrieved. More chunks can improve performance by giving the LLM more context for answer summarization.

Model - LLM for summarization of retrieved chunks into the answer.

Grading prompt style - The prompt choice for model-graded evaluation. As mentioned above, the prompts can be seen here. More prompts would be a welcome addition. For example, with the Descriptive prompt, you will see a more detailed output with model grade justification.

Logging experiments

A user can select the desired configuration and then choose Re-Run Experiment.

This will run the new chain on the existing test set.

The results from all experiments will be summarized in the table and chart.

image

Contributing

Run the backend from api folder:

 pip install -r requirements.txt
 uvicorn evaluator_app:app

Test the api locally:

curl -X POST -F "files=@docs/karpathy-lex-pod/karpathy-pod.txt" -F "num_eval_questions=1" -F "chunk_chars=1000" -F "overlap=100" -F "split_method=RecursiveTextSplitter" -F "retriever_type=similarity-search" -F "embeddings=OpenAI" -F "model_version=gpt-3.5-turbo" -F "grade_prompt=Fast" -F "num_neighbors=3" http://localhost:8000/evaluator-stream

Run the frontend from nextjs folder and view web app at specified URL (e.g., http://localhost:3000/):

yarn install
yarn dev

Environment Variables

Front-end:

.env.local contains the env variables needed to run the project.

Back-end:

Specify the API keys for any models that you want to use.

OPENAI_API_KEY=
ANTHROPIC_API_KEY=

Deployment

The front-end is deployed to Vercel.

The back-end is deployed to Railway.