Awesome

Colivara Evaluation Project

This repository contains a comprehensive evaluation of the Colivara API for document management, search, and retrieval, using a Retrieval-Augmented Generation (RAG) model. This evaluation aims to assess Colivara's capabilities in managing document collections, performing efficient search operations, and calculating relevance metrics to measure performance.

Table of Contents

• Project Overview
• Evaluation Results
• Features
• Requirements
• Installation
• Usage
  • Document Upsert with upsert.py
  • Relevance Evaluation with evaluate.py
  • Collection Management with collection_manager.py
• File Structure
• Configuration
• Technical Details
• Future Enhancements
• License

Project Overview

The goal of this project is to evaluate Colivara’s document retrieval and management features, particularly for applications that rely on high-performance data search and retrieval. This includes testing Colivara's collection and document management, assessing its suitability for various search and retrieval scenarios, and benchmarking the platform with a RAG model to evaluate relevance based on real-world queries.

Evaluation Results

Below are the summarized evaluation results for the Colivara API performance based on NDCG metrics:

Features

• Data Loading: Load document datasets in a structured format for evaluation, with support for processing metadata and converting images to base64.
• Document Management: Manage collections and documents, including creation, updating, and deletion.
• RAG Model Evaluation: Use NDCG (Normalized Discounted Cumulative Gain) to evaluate the relevance of search results.
• Collection Management Tool: A utility for listing, creating, and deleting collections in Colivara.
• Comprehensive Configurations: Load configurations from environment variables for easy setup and deployment.

Requirements

• Python 3.10+
• Colivara API (configured and accessible) either self-hosted locally or using the hosted version.
• colivara-py Python client

Installation

1. Clone the repository:

   git clone https://github.com/tjmlabs/colivara-eval.git
   cd colivara-eval
   

2. Install the dependencies:

   uv venv
   source venv/bin/activate
   uv sync
   

3. Configure Environment Variables:

   • Create a .env file in the root directory.
   • Add the following variables:

     COLIVARA_API_KEY=your_api_key_here
     COLIVARA_BASE_URL=https://api.colivara.com
     

4. Download the Dataset:

   • Download the dataset file(s) for evaluation.
   • Run the following command:

   python src/download_datasets.py
   

Usage

The Colivara Evaluation Project provides a streamlined interface for managing and evaluating document collections within Colivara. The primary entry points for usage are upsert.py for performing document upsert operations and evaluate.py for relevance evaluation.

Document Upsert with upsert.py

The upsert.py script enables you to upsert documents into Colivara collections. It allows selective processing of single datasets or batch processing across all available datasets, making it adaptable for various scenarios.

Key Arguments

• --n_rows: Specify the number of rows to load from the dataset for processing. This is optional; if not provided, the script will load all rows.
• --upsert: Include this flag if you want to upsert documents into Colivara.
• --all_files: Processes all datasets in the DOCUMENT_FILES list.
• --specific_file: Specify a single file to process by name (must match one of the files in DOCUMENT_FILES).
• --collection_name: Use this to define a custom collection name when processing a specific file. If not provided, the script defaults to the predefined collection name for that file.

Example Commands

1. Upserting a Single Dataset

To upsert documents from a specific dataset, run:

python upsert.py --specific_file arxivqa_test_subsampled.pkl --collection_name arxivqa_collection --upsert

This command will upsert all documents from arxivqa_test_subsampled.pkl into arxivqa_collection if it doesn’t already exist.

2. Upserting All Datasets

To upsert documents for all datasets:

python upsert.py --all_files --upsert

This command will loop through all datasets in DOCUMENT_FILES, upserting documents into their corresponding collections.

Relevance Evaluation with evaluate.py

The evaluate.py script is used to evaluate the relevance of document collections within Colivara.

Key Arguments

• --collection_name: Specify the collection name to evaluate.
• --all_files: Evaluate all collections listed in DOCUMENT_FILES.

Example Commands

1. Evaluating a Single Collection

To evaluate the relevance of a specific collection, run:

python evaluate.py  --collection_name arxivqa_test_subsampled

This command will evaluate the specified collection and output the relevance metrics based on NDCG@5. Here is a list of our collection names that are aleady uploaded.

COLLECTION_NAMES = [
    "arxivqa_test_subsampled",
    "docvqa_test_subsampled",
    "infovqa_test_subsampled",
    "shiftproject_test",
    "syntheticDocQA_artificial_intelligence_test",
    "syntheticDocQA_energy_test",
    "syntheticDocQA_government_reports_test",
    "syntheticDocQA_healthcare_industry_test",
    "tabfquad_test_subsampled",
    "tatdqa_test",
]

2. Evaluating All Collections

To evaluate the relevance of all collections:

python evaluate.py --all_files

This command will perform a relevance evaluation (NDCG@5) on all datasets listed in DOCUMENT_FILES and save the results in the out/ directory:

• out/avg_ndcg_scores.pkl – Contains the average NDCG@5 score for each dataset.
• out/ndcg_scores.pkl – Provides detailed NDCG scores for each query.
• out/<collection_name>_ndcg_scores.pkl – Provides detailed NDCG scores for each query in the specified collection.

Collection Management with collection_manager.py

The collection_manager.py script provides utilities for listing and deleting collections within Colivara.

Commands

• List All Collections

  python collection_manager.py --list
  

  Displays all existing collections within Colivara.

• Delete a Collection

  python collection_manager.py --delete <collection_name>
  

  Deletes the specified collection. This action is irreversible, so ensure that the correct collection name is provided.

File Structure

• src/
  • client.py: Initializes the Colivara client.
  • data_loader.py: Handles data loading and base64 image encoding.
  • document_manager.py: Manages document upserting and collection creation.
  • evaluator.py: Evaluates model performance using NDCG.
• collection_manager.py: Provides collection listing and deletion tools.
• upsert.py: upsert script for document upsertion.
• tests/: Contains unit tests for the project.
• data/: Stores the dataset for evaluation.
• .env: Environment configuration file (not included in version control).
• requirements.txt: Lists Python package dependencies.

Configuration

The project configuration relies on environment variables defined in a .env file:

• COLIVARA_API_KEY: API key for authenticating with the Colivara service.
• COLIVARA_BASE_URL: The base URL for accessing Colivara's API.

Use dotenv to load these configurations automatically, ensuring that sensitive information is securely managed.

Technical Details

Discounted Cumulative Gain (DCG)

DCG is a measure of relevance that considers the position of relevant results in the returned list. It assigns higher scores to results that appear earlier.

Normalized Discounted Cumulative Gain (NDCG)

NDCG normalizes DCG by dividing it by the ideal DCG (IDCG) for a given query, providing a score between 0 and 1. In this project, we calculate NDCG@5 to evaluate the top 5 search results for each query.

Search Query Evaluation

The evaluation process includes:

1. Query Processing: Matching queries against document metadata.
2. Relevance Scoring: Using true document IDs to calculate relevance scores.
3. NDCG Calculation: Aggregating scores to calculate the average relevance.

Future Enhancements

1. Parallel Processing: Optimize data loading and evaluation functions for concurrent processing.
2. Extended Metrics: Add other evaluation metrics like Mean Reciprocal Rank (MRR).
3. Benchmarking with Larger Datasets: Test Colivara's scalability with larger data volumes.
4. Automated Testing: Integrate unit and integration tests for CI/CD compatibility.

License

This project is licensed under the MIT License - see the LICENSE file for details.