Home

Awesome

<h1 align="center" style="border-bottom: none"> <div> <a href="https://www.comet.com/site/products/opik?from=llm&utm_source=opik&utm_medium=github&utm_content=header_img&utm_campaign=opik"><picture> <source media="(prefers-color-scheme: dark)" srcset="/apps/opik-documentation/documentation/static/img/logo-dark-mode.svg"> <source media="(prefers-color-scheme: light)" srcset="/apps/opik-documentation/documentation/static/img/opik-logo.svg"> <img alt="Comet Opik logo" src="/apps/opik-documentation/documentation/static/img/opik-logo.svg" width="200" /> </picture></a> <br> Opik </div> Open-source end-to-end LLM Development Platform<br> </h1> <p align="center"> Confidently evaluate, test and monitor LLM applications.  </p> <div align="center">

Python SDK License Build <a target="_blank" href="https://colab.research.google.com/github/comet-ml/opik/blob/master/apps/opik-documentation/documentation/docs/cookbook/opik_quickstart.ipynb">

<!-- <img src="https://colab.research.google.com/assets/colab-badge.svg" alt="Open Quickstart In Colab"/> --> </a> </div> <p align="center"> <a href="https://www.comet.com/site/products/opik?from=llm&utm_source=opik&utm_medium=github&utm_content=website_button&utm_campaign=opik"><b>Website</b></a> • <a href="https://chat.comet.com"><b>Slack community</b></a> • <a href="https://x.com/Cometml"><b>Twitter</b></a> • <a href="https://www.comet.com/docs/opik/?from=llm&utm_source=opik&utm_medium=github&utm_content=docs_button&utm_campaign=opik"><b>Documentation</b></a> • <a href="https://github.com/comet-ml/opik/discussions"><b>Feature Requests</b></a> </p>

Opik thumbnail

🚀 What is Opik?

Opik is an open-source platform for evaluating, testing and monitoring LLM applications. Built by Comet.

<br>

You can use Opik for:

[!TIP]
If you are looking for features that Opik doesn't have today, please raise a new Github discussion topic 🚀

<br>

🛠️ Installation

Opik is available as a fully open source local installation or using Comet.com as a hosted solution. The easiest way to get started with Opik is by creating a free Comet account at comet.com.

If you'd like to self-host Opik, you can do so by cloning the repository and starting the platform using Docker Compose:

# Clone the Opik repository
git clone https://github.com/comet-ml/opik.git

# Navigate to the opik/deployment/docker-compose directory
cd opik/deployment/docker-compose

# Start the Opik platform
docker compose up --detach

# You can now visit http://localhost:5173 on your browser!

For more information about the different deployment options, please see our deployment guides:

Installation methodsDocs link
Local instanceLocal Deployment
KubernetesKubernetes

🏁 Get Started

To get started, you will need to first install the Python SDK:

pip install opik

Once the SDK is installed, you can configure it by running the opik configure command:

opik configure

This will allow you to configure Opik locally by setting the correct local server address or if you're using the Cloud platform by setting the API Key

[!TIP]
You can also call the opik.configure(use_local=True) method from your Python code to configure the SDK to run on the local installation.

You are now ready to start logging traces using the Python SDK.

📝 Logging Traces

The easiest way to get started is to use one of our integrations. Opik supports:

IntegrationDescriptionDocumentationTry in Colab
OpenAILog traces for all OpenAI LLM callsDocumentationOpen Quickstart In Colab
LiteLLMCall any LLM model using the OpenAI formatDocumentationOpen Quickstart In Colab
LangChainLog traces for all LangChain LLM callsDocumentationOpen Quickstart In Colab
LangGraphLog traces for all LangGraph executionsDocumentationOpen Quickstart In Colab
GeminiLog traces for all Gemini LLM callsDocumentationOpen Quickstart In Colab
GroqLog traces for all Groq LLM callsDocumentationOpen Quickstart In Colab
LlamaIndexLog traces for all LlamaIndex LLM callsDocumentationOpen Quickstart In Colab
OllamaLog traces for all Ollama LLM callsDocumentationOpen Quickstart In Colab
PredibaseFine-tune and serve open-source Large Language ModelsDocumentationOpen Quickstart In Colab
RagasEvaluation framework for your Retrieval Augmented Generation (RAG) pipelinesDocumentationOpen Quickstart In Colab
watsonxLog traces for all watsonx LLM callsDocumentationOpen Quickstart In Colab

[!TIP]
If the framework you are using is not listed above, feel free to open an issue or submit a PR with the integration.

If you are not using any of the frameworks above, you can also using the track function decorator to log traces:

import opik

opik.configure(use_local=True) # Run locally

@opik.track
def my_llm_function(user_question: str) -> str:
    # Your LLM code here

    return "Hello"

[!TIP]
The track decorator can be used in conjunction with any of our integrations and can also be used to track nested function calls.

🧑‍⚖️ LLM as a Judge metrics

The Python Opik SDK includes a number of LLM as a judge metrics to help you evaluate your LLM application. Learn more about it in the metrics documentation.

To use them, simply import the relevant metric and use the score function:

from opik.evaluation.metrics import Hallucination

metric = Hallucination()
score = metric.score(
    input="What is the capital of France?",
    output="Paris",
    context=["France is a country in Europe."]
)
print(score)

Opik also includes a number of pre-built heuristic metrics as well as the ability to create your own. Learn more about it in the metrics documentation.

🔍 Evaluating your LLM Application

Opik allows you to evaluate your LLM application during development through Datasets and Experiments.

You can also run evaluations as part of your CI/CD pipeline using our PyTest integration.

🤝 Contributing

There are many ways to contribute to Opik:

To learn more about how to contribute to Opik, please see our contributing guidelines.