<p align="center"> <picture> <source alt="cometLLM" media="(prefers-color-scheme: dark)" srcset="https://github.com/comet-ml/comet-llm/raw/main/logo-dark.svg"> <img alt="cometLLM" src="https://github.com/comet-ml/comet-llm/raw/main/logo.svg"> </picture> </p> <p align="center"> <a href="https://pypi.org/project/comet-llm"> <img src="https://img.shields.io/pypi/v/comet-llm" alt="PyPI version"> </a> <a rel="nofollow" href="https://opensource.org/license/mit/"> <img alt="GitHub" src="https://img.shields.io/badge/License-MIT-blue.svg"> </a> <a href="https://www.comet.com/docs/v2/guides/large-language-models/overview/" rel="nofollow"> <img src="https://img.shields.io/badge/cometLLM-Docs-blue.svg" alt="cometLLM Documentation"> </a> <a rel="nofollow" href="https://pepy.tech/project/comet-llm"> <img style="max-width: 100%;" src="https://static.pepy.tech/badge/comet-llm" alt="Downloads"> </a> <a rel="nofollow" href="https://colab.research.google.com/github/comet-ml/comet-llm/blob/main/examples/CometLLM_Prompts.ipynb"> <img src="https://colab.research.google.com/assets/colab-badge.svg"> </a> </p> <p align="center"> <b>CometLLM</b> is a tool to log and visualize your LLM prompts and chains. Use CometLLM to identify effective prompt strategies, streamline your troubleshooting, and ensure reproducible workflows! </p> </p>

CometLLM Preview

⚡️ Quickstart

Install comet_llm Python library with pip:

pip install comet_llm

If you don't have already, create your free Comet account and grab your API Key from the account settings page.

Now you are all set to log your first prompt and response:

import comet_llm

    prompt="What is your name?",
    output=" My name is Alex.",

🎯 Features

👀 Examples

To log a single LLM call as an individual prompt, use comet_llm.log_prompt. If you require more granularity, you can log a chain of executions that may include more than one LLM call, context retrieval, or data pre- or post-processing with comet_llm.start_chain.

Log a full prompt and response

import comet_llm

    prompt="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: What is your name?\nAnswer:",
    prompt_template="Answer the question and if the question can't be answered, say \"I don't know\"\n\n---\n\nQuestion: {{question}}?\nAnswer:",
    prompt_template_variables={"question": "What is your name?"},
    metadata= {
        "usage.prompt_tokens": 7,
        "usage.completion_tokens": 5,
        "usage.total_tokens": 12,
    output=" My name is Alex.",

Read the full documentation for more details about logging a prompt.

Log a LLM chain

from comet_llm import Span, end_chain, start_chain
import datetime
from time import sleep

def retrieve_context(user_question):
    if "open" in user_question:
        return "Opening hours: 08:00 to 17:00 all days"

def llm_answering(user_question, current_time, context):
    prompt_template = """You are a helpful chatbot. You have access to the following context:
    The current time is: {current_time}
    Analyze the following user question and decide if you can answer it, if the question can't be answered, say \"I don't know\":

    prompt = prompt_template.format(
        user_question=user_question, current_time=current_time, context=context

    with Span(
        inputs={"prompt_template": prompt_template, "prompt": prompt},
    ) as span:
        # Call your LLM model here
        result = "Yes we are currently open"
        usage = {"prompt_tokens": 52, "completion_tokens": 12, "total_tokens": 64}

        span.set_outputs(outputs={"result": result}, metadata={"usage": usage})

    return result

def main(user_question, current_time):
    start_chain(inputs={"user_question": user_question, "current_time": current_time})

    with Span(
        name="Retrieve Context",
        inputs={"user_question": user_question},
    ) as span:
        context = retrieve_context(user_question)

        span.set_outputs(outputs={"context": context})

    with Span(
            "user_question": user_question,
            "current_time": current_time,
            "context": context,
    ) as span:
        result = llm_answering(user_question, current_time, context)

        span.set_outputs(outputs={"result": result})

    end_chain(outputs={"result": result})

main("Are you open?", str(datetime.datetime.now().time()))

Read the full documentation for more details about logging a chain.

⚙️ Configuration

You can configure your Comet credentials and where you are logging data to:

NamePython parameter nameEnvironment variable name
Comet Workspace nameworkspaceCOMET_WORKSPACE
Comet Project nameprojectCOMET_PROJECT_NAME

📝 License

Copyright (c) Comet 2023-present. cometLLM is free and open-source software licensed under the MIT License.