Home

Awesome

Observability for your Azure OpenAI instance

Overview and goals

❗️ This project is in "beta"! Please re-review all deployment parameters, code, and queries before using in a production scenario.

This project aims to create a simple and easy to deploy solution to add observability to your Azure OpenAI instance. The approach adds an API Management instance as a proxy for your existing Azure OpenAI service, and funnels logs / requests / responses to an Application Insights instance. Additionally, a prebuilt query is saved to a workbook for easy access to logs.

Demo

Usage instructions

az group create --name loggerTest --location eastus
az deployment group create --resource-group loggerTest \
--template-file ./main.bicep \
--parameters openAiEndpoint="https://your-instance-hostname.openai.azure.com" \
openAiApiKey="your-openai-api-key"
// example Javascript code to call your Azure OpenAI instance
const { Configuration } = require("openai");

// add your APIM Subscription Key
const apiKey = process.env.MY_APIM_API_KEY;

config = new Configuration({
  // replace endpoint with your new API Management instance endpoint
  basePath: `https://${APIM_ENDPOINT}/openai/deployments/${OPENAI_DEPLOYMENT_NAME}`,

  // be sure to add headers!
  baseOptions: {
    headers: { "api-key": apiKey },
    params: { "api-version": "2023-07-01-preview" },
  },
});

Advanced usage

With your OpenAI calls, you can provide any amount of custom headers to track usage. For example, tracking how many requests a user makes, or how many calls are on a certain plan, or from a region, or by application.

Provide the custom-headers attribute as a string inside of the headers object:


config = new Configuration({
  basePath: `https://${APIM_ENDPOINT}/openai/deployments/${OPENAI_DEPLOYMENT_NAME}`,

  baseOptions: {
    headers: { 
      "api-key": apiKey,
       "custom-headers": JSON.stringify({
            user: "a_unique_id",
            planId: "your_plan_id",
            region: "your_region",
            application: "my-web-app",
            appVersion: "1.0.0"
          }),
    },
    params: { "api-version": "2023-07-01-preview" },
  },
});

As requests come in, the property dimension dropdown will be populated by the attributes of your custom headers. The values will aggregate the requests that came in with those values.

Custom headers analytics

Debugging issues

Architecture footprint

The main components of the architecture include:

Key considerations

There are several important considerations and potential issues to be aware of: