Awesome

AI Gateway

Reliably route to 200+ LLMs with 1 fast & friendly API

<img src="docs/images/demo.gif" width="650" alt="Gateway Demo"><br>

The AI Gateway streamlines requests to 250+ language, vision, audio and image models with a unified API. It is production-ready with support for caching, fallbacks, retries, timeouts, loadbalancing, and can be edge-deployed for minimum latency.

✅  Blazing fast (9.9x faster) with a tiny footprint (~100kb build) <br> ✅  Load balance across multiple models, providers, and keys <br> ✅  Fallbacks make sure your app stays resilient <br> ✅  Automatic Retries with exponential fallbacks come by default <br> ✅  Configurable Request Timeouts to easily handle unresponsive LLM requests <br> ✅  Multimodal to support routing between Vision, TTS, STT, Image Gen, and more models <br> ✅  Plug-in middleware as needed <br> ✅  Battle tested over 480B tokens <br> ✅  Enterprise-ready for enhanced security, scale, and custom deployments <br><br>

[!TIP] ⭐️ Star this repo to get Github release notifications for new provider integrations and features.

Setup & Installation

Use the AI gateway through the hosted API or self-host the open-source or enterprise versions on your environment. <br>

👉 Hosted Gateway on portkey.ai (Fastest)

The hosted API is the fastest way to setup an AI Gateway for your Gen AI application. We process billions of tokens daily and is in production with companies like Postman, Haptik, Turing, MultiOn, SiteGPT, and more.

<a href="https://app.portkey.ai/signup"><img src="https://portkey.ai/blog/content/images/2024/08/Get-API-Key--3-.png" height=50 alt="Get API Key" /></a><br> <br>

👉 Self-hosting the OSS version (MIT License)

To run the AI gateway locally, execute the following command in your terminal. (Needs npx installed) Or, explore deployment guides for Cloudflare, Docker, Node.js and more here.

npx @portkey-ai/gateway

^{Your AI Gateway is now running on http://localhost:8787 🚀} <br>

👉 Self-hosting the Enterprise Version

The AI Gateway's enterprise version offers enterprise-ready capabilities for org management, governance, security and more out of the box. Compare the open source, hosted and enterprise versions here.

The enterprise deployment architecture, supported platforms is available here - Enterprise Private Cloud Deployments

<a href="https://portkey.sh/demo-22"><img src="https://portkey.ai/blog/content/images/2024/08/Get-API-Key--5-.png" height=50 alt="Book an enterprise AI gateway demo" /></a><br>

Making requests through the AI gateway

<img src="docs/images/openai.png" height=15 /> Compatible with OpenAI API & SDKs

The AI Gateway is compatible with the OpenAI API & SDKs, and extends them to call 200+ LLMs reliably. To use the Gateway through OpenAI, update the client to include the gateway's URL and headers and make requests as usual. The AI gateway can translate requests written in the OpenAI format to the signature expected by the specified provider. View examples <br><br>

<img src="https://upload.wikimedia.org/wikipedia/commons/thumb/c/c3/Python-logo-notext.svg/1869px-Python-logo-notext.svg.png" height=15 /> Using the Python SDK   <a href="https://colab.research.google.com/drive/1hLvoq_VdGlJ_92sPPiwTznSra5Py0FuW?usp=sharing"><img src="https://colab.research.google.com/assets/colab-badge.svg"></a>

Portkey Python SDK is a wrapper over the OpenAI Python SDK with added support for additional parameters across all other providers. If you're building with Python, this is the recommended library to connect to the Gateway.

pip install -qU portkey-ai

<img src="https://cdn-icons-png.flaticon.com/512/5968/5968322.png" height=15 /> Using the Node.JS SDK

Portkey JS/TS SDK is a wrapper over the OpenAI JS SDK with added support for additional parameters across all other providers. If you're building with JS or TS, this is the recommended library to connect to the Gateway.

npm install --save portkey-ai

<img src="https://www.svgrepo.com/show/305922/curl.svg" height=15 /> Using the REST APIs

The AI gateway supports OpenAI compatible endpoints with added parameter support for all other providers and models. View API Reference. <br><br>

Other Integrations

Gateway Cookbooks

Trending Cookbooks

• Use models from Nvidia NIM with AI Gateway
• Monitor CrewAI Agents with Portkey!
• Comparing Top 10 LMSYS Models with AI Gateway.

Latest Cookbooks

• Create Synthetic Datasets using Nemotron
• Use Portkey Gateway with Vercel's AI SDK
• Monitor Llama Agents with Portkey

More Examples

Supported Providers

Explore Gateway integrations with 25+ providers and 6+ frameworks.

View the complete list of 200+ supported models here

<br>

Agents

Gateway seamlessly integrates with popular agent frameworks. Read the documentation here.

*Only available on the hosted app. For detailed documentation click here.

Features

These features are configured through the Gateway Config added to the x-portkey-config header or the config parameter in the SDKs.

Here's a sample config JSON showcasing the above features. All the features are optional

{
	"retry": { "attempts": 5 },
	"request_timeout": 10000,
	"strategy": { "mode": "fallback" }, // or 'loadbalance', etc
	"targets": [{
		"provider": "openai",
		"api_key": "sk-***"
	},{
		"strategy": {"mode": "loadbalance"}, // Optional nesting
		"targets": {...}
	}]
}

Then use the config in your API requests to the gateway.

Using Gateway Configs

Here's a guide to use the config object in your request.

Gateway Enterprise Version

Make your AI app more <ins>reliable</ins> and <ins>forward compatible</ins>, while ensuring complete <ins>data security</ins> and <ins>privacy</ins>.

✅  Secure Key Management - for role-based access control and tracking <br> ✅  Simple & Semantic Caching - to serve repeat queries faster & save costs <br> ✅  Access Control & Inbound Rules - to control which IPs and Geos can connect to your deployments <br> ✅  PII Redaction - to automatically remove sensitive data from your requests to prevent inadvertent exposure <br> ✅  SOC2, ISO, HIPAA, GDPR Compliances - for best security practices <br> ✅  Professional Support - along with feature prioritization <br>

Schedule a call to discuss enterprise deployments

Contributing

The easiest way to contribute is to pick an issue with the good first issue tag 💪. Read the contribution guidelines here.

Bug Report? File here | Feature Request? File here

Community

Join our growing community around the world, for help, ideas, and discussions on AI.

• View our official Blog
• Chat with us on Discord
• Follow us on Twitter
• Connect with us on LinkedIn
• Read the documentation in Japanese