Home

Awesome

<!--BEGIN_BANNER_IMAGE--> <picture> <source media="(prefers-color-scheme: dark)" srcset="/.github/banner_dark.png"> <source media="(prefers-color-scheme: light)" srcset="/.github/banner_light.png"> <img style="width:100%;" alt="The LiveKit icon, the name of the repository and some sample code in the background." src="https://raw.githubusercontent.com/livekit/agents/main/.github/banner_light.png"> </picture> <!--END_BANNER_IMAGE--> <!--BEGIN_DESCRIPTION-->

<br /><br /> Looking for the JS/TS library? Check out AgentsJS

✨ [NEW] OpenAI Realtime API support

We are partnering with OpenAI on a new MultimodalAgent API in the Agents framework. This class completely wraps OpenAI’s Realtime API, abstracts away the raw wire protocol, and provide an ultra-low latency WebRTC transport between GPT-4o and your users’ devices. This same stack powers Advanced Voice in the ChatGPT app.

What is Agents?

The Agents framework allows you to build AI-driven server programs that can see, hear, and speak in realtime. Your agent connects with end user devices through a LiveKit session. During that session, your agent can process text, audio, images, or video streaming from a user's device, and have an AI model generate any combination of those same modalities as output, and stream them back to the user.

Features

<!--END_DESCRIPTION-->

Installation

To install the core Agents library:

pip install livekit-agents

Plugins

The framework includes a variety of plugins that make it easy to process streaming input or generate output. For example, there are plugins for converting text-to-speech or running inference with popular LLMs. Here's how you can install a plugin:

pip install livekit-plugins-openai

The following plugins are available today:

PluginFeatures
livekit-plugins-anthropicLLM
livekit-plugins-assemblyaiSTT
livekit-plugins-azureSTT, TTS
livekit-plugins-deepgramSTT
livekit-plugins-cartesiaTTS
livekit-plugins-elevenlabsTTS
livekit-plugins-playhtTTS
livekit-plugins-googleSTT, TTS
livekit-plugins-nltkUtilities for working with text
livekit-plugins-ragUtilities for performing RAG
livekit-plugins-openaiLLM, STT, TTS, Assistants API, Realtime API
livekit-plugins-sileroVAD

Documentation and guides

Documentation on the framework and how to use it can be found here

Example agents

DescriptionDemo LinkCode Link
A basic voice agent using a pipeline of STT, LLM, and TTSdemocode
Voice agent using the new OpenAI Realtime APIdemocode
Super fast voice agent using Cerebras hosted Llama 3.1democode
Voice agent using Cartesia's Sonic modeldemoN/A
Agent that looks up the current weather via function callN/Acode
Voice agent that performs a RAG-based lookupN/Acode
Video agent that publishes a stream of RGB framesN/Acode
Transcription agent that generates text captions from a user's speechN/Acode
A chat agent you can text who will respond back with generated speechN/Acode
Localhost multi-agent conference callN/Acode
Moderation agent that uses Hive to detect spam/abusive videoN/Acode

Contributing

The Agents framework is under active development in a rapidly evolving field. We welcome and appreciate contributions of any kind, be it feedback, bugfixes, features, new plugins and tools, or better documentation. You can file issues under this repo, open a PR, or chat with us in LiveKit's Slack community.

<!--BEGIN_REPO_NAV-->

<br/><table>

<thead><tr><th colspan="2">LiveKit Ecosystem</th></tr></thead> <tbody> <tr><td>Realtime SDKs</td><td><a href="https://github.com/livekit/client-sdk-js">Browser</a> · <a href="https://github.com/livekit/client-sdk-swift">iOS/macOS/visionOS</a> · <a href="https://github.com/livekit/client-sdk-android">Android</a> · <a href="https://github.com/livekit/client-sdk-flutter">Flutter</a> · <a href="https://github.com/livekit/client-sdk-react-native">React Native</a> · <a href="https://github.com/livekit/rust-sdks">Rust</a> · <a href="https://github.com/livekit/node-sdks">Node.js</a> · <a href="https://github.com/livekit/python-sdks">Python</a> · <a href="https://github.com/livekit/client-sdk-unity">Unity</a> · <a href="https://github.com/livekit/client-sdk-unity-web">Unity (WebGL)</a></td></tr><tr></tr> <tr><td>Server APIs</td><td><a href="https://github.com/livekit/node-sdks">Node.js</a> · <a href="https://github.com/livekit/server-sdk-go">Golang</a> · <a href="https://github.com/livekit/server-sdk-ruby">Ruby</a> · <a href="https://github.com/livekit/server-sdk-kotlin">Java/Kotlin</a> · <a href="https://github.com/livekit/python-sdks">Python</a> · <a href="https://github.com/livekit/rust-sdks">Rust</a> · <a href="https://github.com/agence104/livekit-server-sdk-php">PHP (community)</a></td></tr><tr></tr> <tr><td>UI Components</td><td><a href="https://github.com/livekit/components-js">React</a> · <a href="https://github.com/livekit/components-android">Android Compose</a> · <a href="https://github.com/livekit/components-swift">SwiftUI</a></td></tr><tr></tr> <tr><td>Agents Frameworks</td><td><b>Python</b> · <a href="https://github.com/livekit/agents-js">Node.js</a> · <a href="https://github.com/livekit/agent-playground">Playground</a></td></tr><tr></tr> <tr><td>Services</td><td><a href="https://github.com/livekit/livekit">LiveKit server</a> · <a href="https://github.com/livekit/egress">Egress</a> · <a href="https://github.com/livekit/ingress">Ingress</a> · <a href="https://github.com/livekit/sip">SIP</a></td></tr><tr></tr> <tr><td>Resources</td><td><a href="https://docs.livekit.io">Docs</a> · <a href="https://github.com/livekit-examples">Example apps</a> · <a href="https://livekit.io/cloud">Cloud</a> · <a href="https://docs.livekit.io/home/self-hosting/deployment">Self-hosting</a> · <a href="https://github.com/livekit/livekit-cli">CLI</a></td></tr> </tbody> </table> <!--END_REPO_NAV-->