Home

Awesome

:warning: IMPORTANT NOTICE: This repository is no longer maintained.

<p align="center"> <h2 align="center">āš” LangChain Apps on Production with Jina & FastAPI šŸš€</h2> </p> <p align=center> <a href="https://pypi.org/project/langchain-serve/"><img alt="PyPI" src="https://img.shields.io/pypi/v/langchain-serve?label=Release&style=flat-square"></a> <a href="https://discord.jina.ai"><img src="https://img.shields.io/discord/1106542220112302130?logo=discord&logoColor=white&style=flat-square"></a> <a href="https://pypistats.org/packages/langchain-serve"><img alt="PyPI - Downloads from official pypistats" src="https://img.shields.io/pypi/dm/langchain-serve?style=flat-square"></a> <a href="https://github.com/jina-ai/langchain-serve/actions/workflows/cd.yml"><img alt="Github CD status" src="https://github.com/jina-ai/langchain-serve/actions/workflows/cd.yml/badge.svg"></a> </p>

Jina is an open-source framework for building scalable multi modal AI apps on Production. LangChain is another open-source framework for building applications powered by LLMs.

langchain-serve helps you deploy your LangChain apps on Jina AI Cloud in a matter of seconds. You can benefit from the scalability and serverless architecture of the cloud without sacrificing the ease and convenience of local development. And if you prefer, you can also deploy your LangChain apps on your own infrastructure to ensure data privacy. With langchain-serve, you can craft REST/Websocket APIs, spin up LLM-powered conversational Slack bots, or wrap your LangChain apps into FastAPI packages on cloud or on-premises.

Give us a :star: and tell us what more you'd like to see!

ā˜ļø LLM Apps as-a-service

langchain-serve currently wraps following apps as a service to be deployed on Jina AI Cloud with one command.

šŸ”® AutoGPT-as-a-service

AutoGPT is an "AI agent" that given a goal in natural language, will attempt to achieve it by breaking it into sub-tasks and using the internet and other tools in an automatic loop.

<details> <summary>Show usage</summary> </details>

šŸ§  Babyagi-as-a-service

Babyagi is a task-driven autonomous agent that uses LLMs to create, prioritize, and execute tasks. It is a general-purpose AI agent that can be used to automate a wide variety of tasks.

<details> <summary>Show usage</summary> </details>

:panda_face: pandas-ai-as-a-service

pandas-ai integrates LLM capabilities into Pandas, to make dataframes conversational in Python code. Thanks to langchain-serve, we can now expose pandas-ai APIs on Jina AI Cloud in just a matter of seconds.

<details> <summary>Show usage</summary> </details>

šŸ’¬ Question Answer Bot on PDFs

pdfqna is a simple question answering bot that uses LLMs to answer questions on PDF documents, showcasing the how easy it is to integrate langchain apps on Jina AI Cloud.

<details> <summary>Show usage</summary> </details>

šŸ’Ŗ Features

šŸŽ‰ LLM Apps on production

šŸ”„ Secure, Scalable, Serverless, Streaming REST/Websocket APIs on Jina AI Cloud.

šŸ  Self-host LLM Apps with Docker Compose or Kubernetes

šŸ§° Usage

Let's first install langchain-serve using pip.

pip install langchain-serve

šŸ”„ REST APIs using @serving decorator

šŸ‘‰ Let's go through a step-by-step guide to build, deploy and use a REST API using @serving decorator.


šŸ¤–šŸ’¬ Build, Deploy & Distribute Slack bots built with LangChain

langchain-serve exposes a @slackbot decorator to quickly build, deploy & distribute LLM-powered Slack bots without worrying about the infrastructure. It provides a simple interface to any langchain app on and makes them super accessible to users a platform they're already comfortable with.

āœØ Ready to dive in?


šŸ” Authorize your APIs

To add an extra layer of security, we can integrate any custom API authorization by adding a auth argument to the @serving decorator.

<details> <summary>Show code & gotchas</summary>
from lcserve import serving

def authorizer(token: str) -> Any:
    if not token == 'mysecrettoken':            # Change this to add your own authorization logic
        raise Exception('Unauthorized')         # Raise an exception if the request is not authorized

    return 'userid'                             # Return any user id or object

@serving(auth=authorizer)
def ask(question: str, **kwargs) -> str:
    auth_response = kwargs['auth_response']     # This will be 'userid'
    return ...

@serving(websocket=True, auth=authorizer)
async def talk(question: str, **kwargs) -> str:
    auth_response = kwargs['auth_response']     # This will be 'userid'
    return ...
šŸ¤” Gotchas about the auth function
</details>

šŸ™‹ā€ā™‚ļø Enable streaming & human-in-the-loop (HITL) with WebSockets

HITL for LangChain agents on production can be challenging since the agents are typically running on servers where humans don't have direct access. langchain-serve bridges this gap by enabling websocket APIs that allow for real-time interaction and feedback between the agent and a human operator.

Check out this example to see how you can enable HITL for your agents.

šŸ“ Persistent storage on Jina AI Cloud

Every app deployed on Jina AI Cloud gets a persistent storage (EFS) mounted locally which can be accessed via workspace kwarg in the @serving function.

<details> <summary>Show code</summary>
from lcserve import serving

@serving
def store(text: str, **kwargs):
    workspace: str = kwargs.get('workspace')
    path = f'{workspace}/store.txt'
    print(f'Writing to {path}')
    with open(path, 'a') as f:
        f.writelines(text + '\n')
    return 'OK'


@serving(websocket=True)
async def stream(**kwargs):
    workspace: str = kwargs.get('workspace')
    websocket: WebSocket = kwargs.get('websocket')
    path = f'{workspace}/store.txt'
    print(f'Streaming {path}')
    async with aiofiles.open(path, 'r') as f:
        async for line in f:
            await websocket.send_text(line)
    return 'OK'

Here, we are using the workspace to store the incoming text in a file via the REST endpoint and streaming the contents of the file via the WebSocket endpoint.

</details>

šŸš€ Bring your own FastAPI app

If you already have a FastAPI app with pre-defined endpoints, you can use lc-serve to deploy it on Jina AI Cloud.

lc-serve deploy jcloud --app filename:app 
<details> <summary>Show details</summary>

Let's take an example of a simple FastAPI app with directory structure

.
ā””ā”€ā”€ endpoints.py
# endpoints.py
from typing import Union

from fastapi import FastAPI

app = FastAPI()


@app.get("/status")
def read_root():
    return {"Hello": "World"}


@app.get("/items/{item_id}")
def read_item(item_id: int, q: Union[str, None] = None):
    return {"item_id": item_id, "q": q}
lc-serve deploy jcloud --app endpoints:app
</details>

šŸ—ļø Using Secrets during Deployment

You can use secrets during app deployment by passing a secrets file to deployment with the --secrets flag. The secrets file should be a .env file containing the secrets.

lcserve deploy jcloud app --secrets .env
<details> <summary>Show details</summary>

Let's take an example of a simple app that uses OPENAI_API_KEY stored as secrets.

This app directory contains the following files:

.
ā”œā”€ā”€ main.py             # The app
ā”œā”€ā”€ jcloud.yml          # JCloud deployment config file
ā”œā”€ā”€ README.md           # This README file
ā”œā”€ā”€ requirements.txt    # The requirements file for the app
ā””ā”€ā”€ secrets.env         # The secrets file containing the redis credentials

Note secret.env in this directory is a dummy file. You should replace it with your own secrets after creating a Redis instance. (For example with Upstash), such as:

OPENAI_API_KEY=sk-xxx

main.py will look like:

# main.py
from lcserve import serving
from langchain.prompts import PromptTemplate
from langchain.chains import LLMChain
from langchain.chat_models import ChatOpenAI

prompt = PromptTemplate(
    input_variables=["subject"],
    template="Write me a short poem about {subject}?",
)


@serving(openai_tracing=True)
def poem(subject: str, **kwargs):
    tracing_handler = kwargs.get("tracing_handler")

    chat = ChatOpenAI(temperature=0.5, callbacks=[tracing_handler])
    chain = LLMChain(llm=chat, prompt=prompt, callbacks=[tracing_handler])

    return chain.run(subject)

In the above example, the app will use OPENAI_API_KEY provided by the secrets to interact with OpenAI.

Then you can deploy using the following command and interact with the deployed endpoint.

lc-serve deploy jcloud main --secrets secrets.env
</details>

ā±ļø Trigger one-time jobs to run asynchronously

Here's a step-by-step guide to trigger one-time jobs to run asynchronously using @job decorator.

šŸ’» lc-serve CLI

lc-serve is a simple CLI that helps you to deploy your agents on Jina AI Cloud (JCloud)

DescriptionCommand
Deploy your app locallylc-serve deploy local app
Export your app as Kubernetes YAMLlc-serve export app --kind kubernetes --path .
Export your app as Docker Compose YAMLlc-serve export app --kind docker-compose --path .
Deploy your app on JCloudlc-serve deploy jcloud app
Deploy FastAPI app on JCloudlc-serve deploy jcloud --app <app-name>:<app-object>
Update existing app on JCloudlc-serve deploy jcloud app --app-id <app-id>
Get app status on JCloudlc-serve status <app-id>
List all apps on JCloudlc-serve list
Remove app on JCloudlc-serve remove <app-id>
Pause app on JCloudlc-serve pause <app-id>
Resume app on JCloudlc-serve resume <app-id>

šŸ’” JCloud Deployment

āš™ļø Configurations

For JCloud deployment, you can configure your application infrastructure by providing a YAML configuration file using the --config option. The supported configurations are:

For example:

instance: C4
autoscale_min: 0
disk_size: 1.5G

You can alternatively include a jcloud.yaml file in your application directory with the desired configurations. However, please note that if the --config option is explicitly used in the command line interface, the local jcloud.yaml file will be disregarded. The command line provided configuration file will take precedence.

If you don't provide a configuration file or a specific configuration isn't specified, the following default settings will be applied:

instance: C3
autoscale_min: 1
disk_size: 1G

šŸ’° Pricing

Applications hosted on JCloud are priced in two categories:

Base credits

Serving credits

Total credits charged = Base credits + Serving credits. (Jina AI Cloud defines each credit as ā‚¬0.005)

Examples

<details> <summary><b>Example 1</b></summary>

Consider an HTTP application that has served requests for 10 minutes in the last hour and uses a custom config:

instance: C4
autoscale_min: 0
disk_size: 2G

Total credits per hour charged would be 3.33. The calculation is as follows:

C4 instance has an hourly credit rate of 20.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 0 + 2 * 0.104 = 0.208 (since `autoscale_min` is 0)
Serving credits = 20 * 10/60 = 3.33
Total credits per hour = 0.208 + 3.33 = 3.538
</details> <details> <summary><b>Example 2</b></summary>

Consider a WebSocket application that had active connections for 20 minutes in the last hour and uses the default configuration.

instance: C3
autoscale_min: 1
disk_size: 1G

Total credits per hour charged would be 13.33. The calculation is as follows:

C3 instance has an hourly credit rate of 10.
EFS has hourly credit rate of 0.104 per GB.
Base credits = 10 + 1 * 0.104 = 10.104 (since `autoscale_min` is 1)
Serving credits = 10 * 20/60 = 3.33
Total credits per hour = 10.104 + 3.33 = 13.434
</details>

ā“ Frequently Asked Questions

lc-serve command not found

<details> <summary><b>Expand</b></summary>

lc-serve command is registered during langchain-serve installation. If you get command not found: lc-serve error, please replace lc-serve command with python -m lcserve & retry.

</details>

My client that connects to the JCloud hosted App gets timed-out, what should I do?

<details> <summary><b>Expand</b></summary>

If you make long HTTP/ WebSocket requests, the default timeout value (2 minutes) might not be suitable for your use case. You can provide a custom timeout value during JCloud deployment by using the --timeout argument.

Additionally, for HTTP, you may also experience timeouts due to limitations in the OSS we used in langchain-serve. While we are working to permanently address this issue, we recommend using HTTP/1.1 in your client as a temporary workaround.

For WebSocket, please note that the connection will be closed if idle for more than 5 minutes.

</details>

How to pass environment variables to the app?

<details> <summary><b>Expand</b></summary>

We provide 2 options to pass environment variables:

  1. Use --env during app deployment to load env variables from a .env file. For example, lc-serve deploy jcloud app --env some.env will load all env variables from some.env file and pass them to the app. These env variables will be available in the app as os.environ['ENV_VAR_NAME'].

  2. You can also pass env variables while sending requests to the app both in HTTP and WebSocket. envs field in the request body is used to pass env variables. For example

    {
        "question": "What is the meaning of life?",
        "envs": {
            "ENV_VAR_NAME": "ENV_VAR_VALUE"
        }
    }
    
</details>

JCloud deployment failed at pushing image to Jina Hubble, what should I do?

<details> <summary><b>Expand</b></summary>

Please use --verbose and retry to get more information. If you are operating on computer with arm64 arch, please retry with --platform linux/amd64 so the image can be built correctly.

</details>

Debug babyagi playground request/response for external integration

<details> <summary><b>Expand</b></summary> 1. Start textual console in a terminal (exclude following groups to reduce the noise in logging)
```bash
textual console -x EVENT -x SYSTEM -x DEBUG
```

2. Start the playground with --verbose flag. Start interacting and see the logs in the console.

```bash
lc-serve playground babyagi --verbose
```
</details>

šŸ“£ Reach out to us

Want to deploy your LLM apps on your own infrastructure with all capabilities of Jina AI Cloud?

Join us on Discord and we'd be happy to hear more about your use case.