Home

Awesome

<div align="center"> <h1>Template | Worker</h1>

CI | Test Handler Ā  CD | Build-Test-Release

šŸš€ | A simple worker that can be used as a starting point to build your own custom RunPod Endpoint API worker.

</div>

šŸ“– | Getting Started

  1. Clone this repository.
  2. (Optional) Add DockerHub credentials to GitHub Secrets.
  3. Add your code to the src directory.
  4. Update the handler.py file to load models and process requests.
  5. Add any dependencies to the requirements.txt file.
  6. Add any other build time scripts to thebuilder directory, for example, downloading models.
  7. Update the Dockerfile to include any additional dependencies.
  8. Replace the template worker-config.json file with your own (the template one is an example from our vLLM worker).

šŸ”§ | Worker Config

The worker-config.json is a JSON file that is used to build the form that helps users configure their serverless endpoint on the RunPod Web Interface.

Note: This is a new feature and only works for workers that use one model

<details> <summary>Writing your worker-config.json</summary>

The JSON consists of two main parts, schema and versions.

<details> <summary>Example of schema</summary> 
{
  "schema": {
    "TOKENIZER": {
      "env_var_name": "TOKENIZER",
      "value": "",
      "title": "Tokenizer",
      "description": "Name or path of the Hugging Face tokenizer to use.",
      "required": false,
      "type": "text"
    }, 
    "TOKENIZER_MODE": {
      "env_var_name": "TOKENIZER_MODE",
      "value": "auto",
      "title": "Tokenizer Mode",
      "description": "The tokenizer mode.",
      "required": false,
      "type": "select",
      "options": [
        { "value": "auto", "label": "auto" },
        { "value": "slow", "label": "slow" }
      ]
    },
    ...
  }
}
</details> <details> <summary>Example of versions</summary> 
{
  "versions": {
    "0.5.4": {
      "imageName": "runpod/worker-v1-vllm:v1.2.0stable-cuda12.1.0",
      "minimumCudaVersion": "12.1",
      "categories": [
        {
          "title": "LLM Settings",
          "settings": [
            "TOKENIZER", "TOKENIZER_MODE", "OTHER_SETTINGS_SCHEMA_KEYS_YOU_HAVE_SPECIFIED_0", ...
          ]
        },
        {
          "title": "Tokenizer Settings",
          "settings": [
            "OTHER_SETTINGS_SCHEMA_KEYS_0", "OTHER_SETTINGS_SCHEMA_KEYS_1", ...
          ]
        },
        ...
      ]
    }
  }
}
</details> </details>

āš™ļø | CI/CD (GitHub Actions)

As a reference this repository provides example CI/CD workflows to help you test your worker and build a docker image. The three main workflows are:

  1. CI-test_handler.yml - Tests the handler using the input provided by the --test_input argument when calling the file containing your handler.

Test Handler

This workflow will validate that your handler works as expected. You may need to add some dependency installations to the CI-test_handler.yml file to ensure your handler can be tested.

The action expects the following arguments to be available:

Test End-to-End

This repository is setup to automatically build and push a docker image to the GitHub Container Registry. You will need to add the following to the GitHub Secrets for this repository to enable this functionality:

Additionally, the following need to be added as GitHub actions variables:

The CD-docker_dev.yml file will build the image and push it to the dev tag, while the CD-docker_release.yml file will build the image on releases and tag it with the release version.

The CI-test_worker.yml file will test the worker using the input provided by the --test_input argument when calling the file containing your handler. Be sure to update this workflow to install any dependencies you need to run your tests.

Example Input

{
    "input": {
        "name": "John Doe"
    }
}

šŸ’” | Best Practices

System dependency installation, model caching, and other shell tasks should be added to the builder/setup.sh this will allow you to easily setup your Dockerfile as well as run CI/CD tasks.

Models should be part of your docker image, this can be accomplished by either copying them into the image or downloading them during the build process.

If using the input validation utility from the runpod python package, create a schemas python file where you can define the schemas, then import that file into your handler.py file.

šŸ”— | Links

šŸ³ Docker Container