Awesome

Tork is a highly-scalable, general-purpose workflow engine.

Features:

• REST API
• Highly extensible
• Horizontally scalable
• Task isolation - tasks are executed within a container to provide isolation, idempotency, and in order to enforce resource limits
• Automatic recovery of tasks in the event of a worker crash
• Supports both stand-alone and distributed setup
• Retry failed tasks
• Middleware
• Pre/Post tasks
• No single point of failure
• Task timeout
• Full-text search
• Expression Language
• Conditional Tasks
• Parallel Tasks
• For-Each Task
• Subjob Task
• Task Priority
• Sandbox Mode
• Secrets
• Scheduled Jobs
• Web UI

Documentation

See tork.run for the full documentation.

Quick Start

1. Ensure you have Docker with API Version >= 1.42 (use docker version | grep API to check).

2. Download the binary for your system from the releases page.

Hello World

Start in standalone mode:

./tork run standalone

Submit a job in another terminal:

# hello.yaml
---
name: hello job
tasks:
  - name: say hello
    image: ubuntu:mantic #docker image
    run: |
      echo -n hello world
  - name: say goodbye
    image: ubuntu:mantic
    run: |
      echo -n bye world

JOB_ID=$(curl \
  -s \
  -X POST \
  --data-binary @hello.yaml \
  -H "Content-type: text/yaml" \
  http://localhost:8000/jobs | jq -r .id)

Query for the status of the job:

curl -s http://localhost:8000/jobs/$JOB_ID | jq .

{
  "id": "ed0dba93d262492b8cf26e6c1c4f1c98",
  "state": "COMPLETED",
  ...
  "execution": [
    {
      ...
      "state": "COMPLETED",
    }
  ],
}

A slightly more interesting example

The following job:

1. Downloads a remote video file using a pre task to a shared /tmp volume.
2. Converts the first 5 seconds of the downloaded video using ffmpeg.
3. Uploads the converted video to a destination using a post task.

# convert.yaml
---
name: convert a video
inputs:
  source: https://upload.wikimedia.org/wikipedia/commons/1/18/Big_Buck_Bunny_Trailer_1080p.ogv
tasks:
  - name: convert the first 5 seconds of a video
    image: jrottenberg/ffmpeg:3.4-alpine
    run: |
      ffmpeg -i /tmp/input.ogv -t 5 /tmp/output.mp4
    mounts:
      - type: volume
        target: /tmp
    pre:
      - name: download the remote file
        image: alpine:3.18.3
        env:
          SOURCE_URL: "{{ inputs.source }}"
        run: |
          wget \
          $SOURCE_URL \
          -O /tmp/input.ogv
    post:
      - name: upload the converted file
        image: alpine:3.18.3
        run: |
          wget \
          --post-file=/tmp/output.mp4 \
          https://devnull-as-a-service.com/dev/null

Submit the job in another terminal:

JOB_ID=$(curl \
  -s \
  -X POST \
  --data-binary @convert.yaml \
  -H "Content-type: text/yaml" \
  http://localhost:8000/jobs | jq -r .id)

More examples

Check out the examples folder.

REST API

See the REST API documentation.

Swagger Docs

Make sure you have CORS configured in your config file:

[middleware.web.cors]
enabled = true

Start Tork in standalone or coordinator mode.

go run cmd/main.go run standalone

Serve the Swagger Docs

docker compose up -d swagger

Visit http://localhost:9000

Web UI

Tork Web is a web based tool for interacting with Tork.

License

Copyright (c) 2023-present Arik Cohen. Tork is free and open-source software licensed under the MIT License.