Home

Awesome

Kaltura Media Framework

Build Status

A distributed framework for live video streaming. The system is composed of multiple components, each one responsible for a specific function.

The components can be deployed on a single server for small scale deployments/testing, but it is recommended to deploy them separately for a more optimal resource utilization. For example, the transcoder can utilize the GPU, so it would be more cost efficient to deploy the transcoders on GPU-enabled servers, while the other components would run on servers without GPU.

Media is transmitted between the different components internally using custom protocols -

  1. Kaltura Media Protocol (KMP) - a TCP-based protocol for delivering streaming media, conceptually, similar to a single track of fMP4/MPEG-TS
  2. Kaltura Segmented Media Protocol (KSMP) - an HTTP-based protocol for delivering media in segments, conceptually, a super-set of LLHLS/DASH

The orchestration of the different media components is performed by a "controller". The main responsibility of the controller is building the topology of the media pipeline, and updating it in case of failures. The controller gets JSON events from the media components sent as HTTP-POSTs. In addition, all the media processing components expose a JSON-based REST API, that is used by the controller to get the latest status and take actions. A sample controller implementation for an all-in-one server is provided in the conf folder.

Main Features

Getting Started

The conf folder contains sample code and configuration for running an all-in-one server.

Glossary

Sample Topologies

The diagrams below demonstrate a few sample topologies that can be created using the Media-Framework components.

Simple RTMP Passthrough

flowchart LR;
    enc(Encoder);
    ingest(nginx-rtmp-kmp-module);
    live(nginx-live-module);
    pckg(nginx-pckg-module);
    play(Player);
    enc-->|RTMP|ingest;
    ingest-->|KMP|live;
    live-->|KSMP|pckg;
    pckg-->|LLHLS/DASH|play;

Passthrough + S3 Persistence

flowchart LR;
    enc(Encoder);
    ingest(nginx-rtmp-kmp-module);
    live(nginx-live-module);
    pckg(nginx-pckg-module);
    s3(Amazon S3);
    play(Player);
    enc-->|RTMP|ingest;
    ingest-->|KMP|live;
    live-->|HTTP|s3;
    s3-->|HTTP|live;
    live-->|KSMP|pckg;
    pckg-->|LLHLS/DASH|play;

SRT Input + Video Transcoding

flowchart LR;
    enc(Encoder);
    ingest(nginx-mpegts-kmp-module);
    srt(nginx-srt-module);
    trans(transcoder);
    live(nginx-live-module);
    pckg(nginx-pckg-module);
    play(Player);
    enc-->|SRT|srt;
    srt-->|MPEG-TS|ingest;
    ingest-->|KMP video|trans;
    trans-->|KMP video|live;
    ingest-->|KMP audio|live;
    live-->|KSMP|pckg;
    pckg-->|LLHLS/DASH|play;

Closed-captions Decoding

flowchart LR;
    enc(Encoder);
    ingest(nginx-rtmp-kmp-module);
    cc(nginx-cc-module);
    live(nginx-live-module);
    pckg(nginx-pckg-module);
    play(Player);
    enc-->|RTMP|ingest;
    ingest-->|KMP video|cc;
    cc-->|KMP subtitle|live;
    ingest-->|KMP video|live;
    ingest-->|KMP audio|live;
    live-->|KSMP|pckg;
    pckg-->|LLHLS/DASH|play;

Transcoding + RTMP Push

flowchart LR;
    enc(Encoder);
    ingest(nginx-mpegts-kmp-module);
    trans(transcoder);
    live(nginx-live-module);
    pckg(nginx-pckg-module);
    push(nginx-kmp-rtmp-module);
    yt(YouTube);
    play(Player);
    enc-->|MPEG-TS/HTTP|ingest;
    ingest-->|KMP|trans;
    trans-->|KMP|live;
    trans-->|KMP|push;
    push-->|RTMP|yt;
    live-->|KSMP|pckg;
    pckg-->|LLHLS/DASH|play;

Components Overview

Media Components

Important: All stateful nginx-based components (=all except nginx-pckg-module), must be deployed on a single process nginx server (worker_processes 1;). The module state is kept per process, and when multiple processes are used, it is not possible to control which process will get the request. For example, the request to create a channel on the segmenter may arrive to worker 1, while the KMP connection with the actual media, will hit worker 2. In deployments that use containers, this shouldn't be a problem - multiple containers can be deployed on a single server, instead of using multiple nginx processes. Another possibility is to use a patch like arut's per-worker listener, but it will probably need to be updated to apply to stream connections as well.

Dependencies

The following modules are dependencies for building the media components listed above. When compiling nginx, the dependencies must be added (--add-module) before any module that requires them.

Debug Options

Some of the Media-Framework components support optional preprocessor macros for debugging purposes -

To test the modules with valgrind, it is recommended to the apply the no-pool-nginx patch, and configure nginx with --with-cc-opt="-O0 -DNGX_BLOCK_POOL_SKIP -DNGX_LBA_SKIP" and --with-debug.

Kaltura Media Protocol (KMP)

Kaltura Media Protocol is a simple packet-based protocol for streaming media over TCP. A KMP connection can deliver the media of a single video/audio/subtitle track - when multiple tracks are needed, multiple TCP connections are established.

Each packet starts with a header that contains the following fields (32 bits each) -

The structures and constants used in KMP can be found in ngx_live_kmp.h.

KMP Frame Ids

A frame id is a 64-bit integer that uniquely identifies an input frame. The ingest modules (nginx-rtmp-kmp-module / nginx-mpegts-kmp-module) allocate the initial frame id according to the server clock (in timescale units), when an output track is created. In order to avoid the need to send the frame id on each frame that is being sent, the frame ids in KMP are sequential - the id of the N-th frame that is sent on a KMP connection is initial_frame_id + N.

If the input connection (e.g. RTMP) drops and gets re-established, new KMP frame ids will be allocated. Since the default timescale is high (90kHz), and the frame rate is unlikely to exceed 60fps, even in case of a reconnect after a short period of streaming, the initial frame id will be significantly higher than the frame id that was last sent on the previous connection. So, it is extremely unlikely to have a conflict with any previously used frame ids due to reconnect.

Frame ids are used:

The transcoder adds a few complexities to the management of frame ids -

Publisher KMP Packets

The sections below list the KMP packets that can be sent by a KMP publisher.

Connect (cnct)

Sent immediately after the KMP TCP connection is established.

The header contains the following fields:

The data of the connect packet is optional, the expected format of the data is defined by the specific KMP receiver.

Media Info (minf)

Contains the parameters of the media. Some of the fields in the header are shared by all media types, while the rest are defined only for a specific type (union).

The shared header fields are:

The video-specific header fields are:

The audio-specific header fields are:

The data of the media info packet holds the codec private/extra data. For example, when using the h264 codec, the data contains the body of an avcC MP4 box.

KMP receivers should handle media info changes, for example, a change to the video resolution. However, the type of the media (video/audio/subtitle) that is sent in a KMP connection, must not change.

KMP receivers should ignore media info packets, when they are identical to the previously received media info packet.

Frame (fram)

Represents a single video frame / audio frame / subtitle cue.

The frame header contains the following fields:

When the media type is video / audio, the data of the frame packet holds the compressed media. When the media type is subtitle and the codec is WebVTT, the data of the frame follows the WebVTT Sample Format, as specified in ISO/IEC 14496-30 (usually, in this case, a sample is a vttc box, that contains a payl box).

Null (null)

Sent in order to signal "liveness", and prevent idle timers from expiring. Null packets do not carry any data other than the basic KMP header. Parsers must ignore null packets.

End Of Stream (eost)

Used to signal a graceful termination of the publishing session. End of stream packets do not carry any data other than the basic KMP header.

Receiver KMP Packets

The sections below list the KMP packets that can be sent by a KMP receiver.

Ack Frames (ackf)

Acknowledge the receipt of frames.

The KMP receiver decides on the appropriate time to send an ack packet. For example, when persistence is enabled, the segmenter sends an ack only after a segment that contains the frame is saved to storage.

Some receivers do not send acks at all, in this case, the KMP producer must be configured to discard the frames after they are sent (using the resume_from setting)

The packet header contains the following fields:

The data of the ack packets is not used.

Kaltura Segmented Media Protocol (KSMP)

Kaltura Segmented Media Protocol is an HTTP-based protocol for delivering media in segments, similarly to HLS/DASH.

A KSMP request is an HTTP GET request, the following query parameters are defined -

A KSMP response uses KLPF format (see below), with type Serve (serv). The KSMP-specific definitions can be found in ngx_ksmp.h

Kaltura Live Persist File (KLPF)

Kaltura Live Persist File is a serialization scheme that is used in KSMP responses and in the S3 objects created by nginx-live-module.

A KLPF is composed of blocks, similar to MP4 atoms/boxes. Each block has the following header -

A KLPF file is a block whose id is set to klpf. Following the generic block header fields (listed above), a KLPF file has the following fields in its header -

For more details on the internal structure of KLPF blocks, see KLFP-SPEC.md.

To inspect the contents of KLPF objects/KSMP responses, use klpf_parse.py. The script can show the block structure without any additional info, however, in order to parse the fields inside the blocks:

API Overview

All the media processing components expose a JSON-based REST API. This section explains the general properties of the Media-Framework APIs. For a detailed reference of the available API endpoints, see the documentation of the specific modules.

Request Types

The following HTTP verbs are used in the API:

The request body in POST / PUT requests must be a JSON (usually an object), and the request must use the header Content-Type: application/json.

When the size of the request body exceeds a certain threshold, nginx writes it to a temporary file. However, the implementation of the Media-Framework API requires that the request body of POST / PUT requests will be available in memory. If needed, the nginx client_body_buffer_size directive can be used to increase the size of the buffer allocated for the request body.

Status Codes

HTTP status codes are used to return the execution status of API requests.

The following success codes are used:

The following error codes are used:

Multi Request

Setting up a channel in nginx-live-module may require multiple API calls - create the channel, create a timeline, create a variant, etc. In order to avoid the penalty of multiple round trips, the API layer has support for "multi" requests. A multi request bundles together several API requests in a single HTTP request.

Multi requests must use the POST verb, and their URI must be set to /multi. The request body must be a JSON array of objects, each object represents a single API request.

The objects contains the following fields:

The response of multi requests is also a JSON array of objects. The number of elements in the response array always matches the number of elements in the request array, and the order of the objects in the response array matches the order in the request array. In other words, the N-th item of the response array, is the response of the N-th request in the request array.

Each response object contains the following fields:

Copyright & License

All code in this project is released under the AGPLv3 license unless a different license for a particular library is specified in the applicable library path.

Copyright © Kaltura Inc. All rights reserved.