CDN

Voctoweb

The Future of Streaming

For redundancy and ease of use the CDN is managed by a distributed system built on consul.

Damit ist es möglich redundante CDN-Master sowie redundante Ingest-Server zu betreiben und trotzdem ein Web-Backend mit allen relevanten Streaming-Infos bereitzustellen. Weiterhin könnte die Streaming-Webseite intelligent auf das Vorhandensein/Fehlen von Streams im CDN reagieren und zusätzliche Metadaten von Encodern verarbeiten.

Ein weiterer Vorteil des angedachten Setups sind deutlich geringere Latenz durch den fehlenden Fanout, sowie deutlich schnellere Stream-Restarts durch aktive Benachrichtigung der Transcoder.

Übersicht

Repo Links

Stream-api, Upload-Proxy, Upload-Server: https://github.com/voc/stream-api

rtmp-auth daemon: https://github.com/voc/rtmp-auth

Transcoding-Script: https://github.com/voc/transcoding

Architecture

The CDN-Cascade has 5 stages: Master-Encoder, Ingest, Transcoder, Master-Relays and finally Edge-Relays.

1. Master-Encoder

The master stream encoding is created near the stage on the encoder PC running voctomix or on with other videoencoder for third party streams.The master encoding contains a 1080p25 H264 video signal, the slides as 1080p5 and up to 3 tracks of audio encoded in AAC (Native, Translated, Translated-2).The translated audio tracks and the slide video track are optional.

The master encoding is pushed as MPEG-TS via SRT to the ingest stage.

2. Ingest

The ingest machines run both srtrelay and nginx-rtmp to receive pushed streams from the master encoders. This also serves as the entry point for third party streams.

Additionally the ingest machines run a stream-api daemon which scrapes the local stream information from both srtrelay and nginx-rtmp and publishes it to consul KV.The consul KV path is /stream/<stream slug>

# consul kv get /stream/s96
{"format":"mpegts","source":"srt://ingest.c3voc.de:1337?streamid=play/s96","slug":"s96","publishedAt":0}

3. Transcoder

The stream transcoding transforms the mpegts stream into multiple segmented streams (one per quality) via one big ffmpeg script. Additionally it creates thumbnails and preview images.The transcoding script can be found here https://github.com/voc/transcoding.

A python script runs constantly on each transcoder which fetches streams from consul and tries to claim streams which don't have a transcoder assigned yet.The consul KV path is /stream/<stream slug>/transcoder

# consul kv get /stream/s96/transcoder
myloc-transcoder3

Every time a transcoder claims a stream it writes an env file and starts a systemd unit to run the transcoding script.The transcode updates are also published to the #voc-wok IRC.

The transcoding outputs are directly pushed with HTTP uploads to the Master relay. To ensure proper retries the uploads are sent over a local http-proxy (https://github.com/voc/stream-api/tree/master/cmd/upload-proxy).

4. Master Relay

Runs the upload server (https://github.com/voc/stream-api/tree/master/cmd/upload-server) which receives the http uploads and stores the files on local disk.The upload server has some additional logic to rewrite the HLS playlists on the fly in order to properly handle stream restarts.

The segmented streams, thumbnails etc. are served by nginx.

5. Edge-Relays

The edge relays run nginx with caching proxy config and upstream set to the master relay.

Stream-URLs

The resulting stream URLs are as follows:

MPEG-DASH (VPx Multi-Quality + Multi-Lang)

http://cdn.c3voc.de/dash/sX/manifest.mpd

HLS (h.264 Multi-Quality + Multi-Lang)

http://cdn.c3voc.de/hls/sX/native_hd.m3u8

Moar HLS

http://cdn.c3voc.de/hls/sX/translated_hd.m3u8 (auch Multi-Lang, nur anderer Default)

http://cdn.c3voc.de/hls/sX/translated-2_hd.m3u8

http://cdn.c3voc.de/hls/sX/native_sd.m3u8 (nur SD, aber trotzdem Multi-Lang)

http://cdn.c3voc.de/hls/sX/translated_sd.m3u8

http://cdn.c3voc.de/hls/sX/translated-2_sd.m3u8

Audio-MP3:

http://cdn.c3voc.de/sX_native.mp3

http://cdn.c3voc.de/sX_translated.mp3

http://cdn.c3voc.de/sX_translated-2.mp3

Audio-Opus:

http://cdn.c3voc.de/sX_native.opus

http://cdn.c3voc.de/sX_translated.opus

http://cdn.c3voc.de/sX_translated-2.opus

Loadbalancer

The end-user facing domains streaming.media.ccc.de as well as cdn.c3voc.de are served by the load balancers using DNS round robin.

streaming.media.ccc.de

Unter dieser Domain liefern beide LBs unabhängig von einander die Streaming-Webseite aus. Die LBs sind dabei komplett unabhängig voneinander und haben auch je eine eigene Konfiguration. Es ist aufgabe des Website-Deployments die Konfiguration synchron zu halten. Durch dieses Setup kann bei Bedarf der jeweilige LB einfach aus der DNS-Rotation herausgenommen werden.

cdn.c3voc.de

Requests to cdn.c3voc.de are handled by haproxy on the load balancers with redirects to one of the edge relays. The redirects are according to a preconfigured weight distribution.

Because of the behavior of HLS/DASH clients if a client fetches a playlist file and gets a redirect, all further requests go to the same origin. With this trick the clients stay persistently on one relay until they are redirected.

Dynamic RTMP endpoints with RTMP Auth

RTMP Endpoints can be created under https://ingest.c3voc.de/backend/ (Password in keepass under ansible/stream-api/htpasswd). After creating the endpoint the stream can be pushed to any ingest machine.

Die RTMP-Url für den endpunkt stream/qtest23 mit auth token lautet z.B.: rtmp://ingest.c3voc.de/stream/qtest23?auth=token