Differences

This shows you the differences between two versions of the page.

--- cdn [2025/01/31 01:21] – ischluff
+++ cdn [2025/05/22 11:22] (current) – ischluff
@@ Line 2: / Line 2: @@
 <bootnote>
-This documentation describes our Livestreaming CDN as of 2024.
+This documentation describes our Livestreaming CDN as of 2025.
 The File-CDN for media.ccc.de is described on [[software:voctoweb]].
 </bootnote>
-== The Future of Streaming
-For redundancy and ease of use the CDN is managed by a distributed system built on consul.
-Damit ist es möglich redundante CDN-Master sowie redundante Ingest-Server zu betreiben und trotzdem ein Web-Backend mit allen relevanten Streaming-Infos bereitzustellen. Weiterhin könnte die Streaming-Webseite intelligent auf das Vorhandensein/Fehlen von Streams im CDN reagieren und zusätzliche Metadaten von Encodern verarbeiten.
-Ein weiterer Vorteil des angedachten Setups sind deutlich geringere Latenz durch den fehlenden Fanout, sowie deutlich schnellere Stream-Restarts durch aktive Benachrichtigung der Transcoder.
-=== Übersicht
-{{drawio>diagram1.png}}
 ==== Repo Links
@@ Line 26: / Line 13: @@
 == Architecture
-The CDN-Cascade has 5 stages: Master-Encoder, Ingest, Transcoder, Master-Relays and finally Edge-Relays.
+The CDN-Cascade has 6 stages: Master-Encoder, Ingest, Transcoder, Master-Relays, Edge-Relays and finally Load-balancers.
 {{:cdn_overview.png?1200|}}
@@ Line 56: / Line 43: @@
 # consul kv get /stream/s96/transcoder
 myloc-transcoder3
+# To reallocate a transcoder for a stuck stream
+consul kv delete /stream/s96/transcoder
 ```
@@ Line 72: / Line 62: @@
 The edge relays run nginx with caching proxy config and upstream set to the master relay.
-== Stream-URLs
+=== 6. Loadbalancer
+The end-user facing domains ''streaming.media.ccc.de'' as well as ''cdn.c3voc.de'' are served by the load-balancers using DNS round robin.
-The resulting stream URLs are as follows:
+==== streaming.media.ccc.de
+The load-balancers serve the [[:software:streamingwebsite|Streaming-Webseite]] under this domain. The sites run completely independent and are just kept in sync by the deployment script. With this procedure one or more load-balancers can easily be taken out of the DNS rotation if necessary.
+==== cdn.c3voc.de
+Requests to ''cdn.c3voc.de'' are handled by haproxy on the load-balancers with redirects to one of the edge relays. The redirects are performed following a preconfigured weight distribution.
+The relay weight distribution is stored in consul KV and can be tweaked in real time. The haproxy configs automatically get updated by consul-template after a change.
+```
+# Get the current weight distribution, if not explicitly set the default is 100
+consul kv get -recurse /services/haproxy/backends/
+services/haproxy/backends/live.event.c3voc.de/weight:1
+services/haproxy/backends/relive.c3voc.de/weight:1
+```
+Because of the behavior of HLS/DASH clients if a client fetches a playlist file and gets a redirect, all further requests go to the same origin. With this trick the clients stay persistently on one relay until they are redirected.
+== Stream-URLs
-MPEG-DASH (VPx Multi-Quality + Multi-Lang)
+The resulting stream URLs are as follows (replace @slug with your actual stream slug):
-* http://cdn.c3voc.de/dash/sX/manifest.mpd
-HLS (h.264 Multi-Quality + Multi-Lang)
+HLS (H.264 multi-quality + multi-language)
-* http://cdn.c3voc.de/hls/sX/native_hd.m3u8
+* http://cdn.c3voc.de/hls/@slug/native_hd.m3u8
 Moar HLS
-* http://cdn.c3voc.de/hls/sX/translated_hd.m3u8 (auch Multi-Lang, nur anderer Default)
+  * http://cdn.c3voc.de/hls/@slug/translated_hd.m3u8 - also multi-language, just a different default language
-* http://cdn.c3voc.de/hls/sX/translated-2_hd.m3u8
+  * http://cdn.c3voc.de/hls/@slug/translated-2_hd.m3u8
-* http://cdn.c3voc.de/hls/sX/native_sd.m3u8 (nur SD, aber trotzdem Multi-Lang)
+  * http://cdn.c3voc.de/hls/@slug/native_sd.m3u8 - only SD, but still multi-language
-* http://cdn.c3voc.de/hls/sX/translated_sd.m3u8
+  * http://cdn.c3voc.de/hls/@slug/translated_sd.m3u8
-* http://cdn.c3voc.de/hls/sX/translated-2_sd.m3u8
+  * http://cdn.c3voc.de/hls/@slug/translated-2_sd.m3u8
+**Currently not active:**
+MPEG-DASH (VPx multi-quality + multi-language)
+* http://cdn.c3voc.de/dash/@slug/manifest.mpd
 Audio-MP3:
-* http://cdn.c3voc.de/sX_native.mp3
+* http://cdn.c3voc.de/@slug_native.mp3
-* http://cdn.c3voc.de/sX_translated.mp3
+* http://cdn.c3voc.de/@slug_translated.mp3
-* http://cdn.c3voc.de/sX_translated-2.mp3
+* http://cdn.c3voc.de/@slug_translated-2.mp3
 Audio-Opus:
-* http://cdn.c3voc.de/sX_native.opus
+* http://cdn.c3voc.de/@slug_native.opus
-* http://cdn.c3voc.de/sX_translated.opus
+* http://cdn.c3voc.de/@slug_translated.opus
-* http://cdn.c3voc.de/sX_translated-2.opus
+* http://cdn.c3voc.de/@slug_translated-2.opus
+== Dynamic SRT/RTMP endpoints with RTMP Auth
+SRT and RTMP Endpoints can be created under https://ingest.c3voc.de/backend/ (Password in keepass under ansible/stream-api/htpasswd). After creating the endpoint the stream can be pushed to any ingest machine.
-== Loadbalancer
+The RTMP-URL for the endpoint ''stream/qtest23'' on ingest1 would be: ''%%rtmp://ingest.c3voc.de/stream/qtest23?auth=<token>%%''
-The end-user facing domains ''streaming.media.ccc.de'' as well as ''cdn.c3voc.de'' are served by the load balancers using DNS round robin.
-=== streaming.media.ccc.de
+The SRT-URL would be : ''%%srt://ingest.c3voc.de:1337?streamid=publish/qtest23/<token>%%''
-Unter dieser Domain liefern beide LBs unabhängig von einander die [[:software:streamingwebsite|Streaming-Webseite]] aus. Die LBs sind dabei komplett unabhängig voneinander und haben auch je eine eigene Konfiguration. Es ist aufgabe des Website-Deployments die Konfiguration synchron zu halten. Durch dieses Setup kann bei Bedarf der jeweilige LB einfach aus der DNS-Rotation herausgenommen werden.
-=== cdn.c3voc.de
+== Future improvements
-Requests to ''cdn.c3voc.de'' are handled by haproxy on the load balancers with redirects to one of the edge relays. The redirects are according to a preconfigured weight distribution.
-Because of the behavior of HLS/DASH clients if a client fetches a playlist file and gets a redirect, all further requests go to the same origin. With this trick the clients stay persistently on one relay until they are redirected.
+The diagram below shows the envisioned architecture as of 2020. In the meantime many of the improvements have been integrated into the CDN.
+Some more potential improvements are as follows:
+ * Dynamically adapting the streaming-website to the existence/lack of a stream
+ * Building a live status dashboard which shows the current stream->ingest->transcoder pipelines
+ * To gracefully shutdown a relay it could be instructed via consul KV to redirect all clients back to a load balancer while taking it out of the haproxy redirects. That way all clients would be redistributed to different relays.
+ * Build master relay redundancy using dynamic upload client and relay upstream configuration. (Relay upstreams need to be sticky, because the HLS playlists generated on the master relay are not deterministic)
+ * Use DNS rotation on ingest
+ * Finalize srtrelay meshing to allow pushed streams to be pulled from any ingest machine.
-=== Dynamic RTMP endpoints with RTMP Auth
+{{drawio>diagram1.png}}
-RTMP Endpoints can be created under https://ingest.c3voc.de/backend/ (Password in keepass under ansible/stream-api/htpasswd). After creating the endpoint the stream can be pushed to any ingest machine.
-Die RTMP-Url für den endpunkt ''stream/qtest23'' mit auth token lautet z.B.: ''%%rtmp://ingest.c3voc.de/stream/qtest23?auth=token%%''