cdn:high-availability

channel
channel namefranconian.net
statustested
playout slug (CDN)franconiannet
managed_byfranconian.net
dect1 669
contactsxandy, lujoga
ingest_typertmp-push
recording_typesegmented
releasing_typefuse-tracker
releasing_youtubeja: Kanal mediacccde
schedule_roomfranconian.net
studio
namefranconian.net
locationBamberg
live
preproduction✓(remote only)
remote_speakersremote only
contactsxandy, lujoga
mailrc3orga@lists.hackerspace-bamberg.de
dect1 556
full_remote_operationja
plz96052
geo
engelsystem_usernameslujoga, xAndy

This is an old revision of the document!


CDN High Availability

Ideas about points of failure and possible fixes in the CDN structure.

The room encoders are a spof (single point of failure) for every single stream. The same is often true for the network between the encoder and the master relay.

The master relay is currently a spof for all streams, which could be eliminated by having 2 redundant masters.

The following problems have to be solved to allow redundant encoders:

  • encoder-source:
    • easy, both masters pull from the same encoder
  • transcode:
    • Possibility A: single transcode with fallback source and dual outputs: hard
    • Possibility B: transcode twice: easy, but has negative impacts on fanout
  • fanout:
    • easy, each master does it's own fanout locally
    • however, if transcodes are separate the source streams may not match
  • downstream-relays:
    • icecast: hard, has to switch master source
      • reconfigure on the fly? e.g. by using DNS
    • nginx: easy, can use multiple upstreams
      • however if the fanouts are not guaranteed to be equal the upstreams should not be load balanced!

Multiple public relays exist and in case of relay failure only the viewers on that relay will be impacted. However at the moment the streams of these viewers will not properly recover.

The webplayers should automatically retry on another relay if the current connection fails. Currently the load-balancers determine the viewer→relay assignment. The client will need to fetch another redirect from the loadbalancer (use exponential backoff to avoid crashing the lbs).

  • cdn/high-availability.1584453491.txt.gz
  • Last modified: 2020/03/17 13:58
  • by ischluff