CDN High Availability

Ideas about points of failure and possible fixes in the CDN structure.

Encoder

The room encoders are a spof (single point of failure) for every single stream. The same is often true for the network between the encoder and the master relay.

Master-Relay

The master relay is currently a spof for all streams, which could be eliminated by having 2 redundant masters.

Redundant Masters

The following problems have to be solved to allow redundant encoders:

encoder-source:

enough bandwidth: both masters pull from the same encoder

low bandwidth: encoder pushes to first reachable master

transcode:

Possibility A: single transcode with fallback source and dual outputs: hard

Possibility B: transcode twice: easy, but has negative impacts on fanout

fanout:

easy, each master does it's own fanout locally

however, if transcodes are separate the source streams may not match

downstream-relays:

icecast: hard, has to switch master source

reconfigure on the fly? e.g. by using DNS

nginx: easy, can use multiple upstreams

however if the fanouts are not guaranteed to be equal the upstreams should not be load balanced!

Public Relay

Multiple public relays exist and in case of relay failure only the viewers on that relay will be impacted. However at the moment the streams of these viewers will not properly recover.

Auto-Recovering Webclient

The webplayers should automatically retry on another relay if the current connection fails. Currently the load-balancers determine the viewer→relay assignment. The client will need to fetch another redirect from the loadbalancer (use exponential backoff to avoid crashing the lbs).