c3tracker:setup

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
c3tracker:setup [2023/08/27 17:10] – kunsic3tracker:setup [2024/11/04 13:34] (current) – try to fix formatting jtbx
Line 7: Line 7:
   * Compare project properties with previous instalment of the same event   * Compare project properties with previous instalment of the same event
  
-optimal properties from current project 
  
-``` +==== optimal properties from current project ====
-Meta.Acronym                    camp2023 +
-Meta.Album                      Chaos Communication Camp 2023 +
-Meta.License                    Licensed to the public under http://creativecommons.org/licenses/by/4.0 +
-Meta.Year                       2023+
  
-Processing.Auphonic.Enable      no 
-Processing.BasePath             /video/ 
-Processing.MasterMe.Enable      yes 
-Processing.Path.Intros          /video/intros/camp2023 
-Processing.Path.Outro           /video/intros/camp2023/outro.ts 
  
-Publishing.Path                 /video/encoded/camp2023/ +  Meta.Acronym                    camp2023 
-Publishing.Upload.SkipSlaves    speedy,tweety,blade1,blade2,blade3,blade4 +  Meta.Album                      Chaos Communication Camp 2023 
-Publishing.UploadTarget         upload@releasing.c3voc.de:/video/encoded/camp2023/ +  Meta.License                    Licensed to the public under http://creativecommons.org/licenses/by/4.0 
-Publishing.Voctoweb.Enable      yes +  Meta.Year                       2023 
-Publishing.Voctoweb.Host        berlin-ak.media.ccc.de +   
-Publishing.Voctoweb.Path        /cdn.media.ccc.de/events/camp2023 +  Processing.Auphonic.Enable      no 
-Publishing.Voctoweb.Slug        camp2023 +  Processing.BasePath             /video/ 
-Publishing.Voctoweb.Tags <additional tags> +  Processing.MasterMe.Enable      yes 
-Publishing.Voctoweb.Thumbpath   /static.media.ccc.de/conferences/camp2023 +  Processing.Path.Intros          /video/intros/camp2023 
-Publishing.Voctoweb.Url         https://media.ccc.de/v/ +  Processing.Path.Outro           /video/intros/camp2023/outro.ts 
-Publishing.Voctoweb.User        cdn-app +   
-Publishing.YouTube.Category     27 +  Publishing.Upload.SkipSlaves    speedy,tweety,blade1,blade2,blade3,blade4 
-Publishing.YouTube.Enable       yes +  Publishing.UploadTarget         releasing.c3voc.de:/video/encoded/camp2023/ 
-Publishing.YouTube.Playlists    <meep> +  Publishing.Tags                 <additional tags> 
-Publishing.YouTube.Privacy      private +  Publishing.Voctoweb.Enable      yes 
-Publishing.YouTube.Tags         ccc, camp2023, cccamp23, chaos, communication +  Publishing.Voctoweb.Path        /cdn.media.ccc.de/events/camp2023 
-Publishing.YouTube.Token        <meep>+  Publishing.Voctoweb.Slug        camp2023 
 +  Publishing.Voctoweb.Thumbpath   /static.media.ccc.de/conferences/camp2023 
 +  Publishing.YouTube.Category     27 
 +  Publishing.YouTube.Enable       yes 
 +  Publishing.YouTube.Playlists    <meep> 
 +  Publishing.YouTube.Privacy      <one of: public, unlisted, private> 
 +  Publishing.YouTube.Token        <meep> 
 +   
 +  Record.Container                TS 
 +  Record.EndPadding               300 
 +  Record.Slides                   yes 
 +  Record.StartPadding             300 
  
-Record.Container                TS 
-Record.EndPadding               300 
-Record.Slides                   yes 
-Record.StartPadding             300  
-``` 
  
 === Worker Filter Examples === Worker Filter Examples
  
-``` 
-EncodingProfile.IsMaster=no 
-EncodingProfile.IsMaster=yes 
-EncodingProfile.IsMaster= 
-Fahrplan.Room=Servus.at Lab 
-``` 
  
-Please note that the conditions in the "project to worker group" filter are currently evaluated with logical OR. +  EncodingProfile.IsMaster=no 
 +  EncodingProfile.IsMaster=yes 
 +  EncodingProfile.IsMaster= 
 +  Fahrplan.Room=Servus.at Lab 
 + 
 + 
 +Please note that the conditions in the "project to worker group" filter are currently always evaluated with logical OR. 
  
-Specifying a property with an empty value, which is often done for EncodingProfile.IsMaster, will match if this property does not exist at all on a ticket. So for EncodingProfile.IsMaster, specifying an empty filter will match on recording tickets which never have this property.+Specifying a property with an empty value, which is often done for `EncodingProfile.IsMaster`, will match if this property does not exist at all on a ticket. So for `EncodingProfile.IsMaster`, specifying an empty filter will match on recording tickets which never have this property.
 == Pipeline setup during event == Pipeline setup during event
  
Line 62: Line 58:
  
  
-=== Simple: decentralised classic (Variant 2)+=== Simple: single-room setup (Variant 2)
  
 {{drawio>c3tracker:setup-simple.png}} {{drawio>c3tracker:setup-simple.png}}
Line 71: Line 67:
  
 For every worker: For every worker:
-* set room filters in tracker e.g. `Fahrplan.Room = Foobar`+* set `EncodingProfile.IsMaster = yes` to avoid encoding all sub formats 
 +* (set room filters in tracker e.g. `Fahrplan.Room = Foobar`, but this cannot be used at the same times as the above, see the warning below)
  
 For every recoding cube: For every recoding cube:
Line 80: Line 77:
 * start tracker scripts for encoding: `sudo systemctl start crs-encoding.service` * start tracker scripts for encoding: `sudo systemctl start crs-encoding.service`
  
 +
 +<panel type="danger" title="Attention">Since tracker filters are joined via OR and not AND, this setup cannot be extended to multiple rooms without hacks if you want to e.g. limit on-site encoding to master formats.</panel>
  
 === centralised storage (rsync) (Variant 1) === centralised storage (rsync) (Variant 1)
Line 108: Line 107:
 The rest is very similar to above, but with different mounts so `/video/capture` is not hidden: The rest is very similar to above, but with different mounts so `/video/capture` is not hidden:
  
-```bash + 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/fuse /video/fuse +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/fuse /video/fuse 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/video/intros /video/intros +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/video/intros /video/intros 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/tmp /video/tmp +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/tmp /video/tmp 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/encoded /video/encoded +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/encoded /video/encoded 
-```+
      
  
-=== decentralised pipeline (Variant 3)+=== decentralised pipeline aka "even more samba" (Variant 3)
  
 <panel type="danger" title="Attention">The "decentralized pipeline (Variant 3)" should not be used by inexperienced users. Use the information above to find out how to get this variant working, then adjust/improve the documentation here.</panel> <panel type="danger" title="Attention">The "decentralized pipeline (Variant 3)" should not be used by inexperienced users. Use the information above to find out how to get this variant working, then adjust/improve the documentation here.</panel>
  
-Similar to variant 2, but the release encoder (minion) only mounts the /video/fuse/$room/ from each recording cube. The encoded and tmp files life on one minion, the secondary minions mount /video/encoded and /video/tmp from the primary minion. [Reason: It is not guaranteed that the minion which encoded a talk also does the postprocessing (upload) step.]+Similar to variant 2, but extended to work with multiple rooms. Instead of using rsync, recorded snippets remain on the encoding cubes and ''/video/fuse/$event/$room'' are exposed via samba to the minions, while the encoded and tmp files live on one "central" minion; all other minions mount ''/video/encoded'' and ''/video/tmp'' from the primary minion [reasoning: the tracker cannot guarantee that the machine which encoded a talk also does the postprocessing (upload) step, so all minions have to see the same files].
  
-You have to set the room filters only for the recording cubes, the minions can process talks independently.+Tracker filters have to be set only for the recording cubes, minions do not require any filters (but on smaller events without many minions, a ''EncodingProfile.IsMaster=yes'' filter can be a good idea, so sub formats won't crowd out the queues — they can always be encoded off-site later).
  
-  * On recording cubes:   start systemd units for steps A ''crs-recording-scheduler'', B ''crs-mount4cut'', and C ''crs-cut-postprocessor''         +On recording cubes: start the followin systemd units: 
-  * On release encoders:  start systemd units for steps D ''crs-encoding'', E ''crs-postencoding'', and F ''crs-postprocessing''+ * ''crs-recording-scheduler'' 
 + * ''crs-mount4cut'' 
 + * ''crs-cut-postprocessor''
  
 +On all minions, including the one acting as storage, do
  
 +  mkdir -p /video/fuse/$event/{$room1, $room2, ..}
 +  mount.cifs -o uid=voc,password= {//$encoder1.lan.c3voc.de,}/video/fuse/$event/$room1
 +  mount.cifs -o uid=voc,password= {//$encoder2.lan.c3voc.de,}/video/fuse/$event/$room2
 +  ...
  
-==== New example with systemd units and case 1 and 5:+On all minions except the one acting as storage, also mount: 
 + 
 +  mount.cifs -o uid=voc,password= //$storage.lan.c3voc.de/encoded /video/encoded 
 +  mount.cifs -o uid=voc,password= //$storage.lan.c3voc.de/tmp /video/tmp 
 +  mount.cifs -o uid=voc,password= {//$storage.lan.c3voc.de,}/video/intros 
 + 
 +Finally on all minions, including the one acting as storage, start the following systemd units: 
 + * ''crs-encoding'' 
 + * ''crs-postencoding'' 
 + * ''crs-postprocessing'' 
 + 
 +==== Old example with systemd units and case 1 and 5, which was used during jev22 in Munich:
  
 {{drawio>c3tracker:setup-variant-3.png}} {{drawio>c3tracker:setup-variant-3.png}}
Line 147: Line 164:
 on master minion (in this example `storage.lan.c3voc.de`) on master minion (in this example `storage.lan.c3voc.de`)
  
-``` 
-mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
-mount -t cifs -o password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
-mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
  
-sudo systemctl start crs-encoding             # D-encoding +  mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
-sudo systemctl start crs-postencoding         # E-postencoding-auphonic +  mount -t cifs -o password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
-sudo systemctl start crs-postprocessing       # F-postprocessing-upload+  mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
 +   
 +  sudo systemctl start crs-encoding             # D-encoding 
 +  sudo systemctl start crs-postencoding         # E-postencoding-auphonic 
 +  sudo systemctl start crs-postprocessing       # F-postprocessing-upload 
 +   
 +  # check if everything is running as expected – you might have to disable/stop the other CRS workers A-C 
 +  sudo systemctl status -n 0 crs-* 
  
-# check if everything is running as expected – you might have to disable/stop the other CRS workers A-C 
-sudo systemctl status -n 0 crs-*  
- 
-``` 
  
 //(ensure that samba is installed on this master minion aka storage)// //(ensure that samba is installed on this master minion aka storage)//
Line 165: Line 181:
  
 on other minions on other minions
-``` 
-mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
-mount -t cifs -o uid=voc,password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
-mount -t cifs -o uid=voc,password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
-mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp 
-mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros 
  
-```+  mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
 +  mount -t cifs -o uid=voc,password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
 +  mount -t cifs -o uid=voc,password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
 +  mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded 
 +  mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded 
 +  mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp 
 +  mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros
        
        
Line 188: Line 202:
  
 on master minion (in this example minion5) on master minion (in this example minion5)
-``` 
-mount -t cifs -o password= //encoder5.lan.c3voc.de/video/fuse/podstock2019/Aussenbuehne /video/fuse/podstock2019/Aussenbuehne 
-mount -t cifs -o password= //encoder6.lan.c3voc.de/video/fuse/podstock2019/Innenbuehne /video/fuse/podstock2019/Innenbuehne 
-mount -t cifs -o password= //encoder6.lan.c3voc.de/video/intros /video/intros 
-cd /opt/crs/tools/tracker3.0/ 
-sudo ./start screenrc-pipeline # with steps D, E, F 
  
-```+  mount -t cifs -o password= //encoder5.lan.c3voc.de/video/fuse/podstock2019/Aussenbuehne /video/fuse/podstock2019/Aussenbuehne 
 +  mount -t cifs -o password= //encoder6.lan.c3voc.de/video/fuse/podstock2019/Innenbuehne /video/fuse/podstock2019/Innenbuehne 
 +  mount -t cifs -o password= //encoder6.lan.c3voc.de/video/intros /video/intros 
 +  cd /opt/crs/tools/tracker3.0/ 
 +  sudo ./start screenrc-pipeline # with steps D, E, F 
 //(ensure that samba is installed on this master minion)// //(ensure that samba is installed on this master minion)//
  
  
 on other minions on other minions
-``` 
-mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/podstock2019/Aussenbuehne  
-mount -t cifs -o password= {//encoder6.lan.c3voc.de,}/video/fuse/podstock2019/Innenbuehne 
-mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp 
-mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros 
-cd /opt/crs/tools/tracker3.0/ 
-sudo ./start screenrc-encoding-only # only step E 
- 
-``` 
  
 +  mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/podstock2019/Aussenbuehne 
 +  mount -t cifs -o password= {//encoder6.lan.c3voc.de,}/video/fuse/podstock2019/Innenbuehne
 +  mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded
 +  mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded
 +  mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp
 +  mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros
 +  cd /opt/crs/tools/tracker3.0/
 +  sudo ./start screenrc-encoding-only # only step E
  • c3tracker/setup.1693149055.txt.gz
  • Last modified: 2023/08/27 17:10
  • by kunsi