c3tracker:setup

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
c3tracker:setup [2024/08/28 18:37] – kunsic3tracker:setup [2025/10/01 08:49] (current) – kunsi
Line 6: Line 6:
   * Adjust project properties, recommendations: (full reference at the bottom of the page)   * Adjust project properties, recommendations: (full reference at the bottom of the page)
   * Compare project properties with previous instalment of the same event   * Compare project properties with previous instalment of the same event
 +
  
 ==== optimal properties from current project ==== ==== optimal properties from current project ====
  
-``` +Replace `camp2023` with your chosen tracker project slug.
-Meta.Acronym                    camp2023 +
-Meta.Album                      Chaos Communication Camp 2023 +
-Meta.License                    Licensed to the public under http://creativecommons.org/licenses/by/4.0 +
-Meta.Year                       2023+
  
-Processing.Auphonic.Enable      no +  Meta.Acronym                    camp2023 
-Processing.BasePath             /video/ +  Meta.Album                      Chaos Communication Camp 2023 
-Processing.MasterMe.Enable      yes +  Meta.License                    Licensed to the public under http://creativecommons.org/licenses/by/4.0 
-Processing.Path.Intros          /video/intros/camp2023 +  Meta.Year                       2023 
-Processing.Path.Outro           /video/intros/camp2023/outro.ts+   
 +  Processing.Auphonic.Enable      no 
 +  Processing.BasePath             /video/ 
 +  Processing.MasterMe.Enable      yes 
 +  Processing.Path.Intros          /video/intros/camp2023 
 +  Processing.Path.Outro           /video/intros/camp2023/outro.ts 
 +   
 +  Publishing.Upload.SkipSlaves    minion254-1,minion254-2,minion254-3,minion254-4 
 +  Publishing.UploadTarget         releasing.c3voc.de:/video/encoded/camp2023/ 
 +  Publishing.Tags                 <additional tags> 
 +  Publishing.Voctoweb.Enable      yes 
 +  Publishing.Voctoweb.Path        /cdn.media.ccc.de/events/camp2023 
 +  Publishing.Voctoweb.Slug        camp2023 
 +  Publishing.Voctoweb.Thumbpath   /static.media.ccc.de/conferences/camp2023 
 +  Publishing.YouTube.Category     27 
 +  Publishing.YouTube.Enable       yes 
 +  Publishing.YouTube.Playlists    <meep> 
 +  Publishing.YouTube.Privacy      <one of: public, unlisted, private> 
 +  Publishing.YouTube.Token        <meep> 
 +   
 +  Record.Container                TS 
 +  Record.EndPadding               300 
 +  Record.Slides                   yes 
 +  Record.StartPadding             300 
  
-Publishing.Path                 /video/encoded/camp2023/ 
-Publishing.Upload.SkipSlaves    speedy,tweety,blade1,blade2,blade3,blade4 
-Publishing.UploadTarget         upload@releasing.c3voc.de:/video/encoded/camp2023/ 
-Publishing.Tags                 <additional tags> 
-Publishing.Voctoweb.Enable      yes 
-Publishing.Voctoweb.Path        /cdn.media.ccc.de/events/camp2023 
-Publishing.Voctoweb.Slug        camp2023 
-Publishing.Voctoweb.Thumbpath   /static.media.ccc.de/conferences/camp2023 
-Publishing.YouTube.Category     27 
-Publishing.YouTube.Enable       yes 
-Publishing.YouTube.Playlists    <meep> 
-Publishing.YouTube.Privacy      <one of: public, unlisted, private> 
-Publishing.YouTube.Token        <meep> 
- 
-Record.Container                TS 
-Record.EndPadding               300 
-Record.Slides                   yes 
-Record.StartPadding             300  
-``` 
  
 === Worker Filter Examples === Worker Filter Examples
  
-``` +   
-EncodingProfile.IsMaster=no +  # get all non-master tickets 
-EncodingProfile.IsMaster=yes +  EncodingProfile.IsMaster=no 
-EncodingProfile.IsMaster= +   
-Fahrplan.Room=Servus.at Lab +  # get all master tickets 
-```+  EncodingProfile.IsMaster=yes 
 +   
 +  # get all recording tickets 
 +  EncodingProfile.IsMaster= 
 +   
 +  # get all tickets in this room. 
 +  Fahrplan.Room=Servus.at Lab 
 + 
 + 
 +Please note that the conditions in the "project to worker group" filter are currently always evaluated with logical OR. 
 + 
 +Specifying a property with an empty value, which is often done for `EncodingProfile.IsMaster`, will match if this property does not exist at all on a ticket. So for `EncodingProfile.IsMaster`, specifying an empty filter will match on recording tickets which never have this property.
  
-Please note that the conditions in the "project to worker group" filter are currently evaluated with logical OR.  
  
-Specifying a property with an empty value, which is often done for EncodingProfile.IsMaster, will match if this property does not exist at all on a ticket. So for EncodingProfile.IsMaster, specifying an empty filter will match on recording tickets which never have this property.+{{.:pasted:20250919-215601.png}}
 == Pipeline setup during event == Pipeline setup during event
  
Line 58: Line 69:
  
  
-=== Simple: decentralised classic (Variant 2)+=== Simple: single-room setup (Variant A)
  
 {{drawio>c3tracker:setup-simple.png}} {{drawio>c3tracker:setup-simple.png}}
  
  
-This variant is only practical if you have only one room, or at least one release encoder (aka [[hardware:Minion]]) for each recording cube. +This variant is practical if you have only one room, or at least one release encoder (aka [[hardware:Minion]]) for each recording encoder PC. 
 When using this variant with multiple rooms in one Tracker project (like at JEV22), you also have to set room filters in the tracker worker queues. When using this variant with multiple rooms in one Tracker project (like at JEV22), you also have to set room filters in the tracker worker queues.
  
 For every worker: For every worker:
-* set room filters in tracker e.g. `Fahrplan.Room = Foobar`+* set `EncodingProfile.IsMaster = yes` to avoid encoding all sub formats 
 +* (set room filters in tracker e.g. `Fahrplan.Room = Foobar`, but this cannot be used at the same times as the above, see the warning below)
  
 For every recoding cube: For every recoding cube:
Line 77: Line 89:
  
  
-=== centralised storage (rsync) (Variant 1)+<panel type="danger" title="Attention">Since tracker filters are joined via OR and not AND, this setup cannot be extended to multiple rooms without hacks if you want to e.g. limit on-site encoding to master formats. Use the  `CRS_ROOM` filter in bundlewrap if you need both tracker filters and room-specific encoding workers.</panel> 
 + 
 +=== centralised storage (rsync) (Variant B)
  
 {{drawio>c3tracker:setup-central-storage.png}} {{drawio>c3tracker:setup-central-storage.png}}
  
  
-The first variant is typically used for events with more than one room. For bigger events we use the dedicated [[hardware:event-storage|storage]] server in the event server rack, for smaller events a USB connected hard drive to one of the minions might be sufficient. Each recording cube exposes the files via rsyncd, which are pulled by an rsync process running inside a screen on the storage pc.+This variant is typically used for events with more than one room.  
 + 
 +For bigger events we use the dedicated [[hardware:event-storage|storage]] server in the event server rack, for smaller events one of the minions is typically sufficient. If you use a minion as storage machine, you might want to configure `10.73.0.2` (aka `storage.lan.c3voc.de`) as secondary IP address to make things easier: 
 + 
 +  sudo ip a add dev br0 10.73.200.24/16 
 + 
 + 
 +Each recording encoder exposes the files via rsyncd, which are pulled by an rsync process running on this storage machine. 
 + 
 + 
 +For each encoderX start rsync on the storage machine:  
 + 
 +  sudo systemctl start rsync-from-encoder@encoderX.lan.c3voc.de 
 + 
 +If you use a minion as storage machine, you have start the tracker manually (on storage.lan.c3voc.de they are started by default):  
 + 
 +  sudo systemctl start crs-worker.target 
 +   
 +which starts 
 + 
 +- crs-recording-scheduler (A) 
 +- crs-mount4cut.service   (B) 
 +- crs-cut-postprocessor   (C) 
 +- crs-encoding0, crs-encoding1 (D) 
 +- crs-postencoding        (E) 
 +- crs-postprocessing      (F)
  
-For each encoderX start rsync on the central storage: `sudo systemctl start rsync-from-encoder@encoderX.lan.c3voc.de` 
  
-Then, start tracker workers on storage: `sudo systemctl start crs-worker.target` (only needed if you don't use `storage.lan.c3voc.de` - worker scripts get started automatically) 
  
 ==== Minion setup ==== Minion setup
  
-To allow the encoding workers to do their job, they need to mount the storage first: `sudo crs-mount <storage location>`+To allow get additional release encoding workers on the (other) minions, you need to connect to these an mount the storage machine first: 
  
-After mounting, you can start the tracker encoding workers: `sudo systemctl start crs-encoding.service`+  ssh minionX.lan.c3voc.de 
 +  sudo crs-mount <storage location>
  
-The minion VMs running inside our event colo case automatically mount `storage.lan.c3voc.de` via cifs and start their worker scripts. You usually do not need to touch them.+After mounting, you can start the tracker encoding workers:  
 + 
 +  sudo systemctl start crs-encoding0.service 
 + 
 +The virtual minion's living in [[hardware:server-case|event colo case]] automatically mount `storage.lan.c3voc.de` via CIFS and start their worker scripts. You usually do not need to touch them.
      
-==== Cube as worker setup 
  
-At small events, when the talks are finished for today, you can use the recording cubes to encode master MP4 files. +==== Encoder PCs as release encoding workers 
 + 
 +At small events, when all talks are finished for today, you can use the recording encoder PCs to encode master MP4 files.  
 + 
 +First – stop voctocore: 
 + 
 +  sudo systemctl stop voctomix2-voctocore
  
-First: Stop voctocore. 
  
 The rest is very similar to above, but with different mounts so `/video/capture` is not hidden: The rest is very similar to above, but with different mounts so `/video/capture` is not hidden:
  
-```bash + 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/fuse /video/fuse +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/fuse /video/fuse 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/video/intros /video/intros +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/video/intros /video/intros 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/tmp /video/tmp +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/tmp /video/tmp 
-sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/encoded /video/encoded +  sudo mount -t cifs -o uid=voc,password= //storage.lan.c3voc.de/encoded /video/encoded 
-```+
      
  
-=== decentralised pipeline (Variant 3)+=== decentralised pipeline aka "even more samba" (Variant C)
  
-<panel type="danger" title="Attention">The "decentralized pipeline (Variant 3)" should not be used by inexperienced users. Use the information above to find out how to get this variant working, then adjust/improve the documentation here.</panel>+<panel type="danger" title="Attention">The "decentralized pipeline" should not be used by inexperienced users. Use the information above to find out how to get this variant working, then adjust/improve the documentation here.</panel>
  
-Similar to variant 2, but the release encoder (minion) only mounts the /video/fuse/$room/ from each recording cube. The encoded and tmp files life on one minion, the secondary minions mount /video/encoded and /video/tmp from the primary minion. [Reason: It is not guaranteed that the minion which encoded a talk also does the postprocessing (upload) step.]+Similar to variant B, but extended to work with multiple rooms. Instead of using rsync, recorded snippets remain on the encoding cubes and ''/video/fuse/$event/$room'' are exposed via samba to the minions, while the encoded and tmp files live on one "central" minion; all other minions mount ''/video/encoded'' and ''/video/tmp'' from the primary minion [reasoning: the tracker cannot guarantee that the machine which encoded a talk also does the postprocessing (upload) step, so all minions have to see the same files].
  
-You have to set the room filters only for the recording cubes, the minions can process talks independently.+Tracker filters have to be set only for the recording cubes, minions do not require any filters (but on smaller events without many minions, a ''EncodingProfile.IsMaster=yes'' filter can be a good idea, so sub formats won't crowd out the queues — they can always be encoded off-site later).
  
-  * On recording cubes:   start systemd units for steps A ''crs-recording-scheduler'', B ''crs-mount4cut'', and C ''crs-cut-postprocessor''         +On recording encoder PCs start the following systemd units: 
-  * On release encoders:  start systemd units for steps D ''crs-encoding'', E ''crs-postencoding'', and F ''crs-postprocessing''+ * ''crs-recording-scheduler'' 
 + * ''crs-mount4cut'' 
 + * ''crs-cut-postprocessor''
  
 +On all minions, including the one acting as storage, do
  
 +  mkdir -p /video/fuse/$event/{$room1, $room2, ..}
 +  mount.cifs -o uid=voc,password= {//$encoder1.lan.c3voc.de,}/video/fuse/$event/$room1
 +  mount.cifs -o uid=voc,password= {//$encoder2.lan.c3voc.de,}/video/fuse/$event/$room2
 +  ...
  
-==== New example with systemd units and case 1 and 5:+On all minions except the one acting as storage, also mount: 
 + 
 +  mount.cifs -o uid=voc,password= //$storage.lan.c3voc.de/encoded /video/encoded 
 +  mount.cifs -o uid=voc,password= //$storage.lan.c3voc.de/tmp /video/tmp 
 +  mount.cifs -o uid=voc,password= {//$storage.lan.c3voc.de,}/video/intros 
 + 
 +Finally on all minions, including the one acting as storage, start the following systemd units: 
 + * ''crs-encoding'' 
 + * ''crs-postencoding'' 
 + * ''crs-postprocessing'' 
 + 
 + 
 +==== Old example with systemd units and case 1 and 5, which was used during jev22 in Munich:
  
 {{drawio>c3tracker:setup-variant-3.png}} {{drawio>c3tracker:setup-variant-3.png}}
  
 optional: configure `10.73.0.2` (aka `storage.lan.c3voc.de`) on the master minion as secondary ip optional: configure `10.73.0.2` (aka `storage.lan.c3voc.de`) on the master minion as secondary ip
 +
 +  sudo ip a add dev br0 10.73.200.24/16
  
 on recording cubes, mount or copy the intros from their source – here `storage.lan.c3voc.de` on recording cubes, mount or copy the intros from their source – here `storage.lan.c3voc.de`
Line 140: Line 207:
   # check if everything is running as expected – you might have to disable/stop the other CRS workers D-F   # check if everything is running as expected – you might have to disable/stop the other CRS workers D-F
   sudo systemctl status -n 0 crs-*    sudo systemctl status -n 0 crs-* 
 +
  
 on master minion (in this example `storage.lan.c3voc.de`) on master minion (in this example `storage.lan.c3voc.de`)
  
-``` 
-mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
-mount -t cifs -o password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
-mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
  
-sudo systemctl start crs-encoding             # D-encoding +  mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
-sudo systemctl start crs-postencoding         # E-postencoding-auphonic +  mount -t cifs -o password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
-sudo systemctl start crs-postprocessing       # F-postprocessing-upload +  mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
- +   
-# check if everything is running as expected – you might have to disable/stop the other CRS workers A-C +  sudo systemctl start crs-encoding             # D-encoding 
-sudo systemctl status -n 0 crs-* +  sudo systemctl start crs-postencoding         # E-postencoding-auphonic 
 +  sudo systemctl start crs-postprocessing       # F-postprocessing-upload 
 +   
 +  # check if everything is running as expected – you might have to disable/stop the other CRS workers A-C 
 +  sudo systemctl status -n 0 crs-* 
  
-``` 
  
 //(ensure that samba is installed on this master minion aka storage)// //(ensure that samba is installed on this master minion aka storage)//
Line 161: Line 228:
  
 on other minions on other minions
-``` 
-mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
-mount -t cifs -o uid=voc,password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
-mount -t cifs -o uid=voc,password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
-mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp 
-mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros 
  
-```+  mkdir -p /video/fuse/jev22/{Ahlam,Bhavani} 
 +  mount -t cifs -o uid=voc,password= {//encoder1.lan.c3voc.de,}/video/fuse/jev22/Ahlam  
 +  mount -t cifs -o uid=voc,password= {//encoder5.lan.c3voc.de,}/video/fuse/jev22/Bhavani 
 +  mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded 
 +  mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded 
 +  mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp 
 +  mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros
        
        
Line 184: Line 249:
  
 on master minion (in this example minion5) on master minion (in this example minion5)
-``` 
-mount -t cifs -o password= //encoder5.lan.c3voc.de/video/fuse/podstock2019/Aussenbuehne /video/fuse/podstock2019/Aussenbuehne 
-mount -t cifs -o password= //encoder6.lan.c3voc.de/video/fuse/podstock2019/Innenbuehne /video/fuse/podstock2019/Innenbuehne 
-mount -t cifs -o password= //encoder6.lan.c3voc.de/video/intros /video/intros 
-cd /opt/crs/tools/tracker3.0/ 
-sudo ./start screenrc-pipeline # with steps D, E, F 
  
-```+  mount -t cifs -o password= //encoder5.lan.c3voc.de/video/fuse/podstock2019/Aussenbuehne /video/fuse/podstock2019/Aussenbuehne 
 +  mount -t cifs -o password= //encoder6.lan.c3voc.de/video/fuse/podstock2019/Innenbuehne /video/fuse/podstock2019/Innenbuehne 
 +  mount -t cifs -o password= //encoder6.lan.c3voc.de/video/intros /video/intros 
 +  cd /opt/crs/tools/tracker3.0/ 
 +  sudo ./start screenrc-pipeline # with steps D, E, F 
 //(ensure that samba is installed on this master minion)// //(ensure that samba is installed on this master minion)//
  
  
 on other minions on other minions
-``` 
-mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/podstock2019/Aussenbuehne  
-mount -t cifs -o password= {//encoder6.lan.c3voc.de,}/video/fuse/podstock2019/Innenbuehne 
-mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded 
-mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp 
-mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros 
-cd /opt/crs/tools/tracker3.0/ 
-sudo ./start screenrc-encoding-only # only step E 
- 
-``` 
  
 +  mount -t cifs -o password= {//encoder5.lan.c3voc.de,}/video/fuse/podstock2019/Aussenbuehne 
 +  mount -t cifs -o password= {//encoder6.lan.c3voc.de,}/video/fuse/podstock2019/Innenbuehne
 +  mount -t cifs //storage.lan.c3voc.de/encoded /video/encoded
 +  mount -t cifs -o password= //storage.lan.c3voc.de/encoded /video/encoded
 +  mount -t cifs -o password= //storage.lan.c3voc.de/tmp /video/tmp
 +  mount -t cifs -o password= {//storage.lan.c3voc.de,}/video/intros
 +  cd /opt/crs/tools/tracker3.0/
 +  sudo ./start screenrc-encoding-only # only step E
  • c3tracker/setup.1724863038.txt.gz
  • Last modified: 2024/08/28 18:37
  • by kunsi