= Video-Pis

Research-documentation for H264-encoding and streaming with Raspberry Pis.

== gstreamer with Auvidea B101 HDMI-CSI2-Bridge

 * max. supported resolution: 1080p30. [[https://forums.raspberrypi.com/viewtopic.php?t=268787|Theoretically]] CSI-2 with two lanes (card-sized RPis; CM4-module has 4 lanes!) should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;)
 * The iframe-peroid of 2 frames is quiet aggressive.
 * I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate!

Hint: For better reading pipelines are described as pipeline files which one can use with `cat pipeline-file.txt | xargs gst-launch-1.0`.

=== SMPTE2022-1 FEC streaming (multicast/unicast), audio and display-loop-out included

Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: [[https://mathieuduponchelle.github.io/2020-10-09-SMPTE-2022-1-2D-Forward-Error-Correction-in-GStreamer.html|blog-post by Mathieu]]. 

Documentation: [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecdec.html|rtpst2022-1-fecdec]], [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecenc.html|rtpst2022-1-fecenc]]


 * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
 * Works per multicast as well as per unicast.
 * Latency: ca. 1 second
 * Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.
 * Receiver needs to be started before (first) launch of sender. Help needed! [[https://github.com/MathieuDuponchelle/MathieuDuponchelle.github.io/issues/11|Issue at Mathieu]].
 * Don't forget to remove "netsim drop-probability=0.01" before flight!

Sender, using audio from HDMI-In and sending video-input to first HDMI-display port:


```
rtpbin 
  name=rtp 
  fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";'

v4l2src 
  device=/dev/video0 
  io-mode=0
  do-timestamp=true 
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
! tee 
  name=input
! v4l2h264enc
  output-io-mode=0 
  extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 
! "video/x-h264,profile=high, level=(string)4" 
! h264parse 
! mpegtsmux 
  name=mux 
! rtpmp2tpay 
  ssrc=0 
! rtp.send_rtp_sink_0

rtp.send_rtp_src_0 
! udpsink 
  host=224.0.23.1 
  port=5000 
  multicast-iface=eth0 

rtp.send_fec_src_0_0 
! udpsink 
  host=224.0.23.1 
  port=5002 
  async=false 
  multicast-iface=eth0

rtp.send_fec_src_0_1 
! udpsink 
  host=224.0.23.1 
  port=5004 
  async=false 
  multicast-iface=eth0

alsasrc
  device=hw:tc358743 
  do-timestamp=true 
! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
! audioconvert 
! voaacenc 
  bitrate=48000 
! aacparse 
! queue 
! mux.

input. 
! queue 
! videoconvert 
! fbdevsink 
  device="/dev/fb0" 
  sync=false
```

Receiver, with audio-out on Headphone-jack:

```
rtpbin 
  latency=100 
  fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";' 
  name=rtp
    
udpsrc 
  address=224.0.23.1 
  port=5002 
  caps="application/x-rtp, payload=96" 
! queue 
! rtp.recv_fec_sink_0_0

udpsrc
  address=224.0.23.1 
  port=5004 
  caps="application/x-rtp, payload=96" 
! queue 
! rtp.recv_fec_sink_0_1

udpsrc
  address=224.0.23.1
  port=5000
  caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33" 
! queue 
! netsim 
    drop-probability=0.01 
! rtp.recv_rtp_sink_0

rtp. 
! queue 
! decodebin 
    name=decode
! videoconvert 
! queue 
! kmssink

decode. 
! audioconvert 
! queue 
! alsasink 
    card-name="bcm2835 Headphones"
```

=== SRT streaming (unicast), audio and display-loop-out included

 * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
 * Latency: ca. 0.7 seconds
 * SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.
 * starting/restarting-order seems strange and needs to be investigated.

Sender, using audio from HDMI-In and sending video-input to display:

```
v4l2src 
    device=/dev/video0 
    io-mode=0 
    do-timestamp=true 
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" 
! tee 
    name=input
! v4l2h264enc 
    output-io-mode=0 
    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;"
! "video/x-h264,profile=high, level=(string)4"
! mpegtsmux 
    name=mux 
! srtsink 
    uri=srt://:8888 
    latency=40 
    mode=listener 
    wait-for-connection=true

input. 
! queue 
! videoconvert 
! fbdevsink 
    sync=false

alsasrc 
    device=hw:tc358743 
    do-timestamp=true 
! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
! audioconvert 
! voaacenc 
    bitrate=48000 
! aacparse 
! queue 
! mux. 
```

Receiver, with audio-out on Headphone-jack:

```
srtclientsrc 
    latency=20 
    mode=caller 
    uri=srt://${ipAddress}:8888 
! tsdemux 
    latency=40 
    name=demux 

demux. 
! queue 
! decodebin 
    name=decode 
! videoconvert 
! queue 
! kmssink

demux. 
! queue 
! avdec_aac 
! audioconvert 
! alsasink 
    card-name="bcm2835 Headphones"
```

=== RIST streaming (multicast/unicast), audio and display-loop-out included

RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too?

 * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
 * Latency: reducable to 0.5s
 * most errors

Sender, using audio from HDMI-In and sending video-input to display:

```
v4l2src 
    device=/dev/video0 
    io-mode=0 
    do-timestamp=true 
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" 
! tee 
    name=input 
! v4l2h264enc 
    output-io-mode=0 
    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2,sequence_header_mode=0;" 
! "video/x-h264,profile=high, level=(string)4" 
! mpegtsmux 
    name=mux 
! rtpmp2tpay 
! ristsink 
    address=224.0.23.1 
    port=5004 
    multicast-iface=eth0

input. 
! queue 
! videoconvert 
! fbdevsink 
    sync=false \
    
alsasrc 
    device=hw:tc358743 
    do-timestamp=true 
! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
! audioconvert 
! voaacenc 
    bitrate=48000 
! aacparse 
! queue 
! mux.
```

Receiver, with audio-out on headphone-jack:

```
ristsrc 
    address=224.0.23.1 
    receiver-buffer=200 
    reorder-section=20 
! rtpmp2tdepay 
! tsdemux 
    latency=50 
    name=demux 

demux. 
! queue 
! decodebin 
    name=decode 
! videoconvert 
! queue 
! kmssink

demux. 
! queue 
! avdec_aac 
! audioconvert 
! alsasink 
    card-name="bcm2835 Headphones"
```