Video-Pis

Research-documentation for H264-encoding and streaming with Raspberry Pis.

• max. supported resolution: 1080p30. Theoretically CSI-2 with two lanes (card-sized RPis; CM4-module has 4 lanes!) should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;)
• The iframe-peroid of 2 frames is quiet aggressive.
• I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate!

Hint: For better reading pipelines are described as pipeline files which one can use with `cat pipeline-file.txt | xargs gst-launch-1.0`.

Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: blog-post by Mathieu.

Documentation: rtpst2022-1-fecdec, rtpst2022-1-fecenc

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
• Works per multicast as well as per unicast.
• Latency: ca. 1 second
• Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.
• Receiver needs to be started before (first) launch of sender. Help needed! Issue at Mathieu.
• Don't forget to remove “netsim drop-probability=0.01” before flight!

Sender, using audio from HDMI-In and sending video-input to first HDMI-display port:

``` rtpbin

name=rtp 
fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";'

v4l2src

device=/dev/video0 
io-mode=0
do-timestamp=true 

! “video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601” ! tee

name=input

! v4l2h264enc

output-io-mode=0 
extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 

! “video/x-h264,profile=high, level=(string)4” ! h264parse ! mpegtsmux

name=mux 

! rtpmp2tpay

ssrc=0 

! rtp.send_rtp_sink_0

rtp.send_rtp_src_0 ! udpsink

host=224.0.23.1 
port=5000 
multicast-iface=eth0 

rtp.send_fec_src_0_0 ! udpsink

host=224.0.23.1 
port=5002 
async=false 
multicast-iface=eth0

rtp.send_fec_src_0_1 ! udpsink

host=224.0.23.1 
port=5004 
async=false 
multicast-iface=eth0

alsasrc

device=hw:tc358743 
do-timestamp=true 

! “audio/x-raw,format=S16LE,rate=48000,channels=2” ! audioconvert ! voaacenc

bitrate=48000 

! aacparse ! queue ! mux.

input. ! queue ! videoconvert ! fbdevsink

device="/dev/fb0" 
sync=false

```

Receiver, with audio-out on Headphone-jack:

``` rtpbin

latency=100 
fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";' 
name=rtp
  

udpsrc

address=224.0.23.1 
port=5002 
caps="application/x-rtp, payload=96" 

! queue ! rtp.recv_fec_sink_0_0

udpsrc

address=224.0.23.1 
port=5004 
caps="application/x-rtp, payload=96" 

! queue ! rtp.recv_fec_sink_0_1

udpsrc

address=224.0.23.1
port=5000
caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33" 

! queue ! netsim

  drop-probability=0.01 

! rtp.recv_rtp_sink_0

rtp. ! queue ! decodebin

  name=decode

! videoconvert ! queue ! kmssink

decode. ! audioconvert ! queue ! alsasink

  card-name="bcm2835 Headphones"

```

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
• Latency: ca. 0.7 seconds
• SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.
• starting/restarting-order seems strange and needs to be investigated.

Sender, using audio from HDMI-In and sending video-input to display:

``` v4l2src

  device=/dev/video0 
  io-mode=0 
  do-timestamp=true 

! “video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601” ! tee

  name=input

! v4l2h264enc

  output-io-mode=0 
  extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;"

! “video/x-h264,profile=high, level=(string)4” ! mpegtsmux

  name=mux 

! srtsink

  uri=srt://:8888 
  latency=40 
  mode=listener 
  wait-for-connection=true

input. ! queue ! videoconvert ! fbdevsink

  sync=false

alsasrc

  device=hw:tc358743 
  do-timestamp=true 

! “audio/x-raw,format=S16LE,rate=48000,channels=2” ! audioconvert ! voaacenc

  bitrate=48000 

! aacparse ! queue ! mux. ```

Receiver, with audio-out on Headphone-jack:

``` srtclientsrc

  latency=20 
  mode=caller 
  uri=srt://${ipAddress}:8888 

! tsdemux

  latency=40 
  name=demux 

demux. ! queue ! decodebin

  name=decode 

! videoconvert ! queue ! kmssink

demux. ! queue ! avdec_aac ! audioconvert ! alsasink

  card-name="bcm2835 Headphones"

```

RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too?

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
• Latency: reducable to 0.5s
• most errors

Sender, using audio from HDMI-In and sending video-input to display:

``` v4l2src

  device=/dev/video0 
  io-mode=0 
  do-timestamp=true 

! “video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601” ! tee

  name=input 

! v4l2h264enc

  output-io-mode=0 
  extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2,sequence_header_mode=0;" 

! “video/x-h264,profile=high, level=(string)4” ! mpegtsmux

  name=mux 

! rtpmp2tpay ! ristsink

  address=224.0.23.1 
  port=5004 
  multicast-iface=eth0

input. ! queue ! videoconvert ! fbdevsink

  sync=false \
  

alsasrc

  device=hw:tc358743 
  do-timestamp=true 

! “audio/x-raw,format=S16LE,rate=48000,channels=2” ! audioconvert ! voaacenc

  bitrate=48000 

! aacparse ! queue ! mux. ```

Receiver, with audio-out on headphone-jack:

``` ristsrc

  address=224.0.23.1 
  receiver-buffer=200 
  reorder-section=20 

! rtpmp2tdepay ! tsdemux

  latency=50 
  name=demux 

demux. ! queue ! decodebin

  name=decode 

! videoconvert ! queue ! kmssink

demux. ! queue ! avdec_aac ! audioconvert ! alsasink

  card-name="bcm2835 Headphones"

```