Video-Pis

Research-documentation for H264-encoding and streaming with Raspberry Pis.

• max. supported resolution: 1080p30. Theoretically CSI-2 with two lanes should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;)
• The iframe-peroid of 2 frames is quiet aggressive.
• I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate!

Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: blog-post by Mathieu.

Documentation: rtpst2022-1-fecdec, rtpst2022-1-fecenc

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
• Works per multicast as well as per unicast.
• Latency: ca. 1 second
• Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.
• Receiver needs to be started before (first) launch of sender. Help needed! Issue at Mathieu.
• Don't forget to remove “netsim drop-probability=0.01” before flight!

Sender, using Audio from HDMI-In and sending video-input to display:

Hint: For better reading pipelines are described as pipeline files which one can use with gst-launch-1.0 -f pipeline-file.txt.

rtpbin 
  name=rtp 
  fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";'

v4l2src 
  device=/dev/video0 
  io-mode=0
  do-timestamp=true 
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
! tee 
  name=input
! v4l2h264enc
  output-io-mode=0 
  extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 
! "video/x-h264,profile=high, level=(string)4" 
! h264parse 
! mpegtsmux 
  name=mux 
! rtpmp2tpay 
  ssrc=0 
! rtp.send_rtp_sink_0

rtp.send_rtp_src_0 
! udpsink 
  host=224.0.23.1 
  port=5000 
  multicast-iface=eth0 

rtp.send_fec_src_0_0 
! udpsink 
  host=224.0.23.1 
  port=5002 
  async=false 
  multicast-iface=eth0

rtp.send_fec_src_0_1 
! udpsink 
  host=224.0.23.1 
  port=5004 
  async=false 
  multicast-iface=eth0

alsasrc
  device=hw:tc358743 
  do-timestamp=true 
! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
! audioconvert 
! voaacenc 
  bitrate=48000 
! aacparse 
! queue 
! mux.

input. 
! queue 
! videoconvert 
! fbdevsink 
  device="/dev/fb0" 
  sync=false

Receiver, with audio-out on Headphone-jack:

rtpbin 
  latency=100 
  fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";' 
  name=rtp
    
udpsrc 
  address=224.0.23.1 
  port=5002 
  caps="application/x-rtp, payload=96" 
! queue 
! rtp.recv_fec_sink_0_0

udpsrc
  address=224.0.23.1 
  port=5004 
  caps="application/x-rtp, payload=96" 
! queue 
! rtp.recv_fec_sink_0_1

udpsrc
  address=224.0.23.1
  port=5000
  caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33" 
! queue 
! netsim 
    drop-probability=0.01 
! rtp.recv_rtp_sink_0

rtp. 
! queue 
! decodebin 
    name=decode
! videoconvert 
! queue 
! kmssink

decode. 
! audioconvert 
! queue 
! alsasink 
    card-name="bcm2835 Headphones"

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
• Latency: ca. 0.7 seconds
• SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.
• starting/restarting-order seems strange and needs to be investigated.

Sender, using audio from HDMI-In and sending video-input to display:

v4l2src 
    device=/dev/video0 
    io-mode=0 
    do-timestamp=true 
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" 
! tee 
    name=input
! v4l2h264enc 
    output-io-mode=0 
    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;"
! "video/x-h264,profile=high, level=(string)4"
! mpegtsmux 
    name=mux 
! srtsink 
    uri=srt://:8888 
    latency=40 
    mode=listener 
    wait-for-connection=true

input. 
! queue 
! videoconvert 
! fbdevsink 
    sync=false

alsasrc 
    device=hw:tc358743 
    do-timestamp=true 
! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
! audioconvert 
! voaacenc 
    bitrate=48000 
! aacparse 
! queue 
! mux. 

Receiver, with audio-out on Headphone-jack:

srtclientsrc 
    latency=20 
    mode=caller 
    uri=srt://${ipAddress}:8888 
! tsdemux 
    latency=40 
    name=demux 

demux. 
! queue 
! decodebin 
    name=decode 
! videoconvert 
! queue 
! kmssink

demux. 
! queue 
! avdec_aac 
! audioconvert 
! alsasink 
    card-name="bcm2835 Headphones"

RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too?

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
• Latency: reducable to 0.5s
• most errors

Sender, using audio from HDMI-In and sending video-input to display:

v4l2src 
    device=/dev/video0 
    io-mode=0 
    do-timestamp=true 
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" 
! tee 
    name=input 
! v4l2h264enc 
    output-io-mode=0 
    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2,sequence_header_mode=0;" 
! "video/x-h264,profile=high, level=(string)4" 
! mpegtsmux 
    name=mux 
! rtpmp2tpay 
! ristsink 
    address=224.0.23.1 
    port=5004 
    multicast-iface=eth0

input. 
! queue 
! videoconvert 
! fbdevsink 
    sync=false \
    
alsasrc 
    device=hw:tc358743 
    do-timestamp=true 
! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
! audioconvert 
! voaacenc 
    bitrate=48000 
! aacparse 
! queue 
! mux.

Receiver, with audio-out on headphone-jack:

ristsrc 
    address=224.0.23.1 
    receiver-buffer=200 
    reorder-section=20 
! rtpmp2tdepay 
! tsdemux 
    latency=50 
    name=demux 

demux. 
! queue 
! decodebin 
    name=decode 
! videoconvert 
! queue 
! kmssink

demux. 
! queue 
! avdec_aac 
! audioconvert 
! alsasink 
    card-name="bcm2835 Headphones"