Video-Pis

Research-documentation for H264-encoding and streaming with Raspberry Pis.

• max. supported resolution: 1080p30. Theoretically CSI-2 with two lanes (card-sized RPis; CM4-module has 4 lanes!) should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;)

  • The iframe-peroid of 2 frames is quiet aggressive.
  • I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate!

  Hint: For better reading pipelines are described as pipeline files which one can use with cat pipeline-file.txt | xargs gst-launch-1.0.

Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: blog-post by Mathieu.

Documentation: rtpst2022-1-fecdec, rtpst2022-1-fecenc

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.

  • Works per multicast as well as per unicast.
  • Latency: ca. 1 second
  • Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.
  • Receiver needs to be started before (first) launch of sender. Help needed! Issue at Mathieu.
  • Don't forget to remove “netsim drop-probability=0.01” before flight!

  Sender, using audio from HDMI-In and sending video-input to first HDMI-display port:

  rtpbin name=rtp fec-encoders='fec,0=“rtpst2022-1-fecenc\ rows\=4\ columns\=4”;'
  v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! “video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601” ! tee name=input ! v4l2h264enc output-io-mode=0 extra-controls=“controls,h264profile=4,h264level=11,videobitrate=15000000,videobitratemode=0,h264iframeperiod=2;” ! “video/x-h264,profile=high, level=(string)4” ! h264parse ! mpegtsmux name=mux ! rtpmp2tpay ssrc=0 ! rtp.sendrtpsink0
  rtp.sendrtpsrc0 ! udpsink host=224.0.23.1 port=5000 multicast-iface=eth0
  rtp.sendfecsrc00 ! udpsink host=224.0.23.1 port=5002 async=false multicast-iface=eth0
  rtp.sendfecsrc_0_1 ! udpsink host=224.0.23.1 port=5004 async=false multicast-iface=eth0
  alsasrc device=hw:tc358743 do-timestamp=true ! “audio/x-raw,format=S16LE,rate=48000,channels=2” ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux.
  input. ! queue ! videoconvert ! fbdevsink device=“/dev/fb0” sync=false

Receiver, with audio-out on Headphone-jack:

rtpbin 
  latency=100 
  fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";' 
  name=rtp
    
udpsrc 
  address=224.0.23.1 
  port=5002 
  caps="application/x-rtp, payload=96" 
! queue 
! rtp.recv_fec_sink_0_0

udpsrc
  address=224.0.23.1 
  port=5004 
  caps="application/x-rtp, payload=96" 
! queue 
! rtp.recv_fec_sink_0_1

udpsrc
  address=224.0.23.1
  port=5000
  caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33" 
! queue 
! netsim 
    drop-probability=0.01 
! rtp.recv_rtp_sink_0

rtp. 
! queue 
! decodebin 
    name=decode
! videoconvert 
! queue 
! kmssink

decode. 
! audioconvert 
! queue 
! alsasink 
    card-name="bcm2835 Headphones"

• Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.

  • Latency: ca. 0.7 seconds
  • SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.
  • starting/restarting-order seems strange and needs to be investigated.

  Sender, using audio from HDMI-In and sending video-input to display:

  v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! “video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601” ! tee name=input ! v4l2h264enc output-io-mode=0 extra-controls=“controls,h264profile=4,h264level=11,videobitrate=15000000,videobitratemode=0,h264iframeperiod=2;” ! “video/x-h264,profile=high, level=(string)4” ! mpegtsmux name=mux ! srtsink uri=srt://:8888 latency=40 mode=listener wait-for-connection=true
  input. ! queue ! videoconvert ! fbdevsink sync=false
  alsasrc device=hw:tc358743 do-timestamp=true ! “audio/x-raw,format=S16LE,rate=48000,channels=2” ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux.
  Receiver, with audio-out on Headphone-jack:

  srtclientsrc latency=20 mode=caller uri=srt:${ipAddress}:8888 ! tsdemux latency=40 name=demux
  demux. ! queue ! decodebin name=decode ! videoconvert ! queue ! kmssink
  demux. ! queue ! avdec_aac ! audioconvert ! alsasink card-name=“bcm2835 Headphones”

  === RIST streaming (multicast/unicast), audio and display-loop-out included

  RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too?

  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024. * Latency: reducable to 0.5s * most errors

  Sender, using audio from HDMI-In and sending video-input to display:

  v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! “video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601” ! tee name=input ! v4l2h264enc output-io-mode=0 extra-controls=“controls,h264profile=4,h264level=11,videobitrate=15000000,videobitratemode=0,h264iframeperiod=2,sequenceheadermode=0;” ! “video/x-h264,profile=high, level=(string)4” ! mpegtsmux name=mux ! rtpmp2tpay ! ristsink address=224.0.23.1 port=5004 multicast-iface=eth0
  input. ! queue ! videoconvert ! fbdevsink sync=false \
  alsasrc device=hw:tc358743 do-timestamp=true ! “audio/x-raw,format=S16LE,rate=48000,channels=2” ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux.

  Receiver, with audio-out on headphone-jack:

  ristsrc address=224.0.23.1 receiver-buffer=200 reorder-section=20 ! rtpmp2tdepay ! tsdemux latency=50 name=demux
  demux. ! queue ! decodebin name=decode ! videoconvert ! queue ! kmssink
  demux. ! queue ! avdec_aac ! audioconvert ! alsasink card-name=“bcm2835 Headphones”