Video-Pis

Research-documentation for H264-encoding and streaming with Raspberry Pis.

gstreamer with Auvidea B101 HDMI-CSI2-Bridge

max. supported resolution: 1080p30. Theoretically CSI-2 with two lanes should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;)

The iframe-peroid of 2 frames is quiet aggressive.

I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate!

=== SMPTE2022-1 FEC streaming (multicast/unicast), audio and display-loop-out included

Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: blog-post by Mathieu.

Documentation: rtpst2022-1-fecdec, rtpst2022-1-fecenc

Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.

Works per multicast as well as per unicast.

Latency: ca. 1 second

Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.

Receiver needs to be started before (first) launch of sender. Help needed! Issue at Mathieu.

Don't forget to remove “netsim drop-probability=0.01” before flight!

Sender, using Audio from HDMI-In and sending video-input to display:

Hint: For better reading pipelines are described as pipeline files which one can use with gst-launch-1.0 -f pipeline-file.txt.

rtpbin
name=rtp
fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";'
v4l2src
device=/dev/video0
io-mode=0
do-timestamp=true
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
! tee
name=input
! v4l2h264enc
output-io-mode=0
extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;"
! "video/x-h264,profile=high, level=(string)4"
! h264parse
! mpegtsmux
name=mux
! rtpmp2tpay
ssrc=0
! rtp.send_rtp_sink_0
rtp.send_rtp_src_0
! udpsink
host=${ipAddress}
port=5000
multicast-iface=eth0
rtp.send_fec_src_0_0
! udpsink
host=${ipAddress}
port=5002
async=false
multicast-iface=eth0
rtp.send_fec_src_0_1
! udpsink
host=${ipAddress}
port=5004
async=false
multicast-iface=eth0
alsasrc
device=hw:tc358743
do-timestamp=true
! "audio/x-raw,format=S16LE,rate=48000,channels=2"
! audioconvert
! voaacenc
bitrate=48000
! aacparse
! queue
! mux.
input.
! queue
! videoconvert
! fbdevsink
device="/dev/fb0"
sync=false Receiver, with audio-out on Headphone-jack:

rtpbin
latency=100
fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";'
name=rtp
udpsrc
address=224.0.23.1
port=5002
caps="application/x-rtp, payload=96"
! queue
! rtp.recv_fec_sink_0_0
udpsrc
address=224.0.23.1
port=5004
caps="application/x-rtp, payload=96"
! queue
! rtp.recv_fec_sink_0_1
udpsrc
address=224.0.23.1
port=5000
caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33"
! queue
! netsim
drop-probability=0.01
! rtp.recv_rtp_sink_0
rtp.
! queue
! decodebin
name=decode
! videoconvert
! queue
! kmssink
decode.
! audioconvert
! queue
! alsasink
card-name="bcm2835 Headphones"

SRT streaming (unicast), audio and display-loop-out included

Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.

Latency: ca. 0.7 seconds

SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.

starting/restarting-order seems strange and needs to be investigated.

Sender, using audio from HDMI-In and sending video-input to display:

v4l2src
device=/dev/video0
io-mode=0
do-timestamp=true
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
! tee
name=input
! v4l2h264enc
output-io-mode=0
extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;"
! "video/x-h264,profile=high, level=(string)4"
! mpegtsmux
name=mux
! srtsink
uri=srt://:8888
latency=40
mode=listener
wait-for-connection=true
input.
! queue
! videoconvert
! fbdevsink
sync=false
alsasrc
device=hw:tc358743
do-timestamp=true
! "audio/x-raw,format=S16LE,rate=48000,channels=2"
! audioconvert
! voaacenc
bitrate=48000
! aacparse
! queue
! mux.

Receiver, with audio-out on Headphone-jack:

srtclientsrc
latency=20
mode=caller
uri=srt://${ipAddress}:8888
! tsdemux
latency=40
name=demux
demux.
! queue
! decodebin
name=decode
! videoconvert
! queue
! kmssink
demux.
! queue
! avdec_aac
! audioconvert
! alsasink
card-name="bcm2835 Headphones"

RIST streaming (multicast/unicast), audio and display-loop-out included

RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too?

Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.

Latency: reducable to 0.5s

most errors

Sender, using audio from HDMI-In and sending video-input to display:

v4l2src
device=/dev/video0
io-mode=0
do-timestamp=true
! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
! tee
name=input
! v4l2h264enc
output-io-mode=0
extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2,sequence_header_mode=0;"
! "video/x-h264,profile=high, level=(string)4"
! mpegtsmux
name=mux
! rtpmp2tpay
! ristsink
address=224.0.23.1
port=5004
multicast-iface=eth0
input.
! queue
! videoconvert
! fbdevsink
sync=false \
alsasrc
device=hw:tc358743
do-timestamp=true
! "audio/x-raw,format=S16LE,rate=48000,channels=2"
! audioconvert
! voaacenc
bitrate=48000
! aacparse
! queue
! mux.

Receiver, with audio-out on headphone-jack:

ristsrc
address=224.0.23.1
receiver-buffer=200
reorder-section=20
! rtpmp2tdepay
! tsdemux
latency=50
name=demux
demux.
! queue
! decodebin
name=decode
! videoconvert
! queue
! kmssink
demux.
! queue
! avdec_aac
! audioconvert
! alsasink
card-name="bcm2835 Headphones"