hardware:video-pis

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
hardware:video-pis [2024/03/20 00:45] – wtfhardware:video-pis [2024/04/03 21:13] (current) – wtf
Line 3: Line 3:
 Research-documentation for H264-encoding and streaming with Raspberry Pis. Research-documentation for H264-encoding and streaming with Raspberry Pis.
  
-== gstreamer+== gstreamer with Auvidea B101 HDMI-CSI2-Bridge
  
-=== SMPTE2022-1 FEC per UDP streaming with Auvidea B101 HDMI-CSI2-Bridge, audio and display-loop-out included+ * max. supported resolution: 1080p30. [[https://forums.raspberrypi.com/viewtopic.php?t=268787|Theoretically]] CSI-2 with two lanes (card-sized RPis; CM4-module has 4 lanes!) should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;) 
 + * The iframe-peroid of 2 frames is quiet aggressive. 
 + * I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate! 
 + 
 +Hint: For better reading pipelines are described as pipeline files which one can use with `cat pipeline-file.txt | xargs gst-launch-1.0`. 
 + 
 +=== SMPTE2022-1 FEC streaming (multicast/unicast), audio and display-loop-out included
  
-Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer: [[https://mathieuduponchelle.github.io/2020-10-09-SMPTE-2022-1-2D-Forward-Error-Correction-in-GStreamer.html|blog-post by Mathieu]]. +Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: [[https://mathieuduponchelle.github.io/2020-10-09-SMPTE-2022-1-2D-Forward-Error-Correction-in-GStreamer.html|blog-post by Mathieu]]. 
  
 Documentation: [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecdec.html|rtpst2022-1-fecdec]], [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecenc.html|rtpst2022-1-fecenc]] Documentation: [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecdec.html|rtpst2022-1-fecdec]], [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecenc.html|rtpst2022-1-fecenc]]
Line 14: Line 20:
  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
  * Works per multicast as well as per unicast.  * Works per multicast as well as per unicast.
- * Latency: 1-1.1 seconds+ * Latency: ca. 1 second
  * Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.  * Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.
  * Receiver needs to be started before (first) launch of sender. Help needed! [[https://github.com/MathieuDuponchelle/MathieuDuponchelle.github.io/issues/11|Issue at Mathieu]].  * Receiver needs to be started before (first) launch of sender. Help needed! [[https://github.com/MathieuDuponchelle/MathieuDuponchelle.github.io/issues/11|Issue at Mathieu]].
- * I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate! 
- * The iframe-peroid of 2 frames is quiet aggressive. 
  * Don't forget to remove "netsim drop-probability=0.01" before flight!  * Don't forget to remove "netsim drop-probability=0.01" before flight!
  
-Sender, using Audio from HDMI-In and sending video-input to display:+Sender, using audio from HDMI-In and sending video-input to first HDMI-display port:
  
-  gst-launch-1.0  \ 
-      rtpbin name=rtp fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";' \ 
-        v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" ! \ 
-        tee name=input ! \ 
-        v4l2h264enc output-io-mode=0 extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" ! \ 
-        "video/x-h264,profile=high, level=(string)4" ! h264parse ! \ 
-        mpegtsmux name=mux ! rtpmp2tpay ssrc=0 ! rtp.send_rtp_sink_0  \ 
-          rtp.send_rtp_src_0 ! udpsink host=${ipAddress} port=5000 multicast-iface=eth0 \ 
-          rtp.send_fec_src_0_0 ! udpsink host=${ipAddress} port=5002 async=false multicast-iface=eth0 \ 
-          rtp.send_fec_src_0_1 ! udpsink host=${ipAddress} port=5004 async=false multicast-iface=eth0 \ 
-        alsasrc device=hw:tc358743 do-timestamp=true ! audio/x-raw,format=S16LE,rate=48000,channels=2 ! \ 
-        audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux. \ 
-        input. ! queue ! videoconvert ! fbdevsink device="/dev/fb0" sync=false   
  
 +```
 +rtpbin 
 +  name=rtp 
 +  fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";'
 +
 +v4l2src 
 +  device=/dev/video0 
 +  io-mode=0
 +  do-timestamp=true 
 +! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
 +! tee 
 +  name=input
 +! v4l2h264enc
 +  output-io-mode=0 
 +  extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 
 +! "video/x-h264,profile=high, level=(string)4" 
 +! h264parse 
 +! mpegtsmux 
 +  name=mux 
 +! rtpmp2tpay 
 +  ssrc=0 
 +! rtp.send_rtp_sink_0
 +
 +rtp.send_rtp_src_0 
 +! udpsink 
 +  host=224.0.23.1 
 +  port=5000 
 +  multicast-iface=eth0 
 +
 +rtp.send_fec_src_0_0 
 +! udpsink 
 +  host=224.0.23.1 
 +  port=5002 
 +  async=false 
 +  multicast-iface=eth0
 +
 +rtp.send_fec_src_0_1 
 +! udpsink 
 +  host=224.0.23.1 
 +  port=5004 
 +  async=false 
 +  multicast-iface=eth0
 +
 +alsasrc
 +  device=hw:tc358743 
 +  do-timestamp=true 
 +! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
 +! audioconvert 
 +! voaacenc 
 +  bitrate=48000 
 +! aacparse 
 +! queue 
 +! mux.
 +
 +input. 
 +! queue 
 +! videoconvert 
 +! fbdevsink 
 +  device="/dev/fb0" 
 +  sync=false
 +```
  
 Receiver, with audio-out on Headphone-jack: Receiver, with audio-out on Headphone-jack:
  
-  gst-launch-1.0 \ +``` 
-    rtpbin latency=100 fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";' name=rtp \ +rtpbin  
-    udpsrc address=224.0.23.1 port=5002 caps="application/x-rtp, payload=96" ! queue ! rtp.recv_fec_sink_0_0 \ +  latency=100  
-    udpsrc address=224.0.23.1 port=5004 caps="application/x-rtp, payload=96" ! queue ! rtp.recv_fec_sink_0_1 \ +  fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";'  
-    udpsrc address=224.0.23.1 port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33" ! \ +  name=rtp 
-      queue ! netsim drop-probability=0.01 ! rtp.recv_rtp_sink_0 \ +     
-    rtp. ! queue ! decodebin name=decode  ! videoconvert ! queue !  kmssink \ +udpsrc  
-    decode. ! audioconvert ! queue ! alsasink card-name="bcm2835 Headphones"+  address=224.0.23.1  
 +  port=5002  
 +  caps="application/x-rtp, payload=96"  
 +! queue  
 +! rtp.recv_fec_sink_0_0
  
 +udpsrc
 +  address=224.0.23.1 
 +  port=5004 
 +  caps="application/x-rtp, payload=96" 
 +! queue 
 +! rtp.recv_fec_sink_0_1
  
-=== SRT streaming with Auvidea B101 HDMI-CSI2-Bridge, audio and display-loop-out included+udpsrc 
 +  address=224.0.23.1 
 +  port=5000 
 +  caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33"  
 +! queue  
 +! netsim  
 +    drop-probability=0.01  
 +! rtp.recv_rtp_sink_0 
 + 
 +rtp.  
 +! queue  
 +! decodebin  
 +    name=decode 
 +! videoconvert  
 +! queue  
 +! kmssink 
 + 
 +decode.  
 +! audioconvert  
 +! queue  
 +! alsasink  
 +    card-name="bcm2835 Headphones" 
 +``` 
 + 
 +=== SRT streaming (unicast), audio and display-loop-out included
  
  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
- * Latency: 0.6-0.8 seconds + * Latency: ca. 0.7 seconds 
- * SRT-desired latency is with 20ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.+ * SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.
  * starting/restarting-order seems strange and needs to be investigated.  * starting/restarting-order seems strange and needs to be investigated.
  
-Sender:+Sender, using audio from HDMI-In and sending video-input to display:
  
-  gst-launch-1.0 -v \ +``` 
-    v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" ! \ +v4l2src  
-    tee name=input ! \ +    device=/dev/video0  
-    v4l2h264enc output-io-mode=0 extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" ! \ +    io-mode=0  
-    "video/x-h264,profile=high, level=(string)4" ! \ +    do-timestamp=true  
-    mpegtsmux name=mux ! srtsink uri=srt://:8888 latency=40 mode=listener wait-for-connection=true \ +! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"  
-    input. ! queue ! videoconvert ! fbdevsink sync=false \ +! tee  
-    alsasrc device=hw:tc358743 do-timestamp=true ! audio/x-raw,format=S16LE,rate=48000,channels=2 ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux. +    name=input 
 +! v4l2h264enc  
 +    output-io-mode=0  
 +    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 
 +! "video/x-h264,profile=high, level=(string)4" 
 +! mpegtsmux  
 +    name=mux  
 +! srtsink  
 +    uri=srt://:8888  
 +    latency=40  
 +    mode=listener  
 +    wait-for-connection=true 
 + 
 +input.  
 +! queue  
 +! videoconvert  
 +! fbdevsink  
 +    sync=false 
 + 
 +alsasrc  
 +    device=hw:tc358743  
 +    do-timestamp=true  
 +! "audio/x-raw,format=S16LE,rate=48000,channels=2"  
 +! audioconvert  
 +! voaacenc  
 +    bitrate=48000  
 +! aacparse  
 +! queue  
 +! mux.  
 +``` 
 + 
 +Receiver, with audio-out on Headphone-jack: 
 + 
 +``` 
 +srtclientsrc  
 +    latency=20  
 +    mode=caller  
 +    uri=srt://${ipAddress}:8888  
 +! tsdemux  
 +    latency=40  
 +    name=demux  
 + 
 +demux.  
 +! queue  
 +! decodebin  
 +    name=decode  
 +! videoconvert  
 +! queue  
 +! kmssink 
 + 
 +demux.  
 +! queue  
 +! avdec_aac  
 +! audioconvert  
 +! alsasink  
 +    card-name="bcm2835 Headphones" 
 +``` 
 + 
 +=== RIST streaming (multicast/unicast), audio and display-loop-out included 
 + 
 +RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too? 
 + 
 + * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024. 
 + * Latency: reducable to 0.5s 
 + * most errors 
 + 
 +Sender, using audio from HDMI-In and sending video-input to display: 
 + 
 +``` 
 +v4l2src  
 +    device=/dev/video0  
 +    io-mode=0  
 +    do-timestamp=true  
 +! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"  
 +! tee  
 +    name=input  
 +! v4l2h264enc  
 +    output-io-mode=0  
 +    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2,sequence_header_mode=0;"  
 +! "video/x-h264,profile=high, level=(string)4"  
 +! mpegtsmux  
 +    name=mux  
 +! rtpmp2tpay  
 +! ristsink  
 +    address=224.0.23.1  
 +    port=5004  
 +    multicast-iface=eth0 
 + 
 +input.  
 +! queue  
 +! videoconvert  
 +! fbdevsink  
 +    sync=false \
          
-Receiver:+alsasrc  
 +    device=hw:tc358743  
 +    do-timestamp=true  
 +! "audio/x-raw,format=S16LE,rate=48000,channels=2"  
 +! audioconvert  
 +! voaacenc  
 +    bitrate=48000  
 +! aacparse  
 +! queue  
 +! mux. 
 +``` 
 + 
 +Receiver, with audio-out on headphone-jack: 
 + 
 +``` 
 +ristsrc  
 +    address=224.0.23.1  
 +    receiver-buffer=200  
 +    reorder-section=20  
 +! rtpmp2tdepay  
 +! tsdemux  
 +    latency=50  
 +    name=demux 
  
-  gst-launch-1.0 \ +demux.  
-    srtclientsrc latency=20 mode=caller uri=srt://${ipAddress}:8888 ! \ +! queue  
-    tsdemux latency=20 name=demux demux. ! queue ! decodebin name=decode ! videoconvert ! queue ! kmssink \ +! decodebin  
-    demux. ! queue ! avdec_aac ! audioconvert ! alsasink card-name="bcm2835 Headphones"+    name=decode  
 +! videoconvert  
 +! queue  
 +! kmssink
  
 +demux. 
 +! queue 
 +! avdec_aac 
 +! audioconvert 
 +! alsasink 
 +    card-name="bcm2835 Headphones"
 +```
  • hardware/video-pis.1710891934.txt.gz
  • Last modified: 2024/03/20 00:45
  • by wtf