hardware:video-pis

Differences

This shows you the differences between two versions of the page.

Link to this comparison view

Both sides previous revision Previous revision
Next revision		 Previous revision
hardware:video-pis [2024/03/20 00:45] – wtfhardware:video-pis [2024/07/15 16:32] (current) – fix formatting wtf
Line 3: Line 3:
 Research-documentation for H264-encoding and streaming with Raspberry Pis. Research-documentation for H264-encoding and streaming with Raspberry Pis.
  
-== gstreamer+== gstreamer with Auvidea B101 HDMI-CSI2-Bridge
  
-=== SMPTE2022-1 FEC per UDP streaming with Auvidea B101 HDMI-CSI2-Bridge, audio and display-loop-out included+ * max. supported resolution: 1080p30. [[https://forums.raspberrypi.com/viewtopic.php?t=268787|Theoretically]] CSI-2 with two lanes (card-sized RPis; CM4-module has 4 lanes!) should be capable of handling 1080p50 in UYVY colorspace, but this fails so far, even as UYVY seems to be in use. (It's unclear if everything will literally melt, too ;) 
 + * The iframe-peroid of 2 frames is quiet aggressive. 
 + * I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate! 
 + 
 +Hint: For better reading pipelines are described as pipeline files which one can use with `cat pipeline-file.txt | xargs gst-launch-1.0`. 
 + 
 +=== SMPTE2022-1 FEC streaming (multicast/unicast), audio and display-loop-out included
  
-Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer: [[https://mathieuduponchelle.github.io/2020-10-09-SMPTE-2022-1-2D-Forward-Error-Correction-in-GStreamer.html|blog-post by Mathieu]]. +Since 2020 there is an implementation of SMPTE 2022-1 in gstreamer, done in MPEGTS via UDP with 2D-FEC: [[https://mathieuduponchelle.github.io/2020-10-09-SMPTE-2022-1-2D-Forward-Error-Correction-in-GStreamer.html|blog-post by Mathieu]]. 
  
 Documentation: [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecdec.html|rtpst2022-1-fecdec]], [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecenc.html|rtpst2022-1-fecenc]] Documentation: [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecdec.html|rtpst2022-1-fecdec]], [[https://gstreamer.freedesktop.org/documentation/rtpmanager/rtpst2022-1-fecenc.html|rtpst2022-1-fecenc]]
Line 14: Line 20:
  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
  * Works per multicast as well as per unicast.  * Works per multicast as well as per unicast.
- * Latency: 1-1.1 seconds+ * Latency: ca. 1 second
  * Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.  * Receiver needs a static route towards 224.0.0.0/4 on eth0 in order to receive multicast.
  * Receiver needs to be started before (first) launch of sender. Help needed! [[https://github.com/MathieuDuponchelle/MathieuDuponchelle.github.io/issues/11|Issue at Mathieu]].  * Receiver needs to be started before (first) launch of sender. Help needed! [[https://github.com/MathieuDuponchelle/MathieuDuponchelle.github.io/issues/11|Issue at Mathieu]].
- * I/O-mode = auto. Result unclear, it seems popular stating DMAbuf here. Investigate! 
- * The iframe-peroid of 2 frames is quiet aggressive. 
  * Don't forget to remove "netsim drop-probability=0.01" before flight!  * Don't forget to remove "netsim drop-probability=0.01" before flight!
  
-Sender, using Audio from HDMI-In and sending video-input to display:+Sender, using audio from HDMI-In and sending video-input to first HDMI-display port:
  
-  gst-launch-1.0  \ 
-      rtpbin name=rtp fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";' \ 
-        v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" ! \ 
-        tee name=input ! \ 
-        v4l2h264enc output-io-mode=0 extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" ! \ 
-        "video/x-h264,profile=high, level=(string)4" ! h264parse ! \ 
-        mpegtsmux name=mux ! rtpmp2tpay ssrc=0 ! rtp.send_rtp_sink_0  \ 
-          rtp.send_rtp_src_0 ! udpsink host=${ipAddress} port=5000 multicast-iface=eth0 \ 
-          rtp.send_fec_src_0_0 ! udpsink host=${ipAddress} port=5002 async=false multicast-iface=eth0 \ 
-          rtp.send_fec_src_0_1 ! udpsink host=${ipAddress} port=5004 async=false multicast-iface=eth0 \ 
-        alsasrc device=hw:tc358743 do-timestamp=true ! audio/x-raw,format=S16LE,rate=48000,channels=2 ! \ 
-        audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux. \ 
-        input. ! queue ! videoconvert ! fbdevsink device="/dev/fb0" sync=false   
  
 +  rtpbin 
 +    name=rtp 
 +    fec-encoders='fec,0="rtpst2022-1-fecenc\ rows\=4\ columns\=4";'
 +  
 +  v4l2src 
 +    device=/dev/video0 
 +    io-mode=0
 +    do-timestamp=true 
 +  ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"
 +  ! tee 
 +    name=input
 +  ! v4l2h264enc
 +    output-io-mode=0 
 +    extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 
 +  ! "video/x-h264,profile=high, level=(string)4" 
 +  ! h264parse 
 +  ! mpegtsmux 
 +    name=mux 
 +  ! rtpmp2tpay 
 +    ssrc=0 
 +  ! rtp.send_rtp_sink_0
 +  
 +  rtp.send_rtp_src_0 
 +  ! udpsink 
 +    host=224.0.23.1 
 +    port=5000 
 +    multicast-iface=eth0 
 +  
 +  rtp.send_fec_src_0_0 
 +  ! udpsink 
 +    host=224.0.23.1 
 +    port=5002 
 +    async=false 
 +    multicast-iface=eth0
 +  
 +  rtp.send_fec_src_0_1 
 +  ! udpsink 
 +    host=224.0.23.1 
 +    port=5004 
 +    async=false 
 +    multicast-iface=eth0
 +  
 +  alsasrc
 +    device=hw:tc358743 
 +    do-timestamp=true 
 +  ! "audio/x-raw,format=S16LE,rate=48000,channels=2" 
 +  ! audioconvert 
 +  ! voaacenc 
 +    bitrate=48000 
 +  ! aacparse 
 +  ! queue 
 +  ! mux.
 +  
 +  input. 
 +  ! queue 
 +  ! videoconvert 
 +  ! fbdevsink 
 +    device="/dev/fb0" 
 +    sync=false
  
 Receiver, with audio-out on Headphone-jack: Receiver, with audio-out on Headphone-jack:
  
-  gst-launch-1.0 \ +  rtpbin  
-    rtpbin latency=100 fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";' name=rtp \ +    latency=100  
-    udpsrc address=224.0.23.1 port=5002 caps="application/x-rtp, payload=96" ! queue ! rtp.recv_fec_sink_0_0 \ +    fec-decoders='fec,0="rtpst2022-1-fecdec\ size-time\=100000000";'  
-    udpsrc address=224.0.23.1 port=5004 caps="application/x-rtp, payload=96" ! queue ! rtp.recv_fec_sink_0_1 \ +    name=rtp 
-    udpsrc address=224.0.23.1 port=5000 caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33" ! \ +       
-      queue ! netsim drop-probability=0.01 ! rtp.recv_rtp_sink_0 \ +  udpsrc  
-    rtp. ! queue ! decodebin name=decode  ! videoconvert ! queue !  kmssink \ +    address=224.0.23.1  
-    decode. ! audioconvert ! queue ! alsasink card-name="bcm2835 Headphones"+    port=5002  
 +    caps="application/x-rtp, payload=96"  
 +  ! queue  
 +  ! rtp.recv_fec_sink_0_0 
 +   
 +  udpsrc 
 +    address=224.0.23.1  
 +    port=5004  
 +    caps="application/x-rtp, payload=96"  
 +  ! queue  
 +  ! rtp.recv_fec_sink_0_1 
 +   
 +  udpsrc 
 +    address=224.0.23.1 
 +    port=5000 
 +    caps="application/x-rtp, media=video, clock-rate=90000, encoding-name=mp2t, payload=33"  
 +  ! queue  
 +  ! netsim  
 +      drop-probability=0.01  
 +  ! rtp.recv_rtp_sink_0 
 +   
 +  rtp.  
 +  ! queue  
 +  ! decodebin  
 +      name=decode 
 +  ! videoconvert  
 +  ! queue  
 +  ! kmssink 
 +   
 +  decode.  
 +  ! audioconvert  
 +  ! queue  
 +  ! alsasink  
 +      card-name="bcm2835 Headphones"
  
- +=== SRT streaming (unicast), audio and display-loop-out included
-=== SRT streaming with Auvidea B101 HDMI-CSI2-Bridge, audio and display-loop-out included+
  
  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.  * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024.
- * Latency: 0.6-0.8 seconds + * Latency: ca. 0.7 seconds 
- * SRT-desired latency is with 20ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.+ * SRT-desired latency is with 40ms VERY aggressive! The suggested value is 2.5 times the round-trip-time (RTT) between both hosts.
  * starting/restarting-order seems strange and needs to be investigated.  * starting/restarting-order seems strange and needs to be investigated.
  
-Sender:+Sender, using audio from HDMI-In and sending video-input to display: 
 + 
 +  v4l2src  
 +      device=/dev/video0  
 +      io-mode=0  
 +      do-timestamp=true  
 +  ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"  
 +  ! tee  
 +      name=input 
 +  ! v4l2h264enc  
 +      output-io-mode=0  
 +      extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" 
 +  ! "video/x-h264,profile=high, level=(string)4" 
 +  ! mpegtsmux  
 +      name=mux  
 +  ! srtsink  
 +      uri=srt://:8888  
 +      latency=40  
 +      mode=listener  
 +      wait-for-connection=true 
 +   
 +  input.  
 +  ! queue  
 +  ! videoconvert  
 +  ! fbdevsink  
 +      sync=false 
 +   
 +  alsasrc  
 +      device=hw:tc358743  
 +      do-timestamp=true  
 +  ! "audio/x-raw,format=S16LE,rate=48000,channels=2"  
 +  ! audioconvert  
 +  ! voaacenc  
 +      bitrate=48000  
 +  ! aacparse  
 +  ! queue  
 +  ! mux.  
 +   
 +Receiver, with audio-out on Headphone-jack: 
 + 
 +  srtclientsrc  
 +      latency=20  
 +      mode=caller  
 +      uri=srt://${ipAddress}:8888  
 +  ! tsdemux  
 +      latency=40  
 +      name=demux  
 +   
 +  demux.  
 +  ! queue  
 +  ! decodebin  
 +      name=decode  
 +  ! videoconvert  
 +  ! queue  
 +  ! kmssink 
 +   
 +  demux.  
 +  ! queue  
 +  ! avdec_aac  
 +  ! audioconvert  
 +  ! alsasink  
 +      card-name="bcm2835 Headphones" 
 + 
 + 
 +=== RIST streaming (multicast/unicast), audio and display-loop-out included 
 + 
 +RIST seems to be the new kid around the corner of SRT. No FEC, but re-transmission-requests with RTCP, which shall work via multicast, too? 
 + 
 + * Tested by wtf with an Raspberry Pi 4B+ with Raspberry Pi OS 2023-10-10, Debian Bookworm each, in March 2024. 
 + * Latency: reducable to 0.5s 
 + * most errors 
 + 
 +Sender, using audio from HDMI-In and sending video-input to display:
  
-  gst-launch-1.0 -v \ +  v4l2src  
-    v4l2src device=/dev/video0 io-mode=0 do-timestamp=true ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601" ! \ +      device=/dev/video0  
-    tee name=input ! \ +      io-mode=0  
-    v4l2h264enc output-io-mode=0 extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2;" ! \ +      do-timestamp=true  
-    "video/x-h264,profile=high, level=(string)4" ! \ +  ! "video/x-raw,framerate=30/1,format=UYVY,colorimetry=bt601"  
-    mpegtsmux name=mux ! srtsink uri=srt://:8888 latency=40 mode=listener wait-for-connection=true \ +  ! tee  
-    input. ! queue ! videoconvert ! fbdevsink sync=false \ +      name=input  
-    alsasrc device=hw:tc358743 do-timestamp=true ! audio/x-raw,format=S16LE,rate=48000,channels=2 ! audioconvert ! voaacenc bitrate=48000 ! aacparse ! queue ! mux.  +  ! v4l2h264enc  
-     +      output-io-mode=0  
-Receiver:+      extra-controls="controls,h264_profile=4,h264_level=11,video_bitrate=15000000,video_bitrate_mode=0,h264_i_frame_period=2,sequence_header_mode=0;"  
 +  ! "video/x-h264,profile=high, level=(string)4"  
 +  ! mpegtsmux  
 +      name=mux  
 +  ! rtpmp2tpay  
 +  ! ristsink  
 +      address=224.0.23.1  
 +      port=5004  
 +      multicast-iface=eth0 
 +   
 +  input.  
 +  ! queue  
 +  ! videoconvert  
 +  ! fbdevsink  
 +      sync=false \ 
 +       
 +  alsasrc  
 +      device=hw:tc358743  
 +      do-timestamp=true  
 +  ! "audio/x-raw,format=S16LE,rate=48000,channels=2"  
 +  ! audioconvert  
 +  ! voaacenc  
 +      bitrate=48000  
 +  ! aacparse  
 +  ! queue  
 +  ! mux.
  
-  gst-launch-1.0 \ +Receiver, with audio-out on headphone-jack:
-    srtclientsrc latency=20 mode=caller uri=srt://${ipAddress}:8888 ! \ +
-    tsdemux latency=20 name=demux demux. ! queue ! decodebin name=decode ! videoconvert ! queue ! kmssink \ +
-    demux. ! queue ! avdec_aac ! audioconvert ! alsasink card-name="bcm2835 Headphones"+
  
 +  ristsrc 
 +      address=224.0.23.1 
 +      receiver-buffer=200 
 +      reorder-section=20 
 +  ! rtpmp2tdepay 
 +  ! tsdemux 
 +      latency=50 
 +      name=demux 
 +  
 +  demux. 
 +  ! queue 
 +  ! decodebin 
 +      name=decode 
 +  ! videoconvert 
 +  ! queue 
 +  ! kmssink
 +  
 +  demux. 
 +  ! queue 
 +  ! avdec_aac 
 +  ! audioconvert 
 +  ! alsasink 
 +      card-name="bcm2835 Headphones"
  • hardware/video-pis.1710891934.txt.gz
  • Last modified: 2024/03/20 00:45
  • by wtf