Encoding-Quality-Testing

Created with the vmaf-docker repo.

Testing VAAPI vs. software encoding for live streaming

The graphs currently contain no confidence intervals (values may differ, but the difference may not be relevant)

The codec settings are the ones currently in use by the C3VOC for live streaming.

All the reference content is 1080p25.

Codec Settings

vaapi: nothing but keyint and bitrates set

-vaapi_device /dev/dri/renderD128
-hwaccel vaapi -hwaccel_output_format vaapi
-i $ref
-vf 'format=nv12|vaapi,hwupload'
-c:v {codec}_vaapi
-keyint_min:v 75 -g:v 75
-b:v {rate}k -maxrate:v {rate}k -bufsize {rate}k

libvpx-vp9 (current C3VOC transcoding settings)

-i $ref
-c:v libvpx-vp9
-deadline:v realtime -cpu-used:v 8
-threads:v 8
-frame-parallel:v 1 -tile-columns:v 2
-keyint_min:v 75 -g:v 75
-crf:v 23
-b:v {rate}k -maxrate:v {rate}k -bufsize {rate}k

x264 (current C3VOC encoder-stream settings)

-i $ref
-c:v libx264 -preset:v veryfast
-profile:v main -flags +cgop
-threads:v 0 -g:v 75
-crf:v 21
-maxrate:v {rate}k -bufsize {rate}k

The codecs are tested with bitrates from 1Mbit/s to 5.2Mbit/s.Currently the 1080p25 streams are coded with 2.8Mbit/s maxrate.

Versions

CPU: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz

ffmpeg-4.1.4

libvmaf-1.3.15

libvpx-1.7.0 (buster)

libx264-2:0.155.2917+git0a84d98-2 (buster)

Reference

Xiph Content is found on https://media.xiph.org/video/derf/

Ref1 - Noisy MPEG2 36C3

Ref2 - MPEG2 36C3

Red Kayak - Xiph (converted to 8Bit 25FPS)

Rush Hour - Xiph

Results

Reference	Intel 8th Gen (i7-8665U)	AMD Navi (RX 5700 XT) + AMD Ryzen 5 2600 navi_scores.json
Ref1	bitrates_ref1.pdf speed_ref1.pdf * no improvement beyond 2Mbit/s * VAAPI HEVC & h.264 with best quality * libvpx-VP9 better and faster than x264 * VAAPI HEVC not faster than software encoding	navi_bitrates_ref1.nut.pdf navi_speed_ref1.nut.pdf
Ref2	bitrates_ref2.pdf speed_ref2.pdf * no improvement beyond 1.5Mbit/s * similar Mean for all codecs * better Minimum scores for libvpx/x264 * high speeds for libvpx/x264	navi_bitrates_ref2.nut.pdf navi_speed_ref2.nut.pdf
Red Kayak	bitrates_red_kayak.pdf speed_red_kayak.pdf * very good quality never reached * VAAPI VP9 has rate control problems at low bitrates (resulting bitrate large but quality low) * VAAPI HEVC > VAAPI h.264 > libvpx-VP9 at high bitrates * libvpx-VP9 best at low bitrates * libvpx-VP9 better than VAAPI VP9, albeit slower * libvpx-VP9 better and faster than x264	navi_bitrates_red_kayak.nut.pdf navi_speed_red_kayak.nut.pdf
Rush Hour	bitrates_rush_hour.pdf speed_rush_hour.pdf VAAPI HEVC > VAAPI h.264 > libvpx-VP9 > VAAPI-VP9 > x264	navi_bitrates_rush_hour.nut.pdf navi_speed_rush_hour.nut.pdf
Forest	navi_bitrates_rush_hour.nut.pdf navi_speed_rush_hour.nut.pdf

Intel Conclusion

Keep in mind that this only applies to the stream-settings used

Quality: VAAPI HEVC > VAAPI h.264 > libvpx-VP9 > VAAPI-VP9 > x264

Speed: VAAPI h.264 > VAAPI VP9 > VAAPI HEVC >= libvpx-VP9 ~= x264

VAAPI VP9 has lower speeds and quality than VAAPI h.264

libvpx-VP9 has higher quality than x264 at the same speeds

VAAPI-HEVC outperforms VAAPI-h.264 on hard content, but is 3x slower