Encoding-Quality-Testing

This is an old revision of the document!

Created with the vmaf-docker repo.

The graphs currently contain no confidence intervals (values may differ, but the difference may not be relevant)

The codec settings are the ones currently in use by the C3VOC for live streaming.

All the reference content is 1080p25.

Codec Settings

vaapi: nothing but keyint and bitrates set

    -vaapi_device /dev/dri/renderD128
    -hwaccel vaapi -hwaccel_output_format vaapi
    -i $ref
    -vf 'format=nv12|vaapi,hwupload'
    -c:v {codec}_vaapi
    -keyint_min:v 75 -g:v 75
    -b:v {rate}k -maxrate:v {rate}k -bufsize {rate}k

libvpx-vp9 (current C3VOC transcoding settings)

    -i $ref
    -c:v libvpx-vp9
    -deadline:v realtime -cpu-used:v 8
    -threads:v 8
    -frame-parallel:v 1 -tile-columns:v 2
    -keyint_min:v 75 -g:v 75
    -crf:v 23
    -b:v {rate}k -maxrate:v {rate}k -bufsize {rate}k

x264 (current C3VOC encoder-stream settings)

    -i $ref
    -c:v libx264 -preset:v veryfast
    -profile:v main -flags +cgop
    -threads:v 0 -g:v 75
    -crf:v 21
    -maxrate:v {rate}k -bufsize {rate}k

The codecs are tested with bitrates from 1Mbit/s to 5.2Mbit/s. Currently the 1080p25 streams are coded with 2.8Mbit/s maxrate.

Versions

CPU: Intel(R) Core(TM) i7-8665U CPU @ 1.90GHz
ffmpeg-4.1.4
libvmaf-1.3.15
libvpx-1.7.0 (buster)
libx264-2:0.155.2917+git0a84d98-2 (buster)

Reference

Xiph Content is found on https://media.xiph.org/video/derf/

Ref1 - Noisy MPEG2 36C3
Ref2 - MPEG2 36C3
Red Kayak - Xiph (converted to 8Bit 25FPS)
Rush Hour - Xiph

Results

Platform	Label	Hardware Codecs	Comments
Intel 6th Gen (i7-6600U)	skylake	h.264, h.265	h.264 scores better than our VP9 and x264 streaming settings and is pretty fast h.265 even scores a little bit higher but is considerably slower
Intel 8th Gen (i7-8665U)	whiskeylake	h.264, h.265, VP9	h.264 and h.265 results are identical to skylake
AMD Navi (RX 5700 XT)	navi10	h.264, h.265	h.264 is worse than x264 on veryfast… h.265 produces 1920×1088 output?

Intel Conclusion

Keep in mind that this only applies to the stream-settings used

Quality: VAAPI HEVC > VAAPI h.264 > libvpx-VP9 > VAAPI-VP9 > x264
Speed: VAAPI h.264 > VAAPI VP9 > VAAPI HEVC >= libvpx-VP9 ~= x264
VAAPI VP9 has lower speeds and quality than VAAPI h.264
libvpx-VP9 has higher quality than x264 at the same speeds
VAAPI-HEVC outperforms VAAPI-h.264 on hard content, but is 3x slower