Measuring WebRTC video quality for different bitrates - Playing with VMAF

I've been wanting to play with Netflix Video Multi-Method Assessment Fusion (VMAF) for a while and yesterday I found the time and the motivation to give it a try.

Netflix VMAF is an algorithm to generate a video quality score by comparing a reference image/video with a distorted image/video.   To do that VMAF calculates scores using tradicional image quality metrics like VIF or DLM and then aggregate them using a Machine Learning model (SVM) trained with the videos and scores coming from real users.  Smart, isn't it?  (You can see a high level description of those metrics that are aggregated in this Netflix post or the Wikipedia page)

It is important to notice that VMAF works in a per-frame base so it is NOT a good tool to measure the quality impact of many artefacts happening in Real Time Communications (delays, reduced/frozen framerate, audio/video desync).   However we can use it to measure the impact of different encoding settings like the average bitrate of the encoding.

As you can expect coming from Netflix VMAF was designed for video streaming and trained using images from movies.   Anyway I was interested on seeing how it performs with mobile videoconference like videos so I recorded a short typical VGA video with a talking face not very stable.

I reencoded that sample video in VP8 using ffmpeg with different bitrates (50kbps, 100kbps, 200kbps, 400kbps, 600kpbs, 800kpbs, 1.2mbps, 2mpbs) and then used the ffmpeg2vmaf command line tool to calculate the score of those videos and presented them in the following graph:

VMAF scores for 

What we can see in this test is that beyond 600 (or even a little bit lower) the quality improvement is not that high and beyond 1200 it is barely noticeable.  Remember that these results are based on VMAF default model (not tuned for videoconference videos) and for my specific test video but the results don't look very different that what our experience with real users in production tells us.

This kind of test can help us decide the max bitrate we want to use for our WebRTC conferences, although there are other implications beyond quality like battery consumption and the results depend on the type of video, use case and how picky your users of your application are.   That's the reason why we have to be careful when playing with video bitrates in production.    As an example Facebook explained how increasing the bitrate lead to lower user scores because of the implications in battery consumption. In Houseparty we always do A/B testing to quantify the impact of any relevant change like this and decide the optimal video bitrate for our specific use case.

Google is including VMAF in the WebRTC test suite and implementing some frame alignment to overcome the limitation of having to compare a specific reference frame with the corresponding distorted one.   It would be nice if in the future we could expand the VMAF idea including new metrics in the ML algorithm to account for delays, framerate or video desynchronization.   That core idea of multi-method fusion looks very powerful!

You can follow me in Twitter if you are interested in Real Time Communications.

Comments

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay wil...
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms use...
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the...
Read more