Using Native WebRTC simulcast support in Chrome (or how to be as good as Hangouts) [WiP]

Some weeks ago Philipp Hancke presented an awesome analysis of how Google Hangouts uses WebRTC in Chrome.  In that blog post he explained that based on the SDP contents Google is using simulcast but didn't entered in the details of how to activate it.   So I had a lot of curiosity and thought that it could be great if people (beyond Google) could use this feature so I tried to replicate their behavior.

Step one: Add simulcast ssrcs and "SIM" group to the offer SDP

The first thing I tried is to implement some SDP mangling to make my SDP look like the Hangouts SDP.   That means adding 3 substreams grouped by simulcast semantics. This is the code of my quick and dirty implementation:

https://gist.github.com/ggarber/a19b4c33510028b9c657

Result: no simulcast :(

Step two?: Add a renegotiation

I saw a a renegotiation in the initial connection from Google Hangouts (when there are no other participants) and I thought this was needed to enable simulcast and implemented it but this is not needed to have simulcast.

Step three: Add the x-conference flag to the video track in the answer SDP

The third thing I tried was to set those misterious conference flags in the SDP offer:

var lines = answer.sdp.split("\r\n");
lines.splice(lines.length - 1, 0, "a=x-google-flag:conference");
answer.sdp = lines.join("\r\n")

Result:simulcast  !!!!

You can see this lines in Chrome Canary logs:
VP8 number of temporal layers: 3
Simulcast substream 0: 960x540@450-900kbps with  temporal layers
Simulcast substream 1: 1920x1080@600-2500kbps with  temporal layers
Simulcast substream 2: 3840x2160@600-2500kbps with  temporal layers

Bonus track: Apparently from those logs we not only enabled simulcast but also VP8 temporal scalability.   I have a snippet to parse the RTP packets and check it but didn't have time yet to confirm it.

Step four: Decide what substreams you are interested in

Based on the RTCP Receiver Reports  and PLIs received, Chrome will decide which streams to send from the list of streams you configured in the SDP (in Step one).

PENDING: The logic behind these decisions is not clear yet

Apparently you can optimize also the CPU usage by setting the googSkipUnusedFrames constraint that Google Hangouts uses and that way the video substreams that are not going to be sent won't be encoded either.   (To be tested)

Step five: Increase the bandwidth

Chrome changes the number and quality of the different substreams based on the bandwidth estimations.    The configuration (resolution, temporal layers) for a given original resolution and available bandwidth can not be changed as far as I can tell.

For example if I don't send any RTCP packet, by default it just uses 2 substreams:
Stream 320 x 240
Temporal layers 3, min br 50, target br 150, max br 200, qp max 56
Stream 640 x 480
Temporal layers 3, min br 150, target br 500, max br 700, qp max 56

To increase the number of substreams and quality you have to use the REMBs or maybe periodic RR.


Desperation note: At some point I was so frustrated not being able to get it working that I even tried to run my test page simulating I was behind the google.com domain to see if Chrome was behaving differently.

Comments

  1. UnknownJune 14, 2017 at 5:42 AM

    If you are already in or considering going into the field of early childhood education teaching, make sure you read this article first. Your expectations regarding salary and working conditions may be unrealistic, so it's good to be prepared before having your dreams dashed.http://www.careerpanth.xyz/

    ReplyDelete
  2. brogsSeptember 11, 2021 at 1:58 PM

    Hello! I need to know how to able google constraints like googEchoCancellation: true, but in chrome://webrtc-internals the flag doesnt appear, do you have any idea? thanks! michellavezzo@gmail.com

    ReplyDelete
    Replies
    1. brogsSeptember 11, 2021 at 2:27 PM

      I found a solution, just add a (optional) prop in constraints, and add goog flags inside. this link have an example:
      https://groups.google.com/g/meetecho-janus/c/Z0qZko86cO0

      Delete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay wil...
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms use...
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the...
Read more