Sending Packet Loss Feedback in WebRTC SFUs

One of the responsibilities of WebRTC SFUs is to receive and send RTCP packets.  RTCP packets include different types of feedback about audio and video streams and one of the most important RTCP packets is the Receiver Report (RR).

RR packets are sent from the receiver of the media stream towards the sender of that media stream.  In case of an SFU the RR are generated and sent from the SFU to the media stream Sender and also from every stream Receiver to the SFU (Figure 1).

The feedback sent inside RR packets include fields to calculate the round-trip-time delay, the jitter and the packet loss introduced by the network.


The packet loss reported in these RR packets is important because the audio and video being sent will be adjusted based on that parameter:
  • In case of audio streams the packet loss in the network modifies the level of robustness of the OPUS codec.   In presence of high packet loss the sender increases the level of redundancy of the forward error correction (FEC) included in the audio packets.
  • In case of video streams the packet loss modifies the video bitrate encoded and sent.   In presence of high packet loss the sender decreases the bitrate sent to reduce the possible congestion in the network.
Based on that behavior the question is... How should a SFU calculate the packet loss reported in the RR packets sent from the SFU to the Sender to get the best possible user experience?

In the next sections you can find a suggestion for how to handle this in three different types of streams: Audio, Video with simulcast and Video without simulcast.

Audio

Opus FEC is sent inband and because of that FEC is end to end (cannot be added/updated in an SFU) and everybody in the room will receive the same level of FEC.

If we want audio FEC to work well for the participant with the weakest link then we have to make sure we report the worse packet loss to the sender.

Because of that the packet loss reported from the SFU to the sender should be the worse packet of the receivers (max(PL1, PL2)).

For example the packet loss reported to the sender will be 20% if one of the receiving participants is experiencing a 20% packet loss in the downlink even if the sender and rest of receivers are in perfect network conditions.


Video with Simulcast

When using Simulcast the SFU has the ability to send a different video quality to each participant so there is no need to adapt the sender stream to any specific participant.

Because of that the packet loss reported from the SFU to the sender should be the packet loss in that link (Sender-SFU) no matter what the packet loss received the Receiver 1 and 2 is.

For example the packet loss reported to the sender will be 0 if the sender uplink is good even if the receivers are experiencing big packet loss in the downlink.   That will be corrected with retransmissions and bitrate adaptation done in the SFU by selection a lower resolution/layer to be forwarded to those participants.


Video with No-Simulcast  

When there is no Simulcast the SFU has to send the same video quality to every participant.  Because of that it is important that the sender adapts the sending bitrate to the participant with the weakest link (and/or disable the video for some participants).

This bitrate adaptation is mostly controlled by REMB RTCP packets where the SFU includes the bandwith available in the worse receiver.   But packet loss has also an impact on top of the bitrate reported in the REMB packets so we need to decide what packet loss to include in the RR packets.

In this case I think both approaches described in the previous sections can work reasonably well.   Either a) the SFU reports back the worse packet loss of the receivers or b) use the packet loss of each receiver to estimate its bandwidth and report only the sender side packet loss in the RR packets.

Note: If you use P2P connections when there are 2 participants in the room and switch to the SFU with Simulcast when there are 3 or more participants in the room you should be almost never in this Video with No-Simulcast and multiple participants case.


You can follow me in Twitter if you are interested in Real Time Communications.



Comments

  1. sebaDecember 12, 2018 at 4:50 AM

    The correct thing to do is highly dependant n the context. If your SFU has 20 clients with great connections and only 1 with a very bad connection. Do you really want to reduce image quality for everybody?

    ReplyDelete
  2. AnonymousMarch 5, 2019 at 12:43 AM

    Thanks for your great and helpful presentation I like your good service. I always appreciate your post. That is very interesting I love reading and I am always searching for informative information like this.angular 4 training in chennai | angularjs training in chennai | .angularjs best training center in chennai | angularjs training in velachery |

    ReplyDelete
  3. Logical softTechJuly 15, 2019 at 5:46 AM


    Your post is very good. I got to learn a lot from your post. Thank you for sharing your article for us. it is amazing post
    what is seo
    types of seo


    ReplyDelete
  4. HuongkvJanuary 19, 2021 at 10:51 PM

    Mua vé máy bay tại đại lý vé Aivivu, tham khảo

    ve may bay di my gia re

    vé máy bay từ mỹ về việt nam giá bao nhiêu

    vé máy bay từ đức về việt nam

    dịch vụ vé máy bay tại nga

    ReplyDelete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay wil...
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms use...
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the...
Read more