How are retransmissions implemented in WebRTC

Executive Summary: This ended up being more complex than expected, if you are building an SFU just take the nack_module.cc & co files and use them.  Or even better, just use OpenTok or any other existing platform.   Anyway it is fun stuff to play with.

Introduction

When your network is congested and some of the packets are lost you need a mechanism to recover them.   If we ignore signal processing tricks (like stretching the previous and next audio packets in the jitter buffer) there are two alternatives:
  • Forward error correction: in each packet you add some extra information about the previous ones in case they are lost and you need to reconstruct them (flexfec is the new format to do this in WebRTC [1]).
  • Retransmissions: the sender will resend the packets usually after the receiver requests that with a negative acknowledgement message (NACK) indicating which packets were lost.
These mechanisms can be combined depending on the network conditions and can also be tuned for specific cases like scalable video as described in [2]. In Chrome the retransmissions are implemented for both audio and video but by default it is only enabled for video.

This post is about the retransmissions approach, specifically about when to ask for a retransmission and when not to do it.

Disclaimer: Please report any mistake/inaccuracy in this post and I will fix it asap for future reference. BTW this post is not about rtx "encapsulation" vs the legacy way of sending retransmissions but I'm happy to talk about that in twitter :)

Implementation in the RTP Receiver Side


The RTP receiver side is the one requesting the retransmission sending a NACK RTCP packet when it detects a missing packet.

But... should I request for retransmissions immediately after detecting a gap in the sequence numbers?  Google implementation does that plus also have a periodic timer to keep requesting them.

And... for how long should I keep requesting the retransmission? Chrome keeps requesting the retransmission of a specific packet unless the sequence number is "more than 10000 old", the number of missing packets in the list is larger than 1000, you asked for the same packet 10 times already or you have a new decodable full frame (no packet is missing for any other frame it depends on).

This is the Google implementation of WebRTC (and the same code is copied into Firefox and Safari) transcribed as pseudocode:


Implementation in the RTP Sender Side


The RTP sender side is the one receiving the retransmission requests (NACKs) and resending the lost packets if possible.

The main question here is if I should obey all the retransmission requests or not.  The way this is implemented in Google's WebRTC implementation right now is this one:
  1. Keep a copy of the packets sent in the last 1000 msecs (the "history").
  2. When a NACK is received try to send the packets requests if we still have them in the history.
  3. But... (rtp_sender.cc)
    • Ignore the request if the packet has been resent in the last RTT msecs.
    • Ignore the request if we are sending too many retransmissions.  There is a rate limiter (retransmission_rate_limiter) that is apparently configured to the whole channel bandwidth estimation.
    • If pacing is enabled insert this packet in the queue with normal priority.

Comments

  1. UnknownApril 7, 2017 at 3:12 PM

    great post man!
    Have a question. I'm searching the google code for the audio jitter buffer and not the video jitter buffer .
    I want to make some tests and wrap this thing in my app.
    do you know where i can start from?

    ReplyDelete
    Replies
    1. Gustavo GarciaApril 30, 2017 at 8:01 AM

      It is here, but don't expect it to be easy to reuse in your app, it probably has many dependencies with rest of webrtc codebase: https://chromium.googlesource.com/external/webrtc/+/master/webrtc/modules/audio_coding/neteq/

      Delete
  2. pabloNovember 15, 2017 at 11:05 AM

    Thank you for the post.

    A typical video stream has about 100 packets per second. Packet age of 10,000 is about 100 seconds old and 1,000 nack queue length is equal to at least 10 seconds. Is there a use case where 100 or 10 seconds old packets are still useful? Why don't they use 1 second limit as in the receiver side?

    Why do they keep a list of keyframes and not just the most recent one?

    I think they also check for the first packet in a keyframe and not if we got all the frame parts:
    bool is_keyframe =
    packet.is_first_packet_in_frame && packet.frameType == kVideoFrameKey;

    Seems like the webrtc source has moved:
    https://chromium.googlesource.com/external/webrtc/+/master/modules/video_coding/nack_module.cc

    ReplyDelete
  3. UnknownMay 4, 2018 at 2:37 PM


    Hi, i think that i saw you visited my weblog thus i came to “return the favor”.I'm trying to find things to enhance my site!I suppose its ok to use some of your ideas!! facebook sign in

    ReplyDelete
  4. UnknownAugust 20, 2018 at 1:44 AM

    Thanks very much for this post! I'm currently implementing retransmissons for aiortc, my Python implementation of WebRTC and it answers a number of questions I had.

    ReplyDelete
  5. UnknownOctober 31, 2021 at 10:02 PM

    Hi,we are using OPUS audio codec in our application for compressing audio.But we can't properly handle our packet losses like webRTC. Is there any packet loss handling part in webRTC without OPUS?Any suggestion?

    [We also enabled Forward Error Correction]

    ReplyDelete
  6. INFYCLE TECHNOLOGIESDecember 14, 2021 at 2:21 AM

    Don’t follow your role model. Be the Role model person for others. But it's so simple by getting Hadoop training in Chennai. Because it is an assurance course to bounce back from a double salary. For joining call 7502633633.

    ReplyDelete

Post a Comment

Popular posts from this blog

Bandwidth Estimation in WebRTC (and the new Sender Side BWE)

Image
Bandwidth estimation is probably the most critical component in the video engine of WebRTC. The bandwidth estimation (BWE) module is responsible for deciding how much video* traffic you can send without congesting the network to prevent degradation of the video quality. In the past bandwidth estimation algorithms used to be very rudimentary and based mostly on packet loss. Basically we used to start increasing slowly the video bitrate until we detected packets being lost.  To detect the packet loss you use the standard RTCP feedback mechanisms where the receiver side reports packet loss periodically using RTCP Receiver Report (RR) messages. Modern bandwidth estimation algorithms are more advanced and try to detect congestion before it is bad enough to make routers discard packets. These algorithms predicts congestion analyzing the delay between packets. The idea is that when you start having some congestion, the buffers in the routers will start filling and the delay wil...
Read more

Improving Real Time Communications with Machine Learning

Image
When we talk about the applications of Artificial Intelligence / Machine Learning (AI/ML) for Real Time Communications (RTC) we can group them in two different planes: Service Level : There are many features that can be added to a videoconference service, for example identification of the participants, augmented reality, emotion detection, speech transcription or audio translation .  These features are usually based on image and speech recognition and language processing. Infrastructure Level : There are many ways to apply ML that do not provide new features but improve the quality and/or reliability of the audio/video transmission. Service level applications are fun, but they are more for Product Managers and I like technology more, so in the next sections I will try to describe possible applications of AI/ML for Real Time Communications at Infrastructure Level organizing those ideas in five different categories. Optimizing video quality Some of the ML algorithms use...
Read more

Controlling bandwidth usage in WebRTC (and how googSuspendBelowMinBitrate works)

Image
There are cases when we would like to limit the maximum bitrate being transmitted by WebRTC to avoid wasting resources in the user endpoints or save money reducing the bandwidth usage in our servers.   This is because the maximum bitrate by default in Chrome is around 2Mbps and for many use cases a much lower bitrate provides still pretty good quality.   BTW using a lower bitrate can also help with stability of quality in multiparty scenarios by reducing the amount of competition among different streams. There is no simple API to configure the maximum bitrate in WebRTC (although there is one in ORTC) but there are 3 ways to do this by mangling the SDP. 1. U se the standard  b=AS:BITRATE (Chrome) or b=TIAS:BITRATE  (Firefox) attributes in the SDP for the audio or video channel[1] 2. Use codec specific attributes (this work at least for opus audio codec with maxaveragebitrate property) [2] 3. Use the proprietary x-max-bitrate attribute in the video channel of the...
Read more