Bufferbloat FAQ

Bufferbloat is confusing.  Questions are natural. I can’t tell you how much hair I’ve lost scratching my head about what I was seeing. I didn’t have much hair to begin with, and I have much less now.

This FAQ is organized in approximately increasing order of technical difficulty. The last FAQ’s answer applies to everyone.

  1. What is bufferbloat, anyway? What does it do to me?
    See my definition. It wrecks latency (in other words, everything takes longer and the internet is slower).
  2. A 100 Gigabit network is always faster than a 1 megabit network, isn’t it? More bandwidth is always better! I want a faster network!
    No, such a network can easily be much slower.   Bandwidth is a measure of capacity, not a measure of how fast the network can respond. You pick up the phone to send a message to Shanghai immediately, but dispatching a cargo ship full of blu-ray disks will be amazingly slower than the telephone call, even though the bandwidth of the ship is billions and billions of times larger than the telephone line.   So more bandwidth is better only if it’s latency (speed) meets your needs. More of what you don’t need is useless. Bufferbloat destroys the speed we really need.
  3. Why is latency as important as bandwidth? more bandwidth is faster, isn’t it?
    Stuart Chesire has said this better than I ever will, in “It’s the Latency, Stupid“. The details of the technology have changed since he wrote that, but everything he says is as true today as it was in 1996. More bandwidth does not mean faster. To quote Stuart: “In fact, if you were really in a hurry to get to London quickly, you’d take Concorde, which cruises around 1350 miles per hour. It only seats 100 passengers though, so it’s actually the smallest of the three. Size and speed are not the same thing.”
  4. What’s not commonly realized?
    I believe bufferbloat triggered the network neutrality debate, and bufferbloat, by destroying low latency, certainly has serious consequences in this area. And for technical geeks, that buffers much larger than the actual path latency destroys congestion avoidance in transport protocols, and bufferbloat occurs in operating systems, not just routers.
  5. What sort of operations may induce bufferbloat suffering by myself or others who share the network?
    Examples include uploading videos to YouTube, emailing large mail (such as those with many images attached), backing up large file(s) or file systems over many current broadband or 3g networks, downloading large files, such as movies, ISO’s, your kid downloading a Linux distro (or movie) via bittorrent, and, unfortunately, even visiting certain kinds of web pages. On shared networks, such as 3g or 802.11 networks at conferences or hotels, even general traffic may congest links, inducing the severe problems you notice at certain locations and times. The buffers are full, and cause delay, slowing down the network’s “feel”.  The network is slow.
  6. What kind of applications suffer most?
    Applications, such as VOIP (voice over IP), multi-user games, and teleconferencing suffer most.  Bufferbloat is often causing your network to be ten or even a hundred times slower than it should be when suffering from bufferbloat. Even web browsing can become very painful. And your experience will never be consistent, as you or others around you induce bufferbloat. “Daddy, the Internet is slow today” was a constant refrain in my household.
  7. Can I do anything personally to reduce my suffering from bufferbloat?
    Yes, there may be steps you can take immediately to reduce your suffering. Knowledge is power; it is why I’m doing this blog. Ultimately, by understanding the problem and educating others (including, often, your ISP’s or IT departments), we can make a big difference quickly.  Other solutions will take more time, and ultimately voting with your pocket book will be necessary to fully solve the bufferbloat problem. But see the last question below, about bozos.
  8. Has my (fill in your favorite ISP, hardware vendor, operating system vendor) deliberately done this?
    I don’t think so.   Bufferbloat costs them all too much in service calls and missed business opportunities for me to buy into a paranoid view of the world. The conspiracy theorists will have great fun trying to prove otherwise anyway, of course. See the last question below about bozos.
  9. Is this a TCP specific problem?
    No.  Suffering from bufferbloat can be induced by any transport protocol, including UDP, when links saturate, which they always will in many commonly encountered situations.
  10. Is bufferbloat new?  What happens when you add bloated (semi-infinite) buffers to the network?
    No. John Nagle’s cogent explanation of the phenomena in RFC 970 dates from 1985. Some of the buffers now observed in the Internet are, to first approximation, infinite in size. Other buffers found are merely grossly excessive for the bandwidth of the connection. Besides high latency (slow internet behavior), they can cause partial or complete failure of applications, network services or networks themselves.
  11. How common is bufferbloat? How can I tell if I’m suffering from bufferbloat?
    Dismayingly common as the ICSI Netalyzr results have shown.  Other papers show problems elsewhere in the Internet. And since empty buffers can be hidden anywhere, just because you aren’t suffering from bufferbloat now doesn’t mean you won’t in the (even short term) future. You can test for whether you are currently suffering from bufferbloat by running ICSI’s Netalyzr test or performing other tests as I’ve shown in this blog.
  12. Does bufferbloat only occurs when there is 2 (or more) flows? Or can bufferbloat also occur with only 1 single flow? Demonstrating bufferbloat is easy to do with a single TCP connection (two one line commands), on all operating systems other that Windows XP (which does not implement window scaling by default), so long as that machine can saturate a network path.  You can use any other protocol to show bufferbloat; it just may be more involved.  The Netalyzr test for bufferbloat is UDP based, for example.
  13. What exactly happens when bufferbloat occurs (ie: when an excessively big buffer fills up, right?)?
    Buffers can easily fill from any traffic, from any protocol.  Bufferbloat is when those buffers are not being managed, and therefore are oversize for extended periods, imposing excessive latency to traffic transiting those buffers.When there is bufferbloat, your queues are excessively long.  In a packet switched network, queues should, on average, be very short, not running continually full.  That is what AQM can do for you.
  14. Is bufferbloat referring to (1) the fact that an excessively big buffer fills up, or to (2) the fact that a flow can experience excessive delay when an excessively large buffers fills up?
    Both.
  15. Is bufferbloat referring to excessively large buffers preventing TCP flow control?
    There is still flow control, but again, as there are long delays due to the buffering, the remaining flow control is poor and may become “bursty”.  As you can see in my tcp traces, when the buffers are much larger than the normal RTT should be, you’ve destroyed the fast response of the servo system.  So TCP flow control is also suffering from the imposed latency. This actually induces a certain amount of packet loss (significantly more than you would have had in a good network) overall bandwidth usage is not destroyed on modern TCP implementations as SACK and fast retransmit can keep the pipe reasonably full. In the quest to avoid losing bits, we actually lose more bits. What bufferbloat does cause by no timely packet loss or ECN marking is destroying congestion avoidance in TCP and other protocols.
  16. Does the nefarious effect of bufferbloat only occur under network congestion?
    Yes, exactly.  When the network is congested, if there is no queue management on that buffer, the buffers fill and bad things happen.You only suffer pain from bufferbloat just before a saturated bottleneck link.  Congestion routinely occurs in everyday life, whenever you move bulk data; it is therefore very common in our home networks, both due to the commonness of problems in the broadband gear as ICSI’s Netalyzr work showed, and also in our home routers and computers (typically on 802.11).  Similarly we see aggregate forms of bufferbloat both on busy 802.11 networks and 3G networks. And some network operators (both corporate and ISP’s) may fail to run with AQM, and therefore inflict pain on their customers.
  17. What is this AQM thing you keep talking about?
    AQM == Active queue management. The Wikipedia article will get you going. I use AQM in this blog in the most general sense: active management of the queues. There are many techniques to control your queue lengths, but many of them may not be available to you or applicable to your circumstances.
  18. Will TCP by itself fill buffers? Even without bufferbloat?
    Yes; this is why queue management was developed in the first place, starting at least 20 years ago.
  19. What is ECN?
    ECN == Explicit Congestion Notification. Traditionally, the only way to signal congestion is by packet drop. But that does not distinguish from other random loss; packet drop is the only guaranteed way signal of congestion in  a packet network.
    ECN is a second way to signal congestion, by marking a packet with a bit if it transits a congested node in the network. It’s advantage is the work already expended in moving the packet to that point is preserved. ECN hasn’t been used much, as a bunch of broken hardware was shipped years ago that would crash if it saw an ECN bit. Most of that hardware (mostly in home environments) should have been retired by now. So the question is whether we could/should start using ECN, and if so, in what parts of the network.
  20. But I need to manage my queues if I am classifying my traffic?
    Traffic classification is orthogonal to AQM. If you don’t signal congestion at a congested hop in the network (by either packet drop or ECN), there is no way the end points will slow down and keep the queue size sane and transport protocols will fill the buffers at the bottleneck. For whatever class of service that traffic is classified in, the latencies will go to hell.Traffic classification is a useful tool, and may help performance of classes of applications and operation during periods of congestion, but fundamentally not a solution in any sense for bufferbloat. For that, you need to enable some way to manage the length of the queues (e.g. RED). All buffers need to be managed in some fashion. If you manage your queues, classification is much less necessary.
  21. Huh?  Haven’t we known about AQM for a long time? Isn’t it deployed everywhere?
    That was my reaction. As to the story behind why AQM isn’t deployed everywhere, see my blog posting on RED in a Different Light. Be paranoid: “dark buffers” may be laying to trap you.That we need to think about managing buffers in our hosts better, I think has not been a widespread realization. I only got there by the round about investigation as to why I was seeing occasional 802.11 disasters.
  22. Can we use the increasing RTT to work around bufferbloat?
    The other guy’s traffic will kill you. (your kid’s, your wife’s, your co-worker). So you’ve just fixed your contribution to the problem, so you get to suffer more… Game theory says this isn’t a stable solution, and unlikely to be adopted. The technique itself is being explored in the IETF LEDBAT working group to allow for friendlier behavior for bulk transfer protocols like BitTorrent and will be very useful. But as I noted in one of my posts, much of the pain of BitTorrent only occurred because of bufferbloat in the first place. Fundamentally, the queues need to be managed. So we need to fix bufferbloat overall, not just try to work around it.
  23. Am I a bozo to have not understood bufferbloat before?
    No more than the rest of us. I think we are all bozos on this bus. And the bozo bus is extremely large. Bufferbloat is something we’ve all caused, by not realizing that latency is (at least as) important as bandwidth. Engineers (myself included) have made the same mistake many times over the years, and failed to internalize that packet networks are different than wires. Moore’s law has made memory really cheap.  So you are a bozo only if you don’t start fixing the problem, are unwilling to believe there is a problem in the first place, or call others bozos; we’ve all caused this problem together.  We’ve turned the Internet of sports cars into the Internet of the Queen Mary.  Just think of driving the Queen Mary on a crowded highway at rush hour someday (with your driveway directly attached), and you’ll get my drift.
  24. Where can I get more information and/or help solve bufferbloat for me or others?
    Head over to bufferbloat.net where you will find mailing lists, wiki’s, trackers, and other tools to help.

31 Responses to “Bufferbloat FAQ”

  1. Chris E Says:

    Have you seen the following script for Linux? http://lartc.org/wondershaper/

    Whilst not perfect the comments in both the script and the README seem to indicate that it is trying to address buffers in both the DSL modem at on the ISP side of things.

    • gettys Says:

      Yes, and many modern home routers implement at least some of this capability and I certainly have played with some of these features from time to time this fall. It’s clear that document was written in response to early bufferbloat in early DSL systems. Dave Clark told me he first became aware of the problem over six years ago, and that his DSLAM (he runs a micro-isp in Concord, MA to keep his hand in) had 6 seconds of buffering (and then used it to DOS his son’s excessive WOW playing).

      One of the pitfalls of trying it yourself is that, as I understand it, the classification, etc, goes on in the transmit queues in Linux (which are oversize these days; but twisting that knob is easy). The underlying device driver may itself be bloated on modern hardware, and in effect defeat some of what you are trying to achieve. The ring buffer sizes in the device drivers are now a big problem by themselves (typically 256 entries).

      And there are additional places buffers can hide: the device driver itself, and the network chips themselves. You may have little control once the packets are handed off to the lower layers (of which there are sometimes several).

  2. Tony Finch Says:

    How does ECN relate to bufferbloat? Do (or should) routers use ECN marks when their buffers are backlogging?

    • gettys Says:

      Traditionally, the only way to signal congestion is by packet drop. But that does not distinguish from other random loss; packet drop of course is the only guaranteed way to signal congestion.

      ECN is a second way to signal congestion, by marking a packet with a bit if it transits a congested node in the network. It has an advantage as you don’t lose the work already expended in moving the packet to tat point. It hasn’t been used much, as a bunch of broken hardware was shipped years ago that would crash if it saw an ECN bit. Most of that hardware (mostly in home environments) should have been retired by now. So the question is whether we could/should start using ECN, and if so, in what parts of the network. Steve Bauer at MIT has been investigating this, but the results aren’t all in yet.

    • John Burdick Says:

      ECN is impossible unless systems are able to react to congestion. You can’t mark a packet with ECN and also drop the packet. This means that you need AQM before you can have ECN. An effective AQM mechanism is sufficient to counter buffer bloat. Once you enable AQM, you consider ECN as an extra perq.

      Some AQM concepts rely on configuring buffer parameters. If everyone understood buffer sizing well enough to set those parameters, they wouldn’t have misconfigured their buffers so badly in the first place. So AQM that does not require explicit buffer tuning seems more compelling to me.

      • Bengie Says:

        ECN is just a better version of AQM.

        “Why should I drop perfectly good packets when I still have free buffer space?”

        • gettys Says:

          It depends; you may be losing so badly that you really need to drop packets out of the queue to get the queue length down. ECN may have been useful before you are in such dire straights, and have prevented such a situation; but there is never any guarantee…

          Also, ECN is useful for TCP; doesn’t help on UDP.

  3. James M. Leddy Says:

    You mention the hardware ring buffers (tunable by ethtool) as a source of bufferbloat on Linux. I was always under the impression that the ring buffers did head drop, not tail drop. Is this not the case? Would it matter either way?

    • gettys Says:

      It certainly affects the timeliness of notification. But drops still don’t happen until a large latency has been induced.

  4. nona Says:

    A few animations would be nice to illustrate the problem. I’d make them myself if I understood the problem better.

    • gettys Says:

      Yes, it occurred to me early on too; I’ve spent some time trying to find an existing animation, and didn’t find anything suitable, though one was somewhat close, it didn’t quite convey the problem.

      If there are any volunteers who’d like to work on one, please do let me know.

  5. pet Says:

    Am I wrong in thinking that home routers/network cards have big buffers because they need to be able to saturate their supported bandwidth? I mean, you could have a 10mbit adsl line, but be hooked up to a 100mbit lan. You don’t want your shorter buffer to prevent saturation of the lan bandwidth.
    How would you solve this problem? Of course in the “home” router you could have different queues for the forward ports, and a shorter queue for packets that are being forwarded on to the modem. What about the client?

    Thanks for your posts, I’ve spent half a day reading them all (and experimenting).

    • gettys Says:

      Yes, and no; the nominal static buffer sizing rule of thumb is the bandwidth/delay product times the square root of the expected number of flows, historically. Delay, for continental US, should be of order 100MS (tops, though in fact, there are reasons to believe we have situation where people are presuming measured RTT’s, where real delays are what should go into the computation. People’s broadband connections however, vary all over the map in actual bandwidth, and these buffers are getting sized for (at least) the maximum possible. Often they are being oversized (sometimes grossly) when the vendors find they can paper over bugs in their hardware/firmware.

      Then you take these devices, and use them at a fraction of their theoretical bandwidth, without adjusting the buffer sizes, over unknown paths, and unknown presumptions of the number of flows. Pain ensues.

      Certainly the CMTS/cable modem protocol does not currently have any way to even tell the cable modem to use a more appropriate size buffer. I don’t know about the other technologies.

      As far as optimal buffering, you only need one packet buffer to keep a link busy ever: the issue is not as you think there. You do need buffering to deal with the bursty nature of traffic.

      The bottom line: we’ve got dumb devices, with way more buffering than we need, with many unknown variables that go into the “right” buffer size computation, such as number of flows, provisioned bandwidth, and delay. So our static buffers are almost always wrong and too big (and sometimes grossly wrong to cover over other problems in the devices).

      There is no static right answer: we must manage all queues, is I believe the only real solution.

  6. Bruce Hamilton Says:

    Thanks for the FAQ!

    Re: #19: You don’t define RED. I suggest linking to http://en.wikipedia.org/wiki/Random_early_detection

    Re: #18: It would be helpful to link to some sort of list of ECN-bad hardware. Googling around, the “ECN hall of shame” no longer seems to be available.

    • gettys Says:

      Thanks; fixed, as best I can.

      We may want to restart the hall of shame, or at least see if the old database can be resurrected and hosted somewhere.

  7. Josh in California Says:

    Some folks will probably find this useful. It’s a free smokeping service, with three servers (CA, KS, NY): http://www.dslreports.com/smokeping

  8. joostbaas Says:

    I often notice scp, ftp and http transfers report unrealistically high transfer speeds in the first couple seconds, up to a couple times the theoretical bandwidth limit. Is it possible these obviously incorrect numbers could be caused at least partially by buffers filling up, without anything actually transferred yet, and therefor related to bufferbloat?

    • gettys Says:

      More likely other causes: some ISP’s give you a temporary bandwidth boost (e.g. Turboboost on Comcast), and socket buffers in your system.

    • Alex Yuriev Says:

      It amazes me that we continue to listen to people who do not understand why scp/ftp/http transfers return unrealistically high speeds during first few seconds about things that are far more complex.

      joosbaas, you are absolutely correct. When an application writes to a network, it in fact asks OS to transfer the data on behalf of the application. OS puts the data to the socket buffer and immediately returns control to the application. if the buffer is full and the application is using a blocking call, the OS won’t return control to the application until the data application wants to send out is inside the buffer.

      Dont believe me? Make you buffer size 1 KB or 1 page.

  9. Maninder Says:

    Instead of buffer size, wouldn’t the application processing rate on both ends (send and receive) cause what you describe buffer bloat? Irrespective of buffer size, if receiving application can process the packets at the rate they arrive in the queue, there should be no issue. But if the receiving application cannot keep up with the input queue, then sender will start experiencing packet drops and delays.

    • gettys Says:

      You can certainly have bufferbloat inside applications, whether or not you are using TCP or UDP.

      It is an end-to-end phenomena.

  10. Martouf Says:

    “Windows XP, which does not implement TCP window scaling”

    Apologies, but I believe the statement is not technically correct. XP by default does not -enable- window scaling and timestamps but in truth it does -implement- the functions.

    Enabling the functions is a relatively simple matter of registry editing and reload of the TCP stack (which for XP means a system restart).

    see

    http://www.psc.edu/networking/projects/tcptune/OStune/winxp/winxp_stepbystep.html

    and

    http://msdn.microsoft.com/en-us/library/ms819736.aspx

    and

    http://en.wikipedia.org/wiki/TCP_window_scale_option

    • gettys Says:

      Yes, you are correct, though what I said is morally correct: you don’t see window scaling by default on WinXP, and encouraging people to mess with regedit is usually the wrong thing ;-).

  11. Jason Bucata Says:

    Apologies if this isn’t the best place to post this; I couldn’t find an email address on the site.

    I just saw this link for a Linux tool called “wondershaper” dating back to 2002:

    http://lartc.org/wondershaper/

    Note the paragraph at the very bottom…

  12. Jon Daley Says:

    I wonder if you could write up a really simple how-to in order to tell if one is affected by this problem? As far as I can tell with running some tests, I am not affected by it, but based on your various articles, it sounds like that is naively optimistic?

    I do see my ping latencies go up during heavy network traffic, but I can’t figure out if that is really a problem? I don’t really understand the problem, I think. (though in the process of investigating this, I did discover that I had mistakenly reversed my QOS rules for one of my voip phones vs. my computer… I wondered why the one phone was always worse than the other…)

    • gettys Says:

      If your latency go up a lot as shown via ping during heavy network traffic, then you have problems.

      Running Netalyzr will give you a good idea, so long as your bandwidth(s) are below around 20Mbps (Netalyzr can’t go particularly fast right now).

  13. Sortie du noyau Linux 3.3 | PSOTUX Says:

    [...] par une masse impressionnante de posts sur son blog (voir, entre autres, 1 – 2 – 3 – 4), d’articles officiels (celui d’ACMQueue et aussi celui d’IEEE Internet [...]

  14. anonymous Says:

    Great blog and writeups on the latency issues aka bufferbloat. They have been a great read. Keep up the good work.

  15. anonymous Says:

    Forgot to mention. I took your advice and ran some tests. I use pfSense[1] as my router/firewall and turned on traffic shaping. Went from ~500-1000ms to ~34-40ms of latency as recorded by netalyzr[2].

    This has improved my browsing and VOIP. Don’t know why I didn’t do this sooner ;)

    Thank You

  16. anonymous Says:

    oops,

    Here are the links:
    [1] http://www.pfsense.org/
    [2] http://netalyzr.icsi.berkeley.edu/index.html

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 674 other followers

%d bloggers like this: