Bufferbloat FAQ page is up…

Ah, being Slashdotted is sooo much fun. In this case, it’s not the web site that was overwhelmed (since I put my blog on wordpress.com a while back), but me.  And, of course, Slashdot misspelled my name.  I’m  quite happy I didn’t try to have this on my home system, though as I think about it, it would have been a very interesting test of the broadband mitigation I did a while back (but the machine would probably have died; it was > 60,000 page views in 12 hours, and I’ve never tuned my home server). Slashdot, next time don’t Slashdot me on a Friday, so I might have a life on the weekend.

In any case, I put together a FAQ (Frequently Asked Questions) page put together in self defense.  It’s on the right sidebar of the home page. Unfortunately, this delayed several other pages I’d like to get done, like a good overview, but it does provide some better entry points to the bufferbloat postings.

“A committee is a life form with six or more legs and no brain.” – Lazarus Long

When I started on the bufferbloat quest, I was bugged by why RED (or other AQM’s) were not universally used, and talked to Van Jacobson for some enlightenment; Van explained the issues with RED, and that the unpublished “RED in a Different Light” paper explained RED’s problems along with a potential solution.

Today, while being a friendly nag to Van to try to get the finished version of the paper, he told me the following story, that is (part of) why that paper was never published.

I just about fell out of my chair laughing…

One of the reviewer’s comments was that the issues with RED and solution could not possibly be true, and that the authors should go become familiar with the fundamental literature on RED and automatic queue management.

I wonder if the name of the reviewer will ever be known? But that would be cruel.

I hadn’t thought about this kind of danger of blinded reviews at refereed conferences. If ever you are on a program committee, please actually engage your brain.

P.S. Van’s finishing up converting the paper to TeX and fixing up the text as a bug discovered in the nRED algorithm had not been fully reflected in the text.  It should be available soon. He’s also sending me a pointer to some other recent work that’s been published that may be useful for 802.11 wireless bufferbloat.

Bufferbloat in 802.11 and 3G Networks

Any network system with buffering shared among many users is much like a congested highway.  We’ll call them
“big fat networks”. Two such network technologies which show this problem are 802.11 (abgn), and 3g wireless.  In one, the buffers are distributed among the clients (and may also be in the access points and routers); in the other, both possibly in the clients, and the radio controllers they talk to, but also possibly in the backhaul networks.

You have suffered unusable networks at conferences.  Wonder why no more. You can make your life less painful by mitigating your operating system’s and access point’s buffering.

Moral of the Story

Whether you call what we see on 802.11 and 3g networks “congestion collapse” as the 1980’s NSFnet event was called (with high packet loss rates), or something different such as bufferbloat (exhibiting much lower, but still significant packet losses), the effect is the same: horrifyingly bad latency and the resulting application failures. Personally, I’m just as happy with “congestion collapse” as with bufferbloat.

The moral of the story is clear: when the network is running slowly, we really need to absolutely minimize the amount of buffering to achieve anything like decent latencies on shared media. Yet when the network is unloaded, we want to fill this network pipe that may be hundred megabits or more in size. On such a shared, variable performance network: there is no single right answer for buffering. You cannot just “set it, and forget it”. Read on…

RED in a Different Light

Update May 8, 2012: “Controlling Queue Delay” describes a new AQM algorithm by Kathie Nichols and Van Jacobson.

Congestion and queue management in Internet routers has been a topic since the early years of the Internet and its first congestion collapse. “Congestion collapse” does not have the same visceral feel to most using the Internet today, as it does for a few of us older people. Large parts of the early Internet actually stopped functioning almost entirely, and a set of algorithms were added to TCP/IP to ensure collapse not happen in the future.  These include slow start,  congestion avoidance, fast recovery, and at a later date, ECN (Explicit Congestion Notification), which has not so far seen wide use,  and is a subject of ongoing research to determine if it can be deployed.

Bufferbloat much larger than the RTT’s of the paths destroys the fundamental congestion avoidance of the TCP protocol’s servo system as I documented. We have destroyed congestion avoidance, and as I’ll discuss soon when I return to the topic of web browsers and servers, we are playing with fire. Even if nothing as bad as I fear ever happens, the bufferbloat situation today is bad, with multiple second latencies being common.

Bufferbloat was first understood encountered in very early experiments with satellite networking, and active queue management is a very active area of research since the 1980’s and continuing to this day.  With the invention and wide deployment of algorithms such as RED and others, I had thought that the problem was solved. To my surprise (I am not in the field, but due to history have been a somewhat interested bystander), I was wrong, and that queue management is often not enabled even on significant routers in both enterprise networks and the Internet. The reasons why and the limitations of existing AQM algorithms shed light on this aspect of today’s problems.

Conclusions: Active Queue Management is often not enabled or tuned in today’s Internet and corporate networks. Broadband (and some network’s) performance is therefore often significantly worse than necessary since your ISP may never have enabled AQM. If you are operating a network, check that you have correctly enabled and tuned your AQM on all types of your gear. You can have happier customers and fewer service calls. Finally, we need better queue management algorithms than “classic RED” or closely related algorithms  for today’s wireless routers and operating systems.

Read on for more detail…

Mitigations and Solutions of Bufferbloat in Home Routers and Operating Systems

As discussed several days ago we can mitigate (but not solve) broadband bufferbloat to a decent, if not ideal, degree by using bandwidth shaping facilities found in many recent home routers. Unfortunately, life is more complicated and home routers themselves are often typically at fault (if you find a recently designed home router that works right, it may want to be enshrined in a museum where its DNA and evolution analyzed, and its implementors both admired for their accomplishment and despised, for not telling us about what they discovered. Complete robust solutions, unfortunately, will be difficult in the short term (wireless makes it an “interesting” problem) for reasons I’ll get to in this and future posts.

Confounding the situation further, your computer’s/ smartphone/ netbook’s/ tablet’s operating system may also be suffering from bufferbloat, and the its severity may/almost certainly does depend upon the hardware. Your mileage will vary.

You may or may not have enough access to the devices to even manipulate the bufferbloat parameters. Locked down systems come back to bite you. But again, you can probably make the situation much better for you personally, if you at a minimum understand what is causing your pain, and are willing to experiment.

Conclusions

Since any number you pick for buffering is guaranteed to be wrong for many use cases we care about, the general solution will await operating systems implementers revisiting buffering strategies to deal with the realities of the huge dynamic range of today’s networks, but we can mitigate the problem (almost) immediately by tuning without waiting for nirvana to arrive.

Bufferbloat and congestion collapse – Back to the Future?

I hit “publish” accidentally , when I meant to press “save draft” for publication in several weeks..  There is still a bit of the supporting evidence hasn’t been blogged or fully researched yet.  Until I remove this warning, this is a draft.  Sigh. But the conclusions won’t change…

For the last decade at least, we have been frogs in slowly heating water, and have not jumped out, despite at least a few pokes from different directions that all was not well in our Internet pond. Lots of people have noticed individual problems caused by bufferbloat and how to mitigate them. To some extent, we’ve been engineering around this problem without full understanding, by throwing bandwidth at the problem and as gamer routers show. But RAM costs have dropped even faster than network speeds have risen, and rising network speeds encourage yet larger (currently unmanaged) buffers; throwing bandwidth at the problem has been a losing race.

I’ve been losing sleep the last few months. Partially, I’m at an age where for the first time I’ve lost a number of older friends in a short period. And partially, it is very serious worry about the future of the Internet, and that previous attempts to warn of the bufferbloat problem have failed. And partially, I just don’t sleep very well. I’m not quite to the point Bob Metcalfe was in 1995 when he predicted web growth would cause the Internet’s collapse (we came close). But I’m close to such a prediction. That’s seriously bad news.

And no, I’m not going to offer to eat my words, the way Bob did. He looked much too green after consuming his column when I saw him backstage afterwards. I’ve had enough stomach problems  recently and weight problems that is unwise. If I do ever make that prediction, I might eat a small piece of cake if I’m wrong. But so far, it’s still just worry and not prediction. But my worries have grown as I discover more.

Mitigations versus Solutions of Bufferbloat in Broadband

I have distinguished in my writing between what I call “mitigations” and “solutions”.

• mitigations are actions we can take, often immediately, which make the situation better, and improve (possibly greatly) the current grim situation.  Since they may only work some of the time, and may require conscious thought tuning and action by network operators and users, or have other limitations that are often far from optimal, they won’t work in some circumstances or necessarily be implemented everywhere. Often these mitigations will come at some cost, as in the case today’s posting below.
• solutions are full solutions for a problem that get behavior to something approximating optimal.  Sometimes they may be mitigations that can be widely applied in an ISP, even though though they may require thought there. The “just work” for everyone.

But observed facts (e.g. RED or other AQM is far from universally used; more about this in a future post) shows that anything that does not “just work” is often distrusted and under-used (and seldom enabled by default), so such a solution is seldom the optimal solution we should be looking for: really “solving” the problem once and for all.  As good engineers and scientists, we should always be striving for “just works” quality solutions, which we don’t have for bufferbloat in all its forms.

The full “solution” for the entire Internet is going to be hard; we need to solve too many different problems (as you will see) at too many points in all paths your data may traverse, to wave a wand eliminate bufferbloat overnight.  Some of the point solutions will actually require replacement of hardware, and time to research and engineer such hardware along with economics will often take time.  Does that mean we should do nothing?  Of course not: we can immediately make the situation much better than it is, particularly for consumer home Internet service. And remember, your competitor will eventually beat you if you sit on your hands.

Gamers and others have been mitigating bufferbloat in broadband for years. Read on. You’ll suffer much less. Mitigation of home router bufferbloat itself will be tomorrow’s installment.

Bufferbloat and network neutrality – back to the past…

There are at least three topics in which I think bufferbloat intersects the currently hot topic of network neutrality. This is entirely personal opinion, and not that of my employer.

• by happenstance, broadband ISP’s are enjoying a serious competitive advantage with any other provider of telephony  and gaming services. I believe it unlikely this advantage put in place with malice aforethought, though I expect conspiracy theorists will enjoy trying to prove otherwise.  I think we’d have heard of this by now or there are a very few, very smart people out there who have managed to figure bufferbloat out and keep their mouths firmly shut. But if they were that smart, why did they not foresee the pain of the next bullet?
• the impact of bufferbloat on ISP’s needs to be well understood, to understand their motivations. I now believe bittorrent hit ISP’s and their customers much harder everyone understands; but that ISP’s diagnosis of root cause was flawed.
• to preserve future innovation for new applications and services.

We should not set public policy going forward without understanding what may actually have happened, rather than a possibly flawed understanding of technical problems.

Unfortunately, everyone has taken now very public positions based on a probably a flawed analysis of  the very real, painful problems they have experienced. Getting everyone to stop and revisit their presumptions, and rethink and possibly change their positions is going to be hard. Please help!

Sherman, please set the Wayback machine to when bittorrent first deployed (2004).

Whose house is of glasse, must not throw stones at another.

• George Herbert, Outlandish Proverbs, 1640

In my last post I outlined the general bufferbloat problem. This post attempts to explain what is going on, and how I started on this investigation, which resulted in (re)discovering that the Internet’s broadband connections are fundamentally broken (others have been there before me).  It is very likely that your broadband connection is badly broken as well; as is your home router; and even your home computer. And there are things you can do immediately to mitigate the brokenness in part, which will cause applications such as VOIP, Skype and gaming to work much, much better, that I’ll cover in more depth very soon. Coming also soon, how this affects the world wide dialog around “network neutrality.”

Bufferbloat is present in all of the broadband technologies, cable, DSL and FIOS alike. And bufferbloat is present in other parts in the Internet as well.

The criminal mastermind: bufferbloat!

Each of these initial experiments were been designed to clearly demonstrate a now very common problem: excessive buffering in a network path. I call this “bufferbloat“. We all suffer from it end-to-end, and not just in our applications, operating systems and home network, as you will see.

Large network buffers can be thought of as “dark buffers”, analogous to “dark matter” in the universe; they are undetectable under many/most circumstances, and you can detect them only by indirect means.  Buffers do not cause problems when they are empty.  But when they fill they introduce additional latency (and create other problems, possibly very severe) to other traffic sharing the link.

In the past, memory was expensive, and bandwidth on a link fixed; in most parts of the path your bytes take through the network, necessary buffering was easy to predict and there were strong cost incentives to minimize extra buffering. Times have changed, memory  is really cheap, but our engineering intuition is to avoid dropping data.  This intuition turns out to be wrong, and has become counter-productive.