If people have heard of bufferbloat at all, it is usually just an abstraction despite having personal experience with it. Bufferbloat can occur in your operating system, your home router, your broadband gear, wireless, and almost anywhere in the Internet. They still think that if experience poor Internet speed means they must need more bandwidth, and take vast speed variation for granted. Sometimes, adding bandwidth can actually hurt rather than help. Most people have no idea what they can do about bufferbloat.
So I’ve been working to put together several demos to help make bufferbloat concrete, and demonstrate at least partial mitigation. The mitigation shown may or may not work in your home router, and you need to be able to set both upload and download bandwidth.
Two of four cases we commonly all suffer from at home are:
Broadband bufferbloat (upstream)
Home router bufferbloat (downstream)
Rather than attempt to show worst case bufferbloat which can easily induce complete failure, I decided to demonstrate these two cases of “typical” bufferbloat as shown by the ICSI data. As the bufferbloat varies widely as the ICSI datashows, your mileage will also vary widely.
There are two versions of the video:
A short bufferbloat video, of slightly over 8 minutes, which includes both demonstrations, but elides most of the explanation. It’s intent is to get people “hooked” so they will want to know more.
The longer version of the video clocks in at 21 minutes, includes both demonstrations, but gives a simplified explanation of bufferbloat’s cause, to encourage people to dig yet further.
Since bufferbloat only affects the bottleneck link(s), and broadband and WiFi bandwidth are often similar and variable, it’s very hard to predict where you will have trouble. If you to understand that the bloat grows just before the slowest link in a path, (including in your operating system!) you may be able to improve the situation. You have to take action where the queues grow. You may be able to artificially move the bottleneck from a link that is bloated to one that is not. The first demo moves the bottleneck from the broadband equipment to the home router, for example.
To reduce bufferbloat in the home (until the operating systems and home routers are fixed), your best bet is to ensure your actual wireless bandwidth is always greater than your broadband bandwidth (e.g., by using 802.11n and possibly multiple access points) and use bandwidth shaping in the router to “hide” the broadband bufferbloat. You’ll still see problems inside your house, but at least, if you also use the mitigation demonstrated in the demo, you can avoid problems accessing external web sites.
The most adventurous of you may come help out on the CeroWrt project, an experimental OpenWrt router where we are working on both mitigating and eventually fixing bufferbloat in home routers. Networking and ability to reflash routers required!
It’s a really good idea to have a short video to highlight these issues. I would say that the slides are a little bit unhelpful though — too much text to read, etc. A few simple animated illustrations would go a long way.
To use it, I’d suggest starting off a sequence with just a few packets coming in (like g=1:12) — so there’s no buffer build up. Have that going for a few seconds while you talk about what buffers are and why we need them etc. Then you’ll want some more packets to come in (like g=121:3) and you can talk about what happens when the buffer starts to fill up. Finally, you can stick a blue packet or two in somewhere (like in my video) and show how long it takes to get out of the buffer.
The packets going over the “wire” should be long and very flat, to represent the fact the packets take time to move over the wire (I did this in my bottleneck illustration), and back to back, so that people “get” that the bottleneck link is staying full. Right now, it implies the link is mostly idle, when in fact, the link is entirely full (just like the queue is….).
You’re right about the packets going over the wire, of course.
I did originally consider that (having a long flat rectangle), or perhaps having many more packets (so they could be touching on the wire) and a much larger buffer, but in the end I decided that this was an approximation that at the end of the day simplified the presentation a lot.
I had a thought about TCP connections. It’s of course easy for a sender to measure the rate they send traffic. Measuring the rate that the traffic is received by the other end is more difficult, but can be estimated from ACKs.
My understanding is that the congestion window is increased until there is packet loss (and by that point, one of the buffers along the path is probably too full). As a result, the sender allows more unacknowledged packets on the wire than it needs to.
Is it possible to detect when an increase in unacknowledged packets has not resulted in any throughput boost and try to estimate the increased buffer usage and then slow down enough to _undo_ that?
For example, let’s say we’re comfortably sending at 10 packets/second, and getting ACKs that indicate the receiver is getting them at approximately 10 packets/second too. So, we increase the rate to 15 packets/second. 5 seconds later, we get ACKs that indicate the receiver is now getting packets at approximately 12 packets/second. We then estimate that 3 packets/second (15 packets) have been going into buffers somewhere. Instead of waiting until we hit the congestion window / receive window and eventually being forced to severely cut back the sending speed, why not pause for 1.25 seconds (exactly enough to clear the estimated additional buffer) and then continue sending at 12 packets/second? We can periodically try to increase the sending speed as usual, but just be more aggressive about buffer correction.
There must be a good reason why this can’t work, or I suppose someone would have already tried it?
That’s something like what Ledbat does, and there is a version of TCP congestion control which is delay sensitive (Vegas).
There are several problems here:
Getting everyone to agree to use it
If you lose relative to “normal” TCP, and get much less bandwidth as a result, there is a disincentive to use it
So I’m pessimistic about attacking bufferbloat via this method.
So you used bandwidth shaping to move your queue to your router (despite my almost certainty about you making comments previously that shaping is not a solution).
Presumably that only works as long as your router is not bufferbloated, yes?
The problem with this solution though, is that you are of course solving the upstream buffer-bloat problem by not saturating the upstream bandwidth and letting moving it’s queue back to your router where you can control the bufferbloat. But upstream bandwidth in broadband is almost always never enough at full bore, to reduce it even further seems like heresy.
I have a connection which is 14Mb/s upstream and only 1Mb/s downstream. My mom has something on the order of 6-7Mb/s downstream and only about 500Kb/s upstream. That 500Kbp/s seems (from experience, not really research) to be barely enough (and at times, a bit too little) to do a 320×240 video voip call. Restricting it further in order to control queues just sacrifices what little she does have and makes a video call pretty much impossible.
I’ll be glad when all of this is behind us, although I’m not holding my breath on it.
Nothing like being a heretic. But you see what effect it does; you can also look at the difference in smokeping results I posted elsewhere in the blog.
The router is bloated too; but the bloated buffers in it are kept empty; the buffers in the queue discipline that does the bandwidth shaping are shallow. So those buffers are also being kept empty.
I never claimed this was a solution: I don’t like giving up bandwidth too: the right mitigation is shorter queues (which the cable industry is already doing), and the real fix is AQM.
But I’ll in general take a network which has predictable performance over one which does not. Even if you don’t share a link, you can do yourself in, or background programs can do you in (I once found google chome in the background uploading a crash dump of itself).
[…] des sommités (voir la discussion avec Vinton Cerf). Un site web spécifique a été mis en place, des vidéos démontrant le problème ont été réalisées. Cette débauche d’énergie est nécessaire parce que le problème est grave, parce que les […]
If people have heard of bufferbloat at all, it is usually just an abstraction despite having personal experience with it. Bufferbloat can occur in your operating system, your home router, your broadband gear, wireless, and almost anywhere in the Internet. They still think that if experience poor Internet speed means they must need more bandwidth, and take vast speed variation for granted. Sometimes, adding bandwidth can actually hurt rather than help. Most people have no idea what they can do about bufferbloat.
jg's Ramblings 
February 1, 2012 at 6:56 pm |
The demonstrations are great!
It’s a really good idea to have a short video to highlight these issues. I would say that the slides are a little bit unhelpful though — too much text to read, etc. A few simple animated illustrations would go a long way.
February 1, 2012 at 7:02 pm |
Care to try to do the animations? I’d love to have some animations…. I’d redo the video to include them….
February 2, 2012 at 11:25 am |
I thought it might be good to have a flexible utility for making animations of a single buffer+link.
So, here it is along with all the code — I hope you find it useful. If you get stuck feel free to send an email.
https://github.com/kieranclancy/bufferbloat
Here’s a sample video:
http://kieranclancy.com/bufferbloat/sample.avi
To use it, I’d suggest starting off a sequence with just a few packets coming in (like g=1:12) — so there’s no buffer build up. Have that going for a few seconds while you talk about what buffers are and why we need them etc. Then you’ll want some more packets to come in (like g=121:3) and you can talk about what happens when the buffer starts to fill up. Finally, you can stick a blue packet or two in somewhere (like in my video) and show how long it takes to get out of the buffer.
Any comments appreciated too.
February 2, 2012 at 11:40 am
Cool!
You may inspire me to some hacking….
Some nits:
The packets going over the “wire” should be long and very flat, to represent the fact the packets take time to move over the wire (I did this in my bottleneck illustration), and back to back, so that people “get” that the bottleneck link is staying full. Right now, it implies the link is mostly idle, when in fact, the link is entirely full (just like the queue is….).
February 2, 2012 at 11:48 am
You’re right about the packets going over the wire, of course.
I did originally consider that (having a long flat rectangle), or perhaps having many more packets (so they could be touching on the wire) and a much larger buffer, but in the end I decided that this was an approximation that at the end of the day simplified the presentation a lot.
February 2, 2012 at 11:57 am
I think I have a good solution actually, I’m just trying it now.
February 2, 2012 at 12:41 pm |
Check my latest source and sample video. It now shows the link saturation more effectively.
February 2, 2012 at 12:48 pm
Much better, though the packets on the wire are still the same length; they should be longer than the ones in the queue…
February 2, 2012 at 12:57 am |
I had a thought about TCP connections. It’s of course easy for a sender to measure the rate they send traffic. Measuring the rate that the traffic is received by the other end is more difficult, but can be estimated from ACKs.
My understanding is that the congestion window is increased until there is packet loss (and by that point, one of the buffers along the path is probably too full). As a result, the sender allows more unacknowledged packets on the wire than it needs to.
Is it possible to detect when an increase in unacknowledged packets has not resulted in any throughput boost and try to estimate the increased buffer usage and then slow down enough to _undo_ that?
For example, let’s say we’re comfortably sending at 10 packets/second, and getting ACKs that indicate the receiver is getting them at approximately 10 packets/second too. So, we increase the rate to 15 packets/second. 5 seconds later, we get ACKs that indicate the receiver is now getting packets at approximately 12 packets/second. We then estimate that 3 packets/second (15 packets) have been going into buffers somewhere. Instead of waiting until we hit the congestion window / receive window and eventually being forced to severely cut back the sending speed, why not pause for 1.25 seconds (exactly enough to clear the estimated additional buffer) and then continue sending at 12 packets/second? We can periodically try to increase the sending speed as usual, but just be more aggressive about buffer correction.
There must be a good reason why this can’t work, or I suppose someone would have already tried it?
February 2, 2012 at 9:52 am |
That’s something like what Ledbat does, and there is a version of TCP congestion control which is delay sensitive (Vegas).
There are several problems here:
Getting everyone to agree to use it
If you lose relative to “normal” TCP, and get much less bandwidth as a result, there is a disincentive to use it
So I’m pessimistic about attacking bufferbloat via this method.
February 2, 2012 at 7:23 pm |
So you used bandwidth shaping to move your queue to your router (despite my almost certainty about you making comments previously that shaping is not a solution).
Presumably that only works as long as your router is not bufferbloated, yes?
The problem with this solution though, is that you are of course solving the upstream buffer-bloat problem by not saturating the upstream bandwidth and letting moving it’s queue back to your router where you can control the bufferbloat. But upstream bandwidth in broadband is almost always never enough at full bore, to reduce it even further seems like heresy.
I have a connection which is 14Mb/s upstream and only 1Mb/s downstream. My mom has something on the order of 6-7Mb/s downstream and only about 500Kb/s upstream. That 500Kbp/s seems (from experience, not really research) to be barely enough (and at times, a bit too little) to do a 320×240 video voip call. Restricting it further in order to control queues just sacrifices what little she does have and makes a video call pretty much impossible.
I’ll be glad when all of this is behind us, although I’m not holding my breath on it.
February 2, 2012 at 8:29 pm |
Nothing like being a heretic. But you see what effect it does; you can also look at the difference in smokeping results I posted elsewhere in the blog.
The router is bloated too; but the bloated buffers in it are kept empty; the buffers in the queue discipline that does the bandwidth shaping are shallow. So those buffers are also being kept empty.
I never claimed this was a solution: I don’t like giving up bandwidth too: the right mitigation is shorter queues (which the cable industry is already doing), and the real fix is AQM.
But I’ll in general take a network which has predictable performance over one which does not. Even if you don’t share a link, you can do yourself in, or background programs can do you in (I once found google chome in the background uploading a crash dump of itself).
March 19, 2012 at 1:52 pm |
[…] des sommités (voir la discussion avec Vinton Cerf). Un site web spécifique a été mis en place, des vidéos démontrant le problème ont été réalisées. Cette débauche d’énergie est nécessaire parce que le problème est grave, parce que les […]