I was just doing my usual google for bufferbloat, to see what’s going on out there….
I came across Tomato USB w/reduced buffer bloat; it left me deeply unhappy.
I’m unhappy as it’s missing a fundamental point to the bufferbloat mess: there is never any single right answer for buffering in a general purpose system. To give a simple example: on a loaded 802.11 network, even a single packet of buffering can be an issue and add significant latency (1 packet = 1500 bytes = 12000 bits = 12ms/packet @ 1MBPS) ; but when operating with that identical hardware at 802.11n speeds, we’ll need potentially many packets of buffering (and AQM to boot, to control background elephant flows that would otherwise fill those buffers). And bandwidth on wireless can change by an order of magnitude (or more) when you move a machine a centimeters. So there isn’t a simple “tuning” point either; it’s dynamic.
The buffer management problem here is really hard; and will be a real challenge to get right. While twisting knobs to mitigate bufferbloat may be helpful at times (particularly now, while buffering is often so grossly wrong), it’s not going to really this problem.
As a final aside to the above link, so long as the device drivers are radically over buffered, all our fancy QOS queueing and traffic shaping facilities is almost impossible to work right (they can’t even be effective if the OS is dropping the packets into the device driver before noticing that the packets may need to be resorted); OS buffer management needs to be rethought end-to end, and interact much better with today’s “smart” underlying hardware. It’s a really, interesting, difficult problem that has been under the radar for many years, while the situation stewed.
All that being said, I’m happy to see people beginning to take action in ways that may help the home mess (which seems to be where bufferbloat is most serious).
jg's Ramblings 
July 6, 2011 at 8:28 pm |
Have you looked into using the change in RTT caused by buffering as an indication of congestion (like TFRC does)? Yes, that means changing the TCP sending algo, but that’s probably not much harder than adding smart queue management to the entire Internet.
Using the change in RTT as an indication also probably allow reducing dropped packets and retransmissions, so it would reduce congestion, etc. Do you know if any research has been done with that kind of algorithm in TCP ?
July 13, 2011 at 9:00 pm |
Production systems have very long lifetimes; many companies still run Windows XP internally. And bufferbloat isn’t fundamentally about TCP; UDP can just as well fill buffers. The fundamental issue is that if you delay feedback by buffering, any congestion causes high latency in the bottleneck, and for that queue, you are stuck.
This doesn’t mean that I think better TCP congestion avoidance algorithms aren’t worth working on, just that you are fundamentally working around a problem caused by excessive buffering, that causes all sorts of havoc.
July 13, 2011 at 9:37 pm |
I agree with your that we really have to have a two-pronged approach to the problem, working on the core (and on reducing buffers), but we should also look if we can put more intelligence in the edge. And I think that having some buffering is inevitable and using that to improve TCP (and UDP, DCCP, etc) congestion control would probably give us results now rather than having to wait a decade for the big network operators to update their equipment.
July 8, 2011 at 7:12 pm |
I understand the rant perfectly, that there is no one way to do buffering. It’s situational and depends on many factors.
But this is just very early work into dealing with bad bufferbloat that exists out in that particular build of Tomato.
The original Tomato used sfq (with defaults of 128 packet limits, and perturb time of 10 seconds).
The most important knobs we had were to limit our overall outbound limit at the time.
Then came more customized versions of the firmware, which included pfifo and TCP vegas (debatable if vegas even works since the router isn’t the endpoint?) The pfifo queue was hard coded with 256 packets for all classes.
txqueuelen in the WRT54GL has always been 1000 packets, AFAIK. In kernel 2.4 or 2.6 builds.
So what I’m offering people is slightly tuned versions of the firmwares, with 2, 10, and 25 packet limits. I figured a range of values for people depending on their speed.
Mostly this is tuning the uplink knob to not queue so much. In my own testing, with default packet limits, and a modest 1Mb/s upload, my router would never drop packets. The limits didn’t mean anything.
I assume that’s the case for most people who don’t have very fast uplinks.
I want to offer versions of the firmware with tunable settings for queues. However, I first just wanted to get test builds out there with hard-coded settings.
SFQ is by designed hard-coded in that series of kernels. AFAIK, it must be done at compile time. That feature came later in newer kernels.
pfifo though is very much tunable at runtime on these routers. There’s just no GUI interface for it.
And I realize that neither of these classes are necessary the best either, as bufferbloat.net has experimental work in this direction.
I’ve also just compiled a version with the WRT54GL’s wireless card DMA buffers to have much lower limits. I’m working to include ethtool, so that perhaps the ring buffers can be set after compile time. The defaults were 512 buffers in most cases.
So, give me some slack. I’ve barely started working on this. It’s hard enough convincing anybody this is needed. Most people are defending buffers that never drop anything and so the idea gets little traction or interest.
July 13, 2011 at 8:56 pm |
I see how you misunderstood my intent; I scrambled to post part 2 to clarify, and then did my weekend thing and have been elsewhere since.
August 25, 2011 at 2:55 pm |
[…] years. Ultimately, all buffering/queuing in operating systems needs to be carefully managed and be automatically adaptive to the data transfer rates. All network routers (and operating systems!) should be running […]