The bufferbloat front has appeared quiet for several months since two publications hit CACM (1), (2) and several videos hit YouTube, though I have one more article to write for IEEE Spectrum (sigh…).
There has been a lot going on behind the lines, however, and some major announcements are imminent on ways to really fix bufferbloat. But I wanted to take a moment to acknowledge other important work in the meanwhile so they do not get lost in the noise, and to get your juices flowing.
- First off, Linux 3.3 shipped with BQL (byte queue limits) done by Tom Herbert of Google. This is good stuff: finally, the transmit rings in Linux network device drivers won’t cause hundreds of packets of buffering.
- Dave Taht has had good success prototyping in CeroWrt a combination of Linux’s SFQ and RED to good effect: SFQ ensures decent sharing among short lived interactive flows which receive preference to long lived elephant flow TCP sessions. As transient bufferbloat and TSO/GSO GRO/LRO smart NIC’s make clear, no comprehensive solutions for achieving good latency are possible without some sort of “fair” queuing and/or classification. As in all RED based AQM algorithms, tuning SFQRED is a bitch and a better AQM is badly needed; news at 11 on that front. CeroWrt is approaching its first release with all sorts of nice features and I’ll blog about it when it’s soup. In the meanwhile, adventurers can find all they want to know about CeroWrt at the links here.
- The DOCSIS changes to mitigate bufferbloat in cable modems continues on its way. While I haven’t checked in to see when deployment really starts (driven by modification to cable carrier deployment systems), we should see this major improvement later this year.
And, as outlined in other writings on this blog, and demonstrated in this video, you can do things about bufferbloat in your home today.
So there is hope. Really… Stay tuned…
jg's Ramblings 
May 1, 2012 at 4:06 pm |
Hi Jim, does BQL provide or require any knobs to tweak or is it fire & forget? Is there anything I should / could do with regard to BQL in latest CeroWRT?
May 1, 2012 at 4:51 pm |
I think everything is set to go in CeroWrt as far as BQL; however, BQL is ethernet only at this time. Dealing with wireless is going to be “interesting”, as in the Chinese curse.
May 1, 2012 at 8:10 pm |
So there are knobs for the supported drivers in
/sys/class/net/the_ethernet_device/queues/tx-0/byte_queue_limits/limit and limit_max. If you have more than one hardware queue you have tx-* to deal with.
You can also see packets in flight by the inflight parameter.
BQL is optimized by default to take ‘maximum advantage of the hardware’, but no more. It generally results in an optimal *bandwidth* setting especially for GigE and above. It’s not quite an optimal *latency* setting, however. (but it comes close!)
Personally I’m willing to reduce latency in exchange for a little bandwidth, a little more than BQL would pick.
At 100Mbit and below I just set it to 2*MTU and call it a day. (this code is in the debloat script, which also sets ethtool right)
BQL automatically picks about 6-7k for cerowrt when operating at gigE speeds, or at least did, when last I checked. (But I have fixed a couple major throughput bugs since). I’ve seen it go as high as 18k on fast GigE hardware without TSO, and 110k on TSO enabled hw. Don’t have any 10GigE boxes to play with.
Til now, I have cared mostly about getting repeatable results, rather than optimal ones.
So it’s only mildly fiddly – off by a factor of two in the cerowrt case, and that’s 1000s (no joke) of times better than life before BQL.