A discussion with Vint Cerf, Van Jacobson, Nick Weaver, and Jim Gettys

This is part of an ACM Queue case study, accompanying Kathie Nichols and my article that will appear in the January 2012 CACM (Communications of the ACM).

But more formal publication has its merits; in particular, having articles for less directly involved in networking and/or more managerially oriented technical managers is very important. So I’ve been working with/in ACM queue to put together a case study. The first piece of this case study was posted last weekend in ACM Queue and will appear as an article in the January issue of CACM (Communications of the ACM) in dead-tree form.  There will also be a full paper posted in ACM queue, but to make the January CACM, we put that aside to finish the (much shorter) article.

Here’s some prior art from June 2001.  In that period, at Compaq/HP’s Cambridge Research Laboratory, we had ported Linux to the iPAQ handheld (with touch screen & expansion capability). Colleagues of mine, including Jamey Hicks, Andy Christian, Frank Bomba, Ben Curis had built an expansion pack for the iPAQ, called the BackPAQ (just like Apple has an I fetish, Compaq had a paq fetish and liked “i”s as well), with accelerometer, camera, and additional expansion capability including additional battery, for our (and other’s) research as part of “Project Mercury”; it was obvious that such devices would become standard in short order, but no device at the time had them integral. Quite a few BackPAQ’s were built and distributed to researchers around the world (small number of hundreds, if I remember correctly). We wrote some papers, distributed a bunch of BackPAQ’s to like minded researchers around the world, and demonstrated the code at the Usenix conference and elsewhere, and published all the code on handhelds.org (which seems down at the moment). The probability of Apple employees having seen this device and it rotating the screen is an absolute certainty; not only did we show the BackPAQ off at numerous conferences, but we built significant numbers used at universities.

It was blindingly obvious to us that hooking up the accelerometer to be able to rotate the screen would be “a good idea”.  Keith Packard and I wrote the xrandr X Window System extension specifically to support screen rotation, for the iPAQ handheld using his TinyX driver (the X extension then became a standard part of the X Window System releases in X.org).  I wrote (in an hour or two) the first version of the xaccel daemon that took the accelerometer data and controlled the screen rotation.  I first packaged it (in ipgk format, for the iPAQ Familiar Linux distribution) on June 11, 2001 to enable the code’s distribution. Ironically, I like what I remember of xaccel’s behaviour better than what I now see on the iPhone and the iPad I own.

Since I can’t go reading Apple’s patent itself, I’ll just note:

• This is a handheld device, with 802.11 wireless (later versions of the iPAQ became phones).
• It has a touch screen
• It has an accelerometer in the BackPaq
• It used the data from the accelerometer with simple heuristics to control the orientation (portrait or landscape) of the screen (in this case, running the X Window System

Update 1:

Update 2

Jaharks of CMU in a comment below notes that the Itsy folks did gesture based screen rotation on the Itsy.  Quite a few Itsy‘s (the spiritual predecessor to the iPAQ, and to my knowledge the first handheld device to run Linux, and the inspiration/cause of our handhelds.org work) were built and distributed to universities, along with the source code.

/*
 * $XFree86: xc/lib/Xrandr/Xrandr.c,v 1.4 2001/06/07 15:33:43 keithp Exp $
 *
 * Copyright  2000 Compaq Computer Corporation, Inc.
 *
 * Permission to use, copy, modify, distribute, and sell this software and its
 * documentation for any purpose is hereby granted without fee, provided that
 * the above copyright notice appear in all copies and that both that
 * copyright notice and this permission notice appear in supporting
 * documentation, and that the name of Compaq not be used in advertising or
 * publicity pertaining to distribution of the software without specific,
 * written prior permission.  Compaq makes no representations about the
 * suitability of this software for any purpose.  It is provided "as is"
 * without express or implied warranty.
 *
 * COMPAQ DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL COMPAQ
 * BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
 * OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
 * CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 *
 * Author:  Jim Gettys, Compaq Computer Corporation, Inc.
 */

#include <stdio.h>
#include <X11/Xlib.h>
#include <X11/Xproto.h>
#include <X11/extensions/Xrandr.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <math.h>
#include <stdlib.h>
#include <sys/poll.h>
#include <errno.h>

#define ACC_DEV "/dev/backpaq/accel"
#define ANGLE (.5 * 256)
#define RMAX (.75 * 256)
#define RMIN (-.75 * 256)

#define POLLRATE 100
#define NSAMPS 4

static int err = 0;

int
main (int argc, char **argv)
{
  Display *dpy;
  XRRScreenSize   *sizes;
  XRRScreenConfiguration *sc;
  int		    nsize;
  Window	    root;
  int		    n = 0, i;
  Status	    status;
  Rotation	    rotation, current_rotation, rotations;
  XRRScreenChangeNotifyEvent event;
  static Rotation old = 0;
  int accfd;
  int nsamps = 0;
  struct {
    short x_accel;
    short y_accel;
  } racc;

  struct accel {
    int x_accel;
    int y_accel;
  };
  struct accel acc[NSAMPS];
  struct accel ave;
  struct accel cal = { 0, 0 };

  int calibrated = 0;
  int product;
  int asq = ANGLE * ANGLE;

  pid_t pid;

  /*
   * this is coded to sleep for 10 seconds if either:
   *  the X server isn't running, or:
   *  the accelerometer isn't present, so it can run under any circumstance.
   */

  pid = fork();
  if (pid == -1) {
    fprintf(stderr, "Fork failed errno = %d\n", errno);
  }
  if (pid > 0) {
    printf("%d\n", pid);
    exit(0);
  }
  for (;;) {

    dpy = XOpenDisplay (NULL);
    if (dpy == NULL)    {
      sleep(10);
    }

    root = DefaultRootWindow (dpy);

    sc = XRRGetScreenInfo (dpy, root);

    accfd = open(ACC_DEV, O_RDONLY);

    if (accfd == -1) {
      sleep(10);
    }
    else {
      while (1) {
	n += 1;
	if (n >= NSAMPS) n = 0;

	poll (NULL, 0, POLLRATE);
	if (read (accfd, &racc, sizeof(racc)) == -1) {
	  goto leave;
	}

	nsamps += 1;
	/* compute running average, yes, ma, there are better ways */
	acc[n].x_accel = (int) racc.x_accel;
	acc[n].y_accel = (int) racc.y_accel;

	ave.x_accel = 0; ave.y_accel = 0;
	for (i = 0; i < NSAMPS; i++) {
	  ave.x_accel += acc[i].x_accel;
	  ave.y_accel += acc[i].y_accel;
	}

	ave.x_accel = ave.x_accel / NSAMPS - cal.x_accel;
	ave.y_accel = ave.y_accel / NSAMPS - cal.y_accel;

	/* lame attempt at self calibration */

	if (nsamps == NSAMPS) {
	  cal = ave;
	  if (argc > 1) printf("cal x = %d; cal y = %d\n", cal.x_accel, cal.y_accel );
	}
	product = ave.x_accel * ave.x_accel +
	  ave.y_accel * ave.y_accel;

	if (argc > 2) printf("ave x = %d, ave y = %d\n", ave.x_accel, ave.y_accel);

	if (product > asq ) {
	  rotation = 0;

	  /*
	   * if(ave.x_accel > RMAX) rotation = 8;
	   * if(ave.x_accel < RMIN) rotation = 2;
	   * if(ave.y_accel > RMAX) rotation = 4;
	   * if(ave.y_accel < RMIN) rotation = 1;
	   */

	  if(ave.x_accel > RMAX) rotation = 1;
	  if(ave.x_accel < RMIN) rotation = 4;
	  if(ave.y_accel > RMAX) rotation = 8;
	  if(ave.y_accel < RMIN) rotation = 2;
	  /* don't try to rotate unless needed */
	  if (rotation && (rotation != old)) {
	    status = XRRSetScreenConfig (dpy, sc, DefaultRootWindow (dpy),
					 (SizeID) 0, 0, rotation, CurrentTime);
	    if (status == RRSetConfigSuccess) old = rotation;
	  }
	}
      }
      leave: XRRFreeScreenInfo(sc);
    }
  }
}

The cable data spec (DOCSIS) has an engineering change in progress to allow the control of buffering in cable modems that was published early this year.  This will allow operators to at least mitigate (reduce) bufferbloat by setting the buffering to something related to the provisioned bandwidth, rather than the current state of buffering, which is either a) whatever size RAM happened to be available in the device, or b) is sized to be the (invalid) BDP “rule of thumb” for the maximum possible bandwidth that hardware might ever be used in (resulting in greatly overbuffered devices when used by most people at the typical bandwidths (e.g. 10Mbps service with a DOCSIS 3 modem capable of 100Mbps). The feature is called “buffer control” for those who want to dig into the specs.

As a concrete example, this might allow the modem I happen to have to have its worse case latency reduced from the 1.2 seconds I observed at 20Mbps to of order 100ms.  This isn’t perfect, but it’s a whole lot better indeed.  To really solve the problem to get latencies under load where they could be, we’ll need real AQM that can work in this environment, which is more of a challenge as noted for reasons elsewhere in this blog. I have no information as to whether any deployed cable modems will ever see new firmware to support this addition to the DOCSIS specification.

I gather than new cable modems that support this change to the DOCSIS spec will be in the market by late this year; but note that deploying the support elsewhere in ISP’s networks to configure them will take more time, and probably won’t happen until next year.

I have no information of if there are similar changes underway to modify the other widespread broadband technologies (e.g. DSL and fiber).

It is a good start to getting things fixed .  Hopefully the market will now help to spread such mitigation steps in the industry.

1. I’m really, really, really, happy to see the work in Tomato USB w/reduced buffer bloat, as I am in the work going on to control buffering in DOCSIS (cable modems). Let me make this clear up front as somewhat of an apology to hechacker1.The enemy of the good is the perfect, and we can and should do what we can quickly to suffer less.
2. Work to fix bufferbloat is going to be both a “do what we can right away” activity, as well as a long term fundamental redesign problem.  I am deeply unhappy as to quite the depth of that redesign problem.

I came across Tomato USB w/reduced buffer bloat; it left me deeply unhappy.

I’m unhappy as it’s missing a fundamental point to the bufferbloat mess: there is never any single right answer for buffering in a general purpose system.  To give a simple example: on a loaded 802.11 network, even a single packet of buffering can be an issue and add significant latency (1 packet = 1500 bytes = 12000 bits = 12ms/packet @ 1MBPS) ; but when operating with that identical hardware at 802.11n speeds, we’ll need potentially many packets of buffering (and AQM to boot, to control background elephant flows that would otherwise fill those buffers). And bandwidth on wireless can change by an order of magnitude (or more) when you move a machine a centimeters.  So there isn’t a simple “tuning” point either; it’s dynamic.

The buffer management problem here is really hard; and will be a real challenge to get right.  While twisting knobs to mitigate bufferbloat may be helpful at times (particularly now, while buffering is often so grossly wrong), it’s not going to really this problem.

As a final aside to the above link, so long as the device drivers are radically over buffered, all our fancy QOS queueing and traffic shaping facilities is almost impossible to work right (they can’t even be effective if the OS is dropping the packets into the device driver before noticing that the packets may need to be resorted); OS buffer management needs to be rethought end-to end, and interact much better with today’s “smart” underlying hardware.  It’s a really, interesting, difficult problem that has been under the radar for many years, while the situation stewed.

All that being said, I’m happy to see people beginning to take action in ways that may help the home mess (which seems to be where bufferbloat is most serious).

I’ll be giving an abbreviated version of this talk at the NANOG 52 meeting (North American Network Operators Group) in Denver at the Sheraton, on Tuesday, June 14, in the Grand Ballroom at 11:30AM. I’ll try to focus that talk a bit more on the consequences to those operating networks, as best I can given limited time.

Note that this is *at best* a short term hack to reduce pain, and the wrong answer in general, and on some hardware will cause your system to go completely catatonic.  Please, please, don’t just blindly go twisting knobs without understanding what you are doing…

There are many potential places where buffers may hide today. These include (at least):

1. the Linux transmit queue (which txqueuelen controls)
2. device drivers themselves may hide one or more packets (e.g. the Libertas driver) internally, which simplified its implementation
3. Most current hardware has very large DMA ring buffers, often allowing for up to 4096 packets/queue in the hardware itself; in the drivers we’ve examined, the default size seems to be in the 200-300 packet range (also true on some Mac and Windows ethernet drivers we’ve played width).
4. Sometimes the hardware itself may also have packet buffers buried in them. Again, from OLPC, the wireless module there has 4 packets of buffering hidden out in the device.
5. (?) encryption buffers.

Old hardware often has very limited buffering in the drivers and hardware; this is part of the history as to how we got to where we are.

Some buffering is necessary for your network stack to work properly.  The only reason I was setting txqueuelen to zero from 1000 was that I had figured out there were and additional 256 packets of buffering in the Intel wireless and ethernet drivers I was using.  Normally, for classification to be able to work, we’d like to have the Linux transmit queue set to some reasonable (small value), so that we can play nice traffic games of various sorts. So in my experiments, I knew there were already 256 buffers available, and my system would continue to function.

Now the question is: how much buffering is “enough”?

And the answer is, unfortunately, not simple. The buffering that should be present depends upon the bandwidth (which may vary by orders of magnitude) and the delay (which is anywhere from 10ms to a couple hundred if you are going around the world).  The rule of thumb has been the bandwidth delay product, where the delay has been presumed to be around 100ms. And it also depends on workload. The rub is that ethernet spans 3 orders of magnitude in bandwidth, and on wireless, it’s even worse, where moving your laptop a few inches can change your performance by orders of magnitude.

What a server system’s buffering needs on 1G or 10G networks is very different than what you will need on an 802.11g network (which at best runs about 20Mbps, and often runs much more slowly).  But right now, the knobs, for historical reasons, were often set to maximize bandwidth performance for such systems without regard to latency under load on computers in most people’s homes.

So whatever we set these knob(s) to, it is guaranteed to be wrong much of the time for many systems.  At best, until we have better tools at hand, we can mitigate our pain a bit by twisting the knobs to something that may make more sense for the environment where you are running most of the time, and some of our default values in our operating systems and device drivers may need tuning in the short term to the bandwidth. So some short term mitigation is possible by being slightly more clever.

The real long term solution, however, is AQM (active queue management) in the most general sense: the buffering at all layers of the system needs proper integration and management (not just router queues), and it needs to be very dynamic in nature: ergo the interest we have in eBDP, SFB, algorithms and we hope RED Light soon.  We need to signal the end points to slow down appropriately. And getting the operating systems to manage both their buffering in concert with the underlying device drivers and hardware is why this is going to be an interesting problem (as in the Chinese curse).

Loaded guns can hurt if you aim them at your foot and pull the trigger. So please do be careful, and think…