Apple Patents Portrait-Landscape Flipping: the patent system is broken…

I noticed with interest Slashdot’s article last week on Apple Patenting Portrait-Landscape flipping based on control of one or more accelerometers in Slashdot last week.  As I work at Bell Labs these days, I don’t read patents, so I’ll just go on the summary I read there.

Here’s some prior art from June 2001.  In that period, at Compaq/HP’s Cambridge Research Laboratory, we had ported Linux to the iPAQ handheld (with touch screen & expansion capability). Colleagues of mine, including Jamey Hicks, Andy Christian, Frank Bomba, Ben Curis had built an expansion pack for the iPAQ, called the BackPAQ (just like Apple has an I fetish, Compaq had a paq fetish and liked “i”s as well), with accelerometer, camera, and additional expansion capability including additional battery, for our (and other’s) research as part of “Project Mercury”; it was obvious that such devices would become standard in short order, but no device at the time had them integral. Quite a few BackPAQ’s were built and distributed to researchers around the world (small number of hundreds, if I remember correctly). We wrote some papers, distributed a bunch of BackPAQ’s to like minded researchers around the world, and demonstrated the code at the Usenix conference and elsewhere, and published all the code on handhelds.org (which seems down at the moment). The probability of Apple employees having seen this device and it rotating the screen is an absolute certainty; not only did we show the BackPAQ off at numerous conferences, but we built significant numbers used at universities.

It was blindingly obvious to us that hooking up the accelerometer to be able to rotate the screen would be “a good idea”.  Keith Packard and I wrote the xrandr X Window System extension specifically to support screen rotation, for the iPAQ handheld using his TinyX driver (the X extension then became a standard part of the X Window System releases in X.org).  I wrote (in an hour or two) the first version of the xaccel daemon that took the accelerometer data and controlled the screen rotation.  I first packaged it (in ipgk format, for the iPAQ Familiar Linux distribution) on June 11, 2001 to enable the code’s distribution. Ironically, I like what I remember of xaccel’s behaviour better than what I now see on the iPhone and the iPad I own.

SProject Mercury BackPAQince I can’t go reading Apple’s patent itself, I’ll just note:

  • This is a handheld device, with 802.11 wireless (later versions of the iPAQ became phones).
  • It has a touch screen
  • It has an accelerometer in the BackPaq
  • It used the data from the accelerometer with simple heuristics to control the orientation (portrait or landscape) of the screen (in this case, running the X Window System
Now, maybe you’d like to quibble and claim the idea of putting an accelerometer in a hand-held device is non-obvious.  I think it was pretty obvious, myself, and doing that goes to the group working on Project Mercury. I don’t remember any patent being filed there. And having done so, it seemed obvious to hook it up to the screen. I know we did not file any patents. Are either of these ideas worth a patent? Personally, I think both ideas are pretty obvious, the first idea more original than the second.
But I’m sure the first handheld device with touch screen, with accelerometer, rotating the screen under control of that accelerometer was in my hand running my code below, sometime in the year 2000 or 2001 (I haven’t tried to excavate the exact date),  and that it was widely published on the Internet and used by hundreds of people.
Since handhelds.org seems down at the moment, I spent 5 minutes digging around for the code itself elsewhere.  It’s short enough I include it below (looks like the copyright notice got cut and pasted from the xrandr code); it was called xaccel.c, strangely.

Update 1:

Comments make it clear I fired before aiming carefully: the patent at question apparently is on multitouch gestural overrides to accelerometer screen flipping, apparently. If so, my apologies to Apple.

We have three problems here:

  1. prior art, which may not apply to my example certainly we did not have a multi-touch screen to play with and did not explore that area.
  2. Obviousness may be in the eye of the beholder, but certainly I’ve seen ideas which were non-obvious. The current broken patent system is encouraging filing of patents just for protection of every trivial idea, and to use as weapons against competitors, whether there is merit in them or not.
  3. the treble damages problem, which is why I did not go read the patent in the first place, and stifles actual innovation (independent of whether you thing software patents are a good or bad idea, being unable to know what is going on elsewhere defeats part of the original bargain of why patents were granted in the first place.
And I still like my algorithm better than what I experience on the iPad, which often flips the screen when I don’t want it to flip and begs out for overriding.

Update 2

Jaharks of CMU in a comment below notes that the Itsy folks did gesture based screen rotation on the Itsy.  Quite a few Itsy‘s (the spiritual predecessor to the iPAQ, and to my knowledge the first handheld device to run Linux, and the inspiration/cause of our handhelds.org work) were built and distributed to universities, along with the source code.

/*
 * $XFree86: xc/lib/Xrandr/Xrandr.c,v 1.4 2001/06/07 15:33:43 keithp Exp $
 *
 * Copyright  2000 Compaq Computer Corporation, Inc.
 *
 * Permission to use, copy, modify, distribute, and sell this software and its
 * documentation for any purpose is hereby granted without fee, provided that
 * the above copyright notice appear in all copies and that both that
 * copyright notice and this permission notice appear in supporting
 * documentation, and that the name of Compaq not be used in advertising or
 * publicity pertaining to distribution of the software without specific,
 * written prior permission.  Compaq makes no representations about the
 * suitability of this software for any purpose.  It is provided "as is"
 * without express or implied warranty.
 *
 * COMPAQ DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL
 * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL COMPAQ
 * BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES
 * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION
 * OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN
 * CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE.
 *
 * Author:  Jim Gettys, Compaq Computer Corporation, Inc.
 */

#include <stdio.h>
#include <X11/Xlib.h>
#include <X11/Xproto.h>
#include <X11/extensions/Xrandr.h>
#include <sys/types.h>
#include <sys/stat.h>
#include <fcntl.h>
#include <unistd.h>
#include <math.h>
#include <stdlib.h>
#include <sys/poll.h>
#include <errno.h>

#define ACC_DEV "/dev/backpaq/accel"
#define ANGLE (.5 * 256)
#define RMAX (.75 * 256)
#define RMIN (-.75 * 256)

#define POLLRATE 100
#define NSAMPS 4

static int err = 0;

int
main (int argc, char **argv)
{
  Display *dpy;
  XRRScreenSize   *sizes;
  XRRScreenConfiguration *sc;
  int		    nsize;
  Window	    root;
  int		    n = 0, i;
  Status	    status;
  Rotation	    rotation, current_rotation, rotations;
  XRRScreenChangeNotifyEvent event;
  static Rotation old = 0;
  int accfd;
  int nsamps = 0;
  struct {
    short x_accel;
    short y_accel;
  } racc;

  struct accel {
    int x_accel;
    int y_accel;
  };
  struct accel acc[NSAMPS];
  struct accel ave;
  struct accel cal = { 0, 0 };

  int calibrated = 0;
  int product;
  int asq = ANGLE * ANGLE;

  pid_t pid;

  /*
   * this is coded to sleep for 10 seconds if either:
   *  the X server isn't running, or:
   *  the accelerometer isn't present, so it can run under any circumstance.
   */

  pid = fork();
  if (pid == -1) {
    fprintf(stderr, "Fork failed errno = %d\n", errno);
  }
  if (pid > 0) {
    printf("%d\n", pid);
    exit(0);
  }
  for (;;) {

    dpy = XOpenDisplay (NULL);
    if (dpy == NULL)    {
      sleep(10);
    }

    root = DefaultRootWindow (dpy);

    sc = XRRGetScreenInfo (dpy, root);

    accfd = open(ACC_DEV, O_RDONLY);

    if (accfd == -1) {
      sleep(10);
    }
    else {
      while (1) {
	n += 1;
	if (n >= NSAMPS) n = 0;

	poll (NULL, 0, POLLRATE);
	if (read (accfd, &racc, sizeof(racc)) == -1) {
	  goto leave;
	}

	nsamps += 1;
	/* compute running average, yes, ma, there are better ways */
	acc[n].x_accel = (int) racc.x_accel;
	acc[n].y_accel = (int) racc.y_accel;

	ave.x_accel = 0; ave.y_accel = 0;
	for (i = 0; i < NSAMPS; i++) {
	  ave.x_accel += acc[i].x_accel;
	  ave.y_accel += acc[i].y_accel;
	}

	ave.x_accel = ave.x_accel / NSAMPS - cal.x_accel;
	ave.y_accel = ave.y_accel / NSAMPS - cal.y_accel;

	/* lame attempt at self calibration */

	if (nsamps == NSAMPS) {
	  cal = ave;
	  if (argc > 1) printf("cal x = %d; cal y = %d\n", cal.x_accel, cal.y_accel );
	}
	product = ave.x_accel * ave.x_accel +
	  ave.y_accel * ave.y_accel;

	if (argc > 2) printf("ave x = %d, ave y = %d\n", ave.x_accel, ave.y_accel);

	if (product > asq ) {
	  rotation = 0;

	  /*
	   * if(ave.x_accel > RMAX) rotation = 8;
	   * if(ave.x_accel < RMIN) rotation = 2;
	   * if(ave.y_accel > RMAX) rotation = 4;
	   * if(ave.y_accel < RMIN) rotation = 1;
	   */

	  if(ave.x_accel > RMAX) rotation = 1;
	  if(ave.x_accel < RMIN) rotation = 4;
	  if(ave.y_accel > RMAX) rotation = 8;
	  if(ave.y_accel < RMIN) rotation = 2;
	  /* don't try to rotate unless needed */
	  if (rotation && (rotation != old)) {
	    status = XRRSetScreenConfig (dpy, sc, DefaultRootWindow (dpy),
					 (SizeID) 0, 0, rotation, CurrentTime);
	    if (status == RRSetConfigSuccess) old = rotation;
	  }
	}
      }
      leave: XRRFreeScreenInfo(sc);
    }
  }
}


About these ads

20 Responses to “Apple Patents Portrait-Landscape Flipping: the patent system is broken…”

  1. Ivan Says:

    Hi,

    While the patent is outrageous, and I agree that some software patents are just plain ridiculous (including this one). Perhaps the ambiguity of this one is particularly confusing. It seems, by overlooking at the patent application, that they patented a gesture that rotates, overriding the acceleration sensors.

    So rotation by sensors has been done, rotation with configuration (such as supported by windows) has done. But now the slippery slope comes with multitouch devices where Apple now wants to patent everything “done with a multi-touch gesture on a portable multifunction device”.

    So the problem is not that they patented something that existed. The problem is that seems like an obvious extension of their product. If someone were to invent mind controlled computers, the “thought required for rotating the screen” could be also patented, because it hasn’t (of course) before.

    That’s probably the ridiculous part of such patent application.

    • gettys Says:

      You are stating (in another useful way) exactly my problem with the patent system. Any simple combination of any feature seems to be patentable now, even though the result of that combination may have been blindingly obvious from the get-go This is causing a multiplicative explosion of microscopic patents, even ignoring prior art problems.

  2. Ken Nicholas Says:

    Don’t forget the work on Itsy. Here is a Compaq WRL paper from May 2000.

    http://www.vs.inf.ethz.ch/edu/SS2005/DS/papers/embodied/bartlett-rocknscroll.pdf

    • gettys Says:

      Yes, exactly, thanks for reminding me: I should have dug up that link. The Linux iPAQ work and the BackPAQ both were preceded by Itsy done by our colleagues at DEC/Compaq WRL. The biggest difference was the lack of wireless networking in Itsy. What I don’t know offhand (not having dug for it) if the code for the accelerometer was published or not for Itsy, and I don’t remember if the rotation was generalized (which it was in our X work slightly later). Also, the iPAQ + BackPAQ was much more widely available that Itsy’s were.

      To make it very clear: the Itsy was the work that inspired Linux on the Compaq iPAQ in the first place, and Keith and I first ported X to the Itsy; I don’t remember if we started work on xrandr on Itsy, or if that occurred later. And it’s entirely possible the Itsy was really fully equivalent to what we did later on the iPAQ with the accelerometer; but memory after over a decade is faded enough I don’t remember that level of detail: I do remember writing the code to control xrandr on the iPAQ with the accelerometer.

      • jaharkes Says:

        I actually have a full Itsy development source tree laying around and there is an example photo album application in apps/rocknscroll that uses gestures to change orientation as well as gestures to flip to the previous and next picture.

        Here is the top bit of Software/apps/apps/rocknscroll/album.c.

        /*
        Rock ‘n’ Scroll Picture Album

        Copyright (C) 1998, 1999 Compaq Computer Corporation. All rights reserved.

        Basic program structure:

        1. Read each picture and build pixel map in memory.
        2. Read each thumb nail and build menu pixel map in memory.
        3. Open LCD and put up configuration text.
        4. Configure RnS. Use rotates device in a vertical plane and
        then holds it still at a good viewing angle.
        5. Display menu
        6. Control loop:

        if pitch down gesture then
        restore display position
        lock display
        wait for unlock / recalibrate / reverse scroll or
        button push to quit
        else
        if still on edge then
        re-orient
        if scroll then
        scroll pictures or menu
        if either roll gesture then
        if in menu then
        save gesture as next picture gesture
        display picture
        else
        if next picture gesture then
        display the next picture
        else
        display the next picture
        */

        • gettys Says:

          Cool!

          I have no memory of having played with the gesture stuff on the Itsy. But I’m not surprised that the Itsy folks did that in the slightest…

          I was more motivated to get my favorite window system running on the Itsy (and then the iPAQ), and making rotation with the accelerometer work with all X applications by teaching it what to do.

  3. ChrisK Says:

    The argument here isn’t so much as to whether it was obvious or not. The real problem here is that the patent examiners are not “skilled in the art” and apparently unaware of prior art. So sad. Now it will take millions of dollars for someone to try to invalidate the patent.

  4. Joel Says:

    Europe, please hold on refusing software patents, please!

  5. Engineer Says:

    You’re flat out misrepresenting the patent, which you “proudly” state you haven’t read. Further you’re pretending like a sylus driven input is the same as a modern finger driven touch screen. That is flat out dishonest. The reason styluses were used in those devices is it is damn hard to figure out what someone is touching when the pixel is less than 1% of the surface area of the finger. That being damn hard, and never done before, is a big component of what Apple has invented.

    What’s broken is the level of dishonesty and misinformation people are perpetuating. And the agenda is obvious– you want to deny Apple, the singular computer hardware company that still actually innovates– the fruits of their inventions so that you can get a ripoff android clone for a few bucks cheaper.

    I once got a patent that was covered by slashdot. The patent was on keeping distributed clocks in sync on a realtime distributed database, yet the way slashdot argued it they claimed we had patented the “idea” of the MMO— and pointed to Ultima Online as “prior art” — a game that came out *after* the patent was filed.

    Shame on you.

    Your entire premise is based on a lie, and perpetuates the myth that these patents are over ideas like using an accelerometer to shift screen orientation.

    • gettys Says:

      Not proudly: again, the patent system is the reason I can’t go read the patent, due to the possibility of treble damages.

  6. Elmer Says:

    Your assumption about what the patent covers is incorrect. Apple did not patent portrait-landscape flipping, and the prior art on that is even cited in plain language in the background section of the patent.

    What Apple patented was a heuristic for conditionally overriding the input from the accelerometer in a way that is not included in any of the prior art you have cited.

    If you would read the patent I am certain you would agree that it refers to an enhancement above and beyond what existed before. But as you seem disinclined to do so, I urge you instead to not make claims about patents that you have not read, as it encourages others to believe false interpretations. In essence, it is misinformation. Whether you intended to spread misinformation or not, you can’t trust people on the Internet to independently verify your claims.

    • gettys Says:

      A third problem with the patent system is the treble damages in case of knowingly violating patents: this causes most company policies to be that engineers avoid reading patents. Since the original intent of patent system was intended to be trade between a limited monopoly in return for publication, this aspect is also broken.

      So no, I can’t go read the patent. The patent system is broken.

      And again, I’d have to ask if the enhancement is obvious or not: there are far too many patents on pretty obvious problems.

  7. kinkfisher Says:

    Could people please, please, PLEASE read the claims before declaring a patent obvious and trivial and the patent system broken? And if you “don’t read patents”, please refrain from making posts and comments thereon.

    Maybe this will make you feel better about the USPTO: this patent’s claims involve gesture input for flipping, so the code above is not too relevant. If you need more details, the link is right there on the /. article.

    PS: http://en.wikipedia.org/wiki/Software_patent#Europe

  8. Shimshon Says:

    I was lurking at the time on the IRC channel that people involved in developing Familiar were using. I remember this, as well as many other things.

  9. Mary Branscombe Says:

    I’m going to suggest that Bell Labs has a patent librarian who can help you with researching patents (you obviously can’t read them yourself so every research lab has people like this as cutouts to protect you). As I’m sure you well know, it’s the exact combination of claims that is protected, so you have to know what they are to know if the patent is worthy or insultingly trivial. The argument would have to be be not on prior art but on whether this next step of over-riding the accelerometer rotation with a multi-touch gesture is obvious given prior art but without over-accurate hindsight… this is proving a tricky area and the reform on patent reexanimations is going to be interesting.

    • gettys Says:

      Good point; I’ll see if I can get some help and take a look at the exact claim. It will likely be an lawyer, however.

      So we have three problems here:

      1) prior art, which may not apply to my example certainly we did not have a multi-touch screen to play with and did not explore that area.

      2) Obviousness may be in the eye of the beholder, but certainly I’ve seen ideas which were non-obvious. The current broken patent system is encouraging filing of patents just for protection of every trivial idea.

      3) the treble damages problem, which is why I did not go read the patent in the first place, and stifles actual innovation (independent of whether you thing software patents are a good or bad idea, being unable to know what is going on elsewhere defeats part of the fundamental thesis of why patents were granted in the first place.

      I like my algorithm: the one on the iPAD is badly flawed and rotates the screen at times I really want it to stay put. As I remember it, my code only rotates the screen if the screen is within 45 degrees of the vertical; you end up with a very natural gesture (moving the entire device, not a gesture on the screen) to indicate when you want the screen to rotate, and it doesn’t flip when you have it approximately in the horizontal plane (which the iPad’s algorithm does). Overriding the rotation is seldom necessary.

  10. Bystander Says:

    When you write that you’re worried about treble damages, you’re basically concerned about the charge of willful infringement. A fairly recent ruling by the Federal Circuit significantly changed the ground rules for winning a claim of willful infringement.

    http://www.irmi.com/expert/articles/2008/warren05-intellectual-property-law.aspx

    “The Court of Appeals for the Federal Circuit recently redefined willful infringement. Patent cases are heard by federal district courts, and appeals to patent cases are heard by the Federal Circuit. As such, the Federal Circuit creates much of the controlling precedent for patent cases. On August 20, 2007, the Federal Circuit ruled on willful infringement in its decision, In re Seagate Tech., LLC., 497 F.3d 1360 (Fed. Cir. 2007). In the decision, the Federal Circuit overruled its own precedent and held that “proof of willful infringement permitting enhanced damages requires at least a showing of objective recklessness.” (Previously, regarding willfulness, an infringer was only under a duty of care to avoid infringement.) That is to say, for a court to increase a damages award on the basis of willful infringement, the patentee must show that the infringer was objectively reckless. The Federal Circuit explained this standard by stating:

    a patentee must show by clear and convincing evidence that the infringer acted despite an objectively high likelihood that its actions constituted infringement of a valid patent. “

    http://www.americanbar.org/content/dam/aba/migrated/intelprop/magazine/LandslideJan09_LaFuze.authcheckdam.pdf

    “The switch from the affirmative duty of care to the objective recklessness standard of Seagate expanded the avenues for an accused infringer to defeat an allegation of willful infringement. Prior to Seagate, willful infringement could be shown by establishing that, despite having knowledge of the patent at issue, the accused infringer failed to exercise due care to investigate whether its actions might infringe a valid patent. The Seagate decision changed that by doing away with the accused infringer’s affirmative duty of care.1

    The objective recklessness standard of Seagate requires the patentee to prove that the accused infringer (1) acted despite an objectively high likelihood that its actions constituted infringement of a valid patent and (2) knew or should have known of that objectively high risk.2 “

    “Moreover, the new standard alters the semantics of how a patentee should frame the accused infringer’s alleged willfulness. Instead of merely asserting that the accused infringer failed to adequately investigate, the patentee must now also focus heavily on the overall, objective strength of the patentee’s liability case.3 “

    What this means is that the burden for showing willful infringement has shifted from a defendant needing to show they diligently investigated possible infringement to the plaintiff needing to show how the defendant was objectively reckless. This is a change in the rules for the better.

    • gettys Says:

      That’s really good to know. The willful infringement problem was one of the most egregious problems with the patent system.

      It isn’t clear such word has spread through company policies as yet. I will certainly take it up with my company’s legal department.

  11. Right Wing-nut Says:

    I’m afraid I’m going to drive-by comment, but here goes.

    As part of Algore’s “Reinventing Government” scheme, the charter of the USPO was changed from “issue valid patents” to “help our customers obtain patents”. Ever since then, there is NO presumption of validity attached to a patent. For that you need to hire lawyers to take things to the courts. Lots of lawyers.

Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s


Follow

Get every new post delivered to your Inbox.

Join 674 other followers

%d bloggers like this: