I noticed with interest Slashdot’s article last week on Apple Patenting Portrait-Landscape flipping based on control of one or more accelerometers in Slashdot last week. As I work at Bell Labs these days, I don’t read patents, so I’ll just go on the summary I read there.
Here’s some prior art from June 2001. In that period, at Compaq/HP’s Cambridge Research Laboratory, we had ported Linux to the iPAQ handheld (with touch screen & expansion capability). Colleagues of mine, including Jamey Hicks, Andy Christian, Frank Bomba, Ben Curis had built an expansion pack for the iPAQ, called the BackPAQ (just like Apple has an I fetish, Compaq had a paq fetish and liked “i”s as well), with accelerometer, camera, and additional expansion capability including additional battery, for our (and other’s) research as part of “Project Mercury”; it was obvious that such devices would become standard in short order, but no device at the time had them integral. Quite a few BackPAQ’s were built and distributed to researchers around the world (small number of hundreds, if I remember correctly). We wrote some papers, distributed a bunch of BackPAQ’s to like minded researchers around the world, and demonstrated the code at the Usenix conference and elsewhere, and published all the code on handhelds.org (which seems down at the moment). The probability of Apple employees having seen this device and it rotating the screen is an absolute certainty; not only did we show the BackPAQ off at numerous conferences, but we built significant numbers used at universities.
It was blindingly obvious to us that hooking up the accelerometer to be able to rotate the screen would be “a good idea”. Keith Packard and I wrote the xrandr X Window System extension specifically to support screen rotation, for the iPAQ handheld using his TinyX driver (the X extension then became a standard part of the X Window System releases in X.org). I wrote (in an hour or two) the first version of the xaccel daemon that took the accelerometer data and controlled the screen rotation. I first packaged it (in ipgk format, for the iPAQ Familiar Linux distribution) on June 11, 2001 to enable the code’s distribution. Ironically, I like what I remember of xaccel’s behaviour better than what I now see on the iPhone and the iPad I own.
Since I can’t go reading Apple’s patent itself, I’ll just note:
- This is a handheld device, with 802.11 wireless (later versions of the iPAQ became phones).
- It has a touch screen
- It has an accelerometer in the BackPaq
- It used the data from the accelerometer with simple heuristics to control the orientation (portrait or landscape) of the screen (in this case, running the X Window System
Update 1:
We have three problems here:
- prior art, which may not apply to my example certainly we did not have a multi-touch screen to play with and did not explore that area.
- Obviousness may be in the eye of the beholder, but certainly I’ve seen ideas which were non-obvious. The current broken patent system is encouraging filing of patents just for protection of every trivial idea, and to use as weapons against competitors, whether there is merit in them or not.
- the treble damages problem, which is why I did not go read the patent in the first place, and stifles actual innovation (independent of whether you thing software patents are a good or bad idea, being unable to know what is going on elsewhere defeats part of the original bargain of why patents were granted in the first place.
Update 2
Jaharks of CMU in a comment below notes that the Itsy folks did gesture based screen rotation on the Itsy. Quite a few Itsy‘s (the spiritual predecessor to the iPAQ, and to my knowledge the first handheld device to run Linux, and the inspiration/cause of our handhelds.org work) were built and distributed to universities, along with the source code.
/* * $XFree86: xc/lib/Xrandr/Xrandr.c,v 1.4 2001/06/07 15:33:43 keithp Exp $ * * Copyright 2000 Compaq Computer Corporation, Inc. * * Permission to use, copy, modify, distribute, and sell this software and its * documentation for any purpose is hereby granted without fee, provided that * the above copyright notice appear in all copies and that both that * copyright notice and this permission notice appear in supporting * documentation, and that the name of Compaq not be used in advertising or * publicity pertaining to distribution of the software without specific, * written prior permission. Compaq makes no representations about the * suitability of this software for any purpose. It is provided "as is" * without express or implied warranty. * * COMPAQ DISCLAIMS ALL WARRANTIES WITH REGARD TO THIS SOFTWARE, INCLUDING ALL * IMPLIED WARRANTIES OF MERCHANTABILITY AND FITNESS, IN NO EVENT SHALL COMPAQ * BE LIABLE FOR ANY SPECIAL, INDIRECT OR CONSEQUENTIAL DAMAGES OR ANY DAMAGES * WHATSOEVER RESULTING FROM LOSS OF USE, DATA OR PROFITS, WHETHER IN AN ACTION * OF CONTRACT, NEGLIGENCE OR OTHER TORTIOUS ACTION, ARISING OUT OF OR IN * CONNECTION WITH THE USE OR PERFORMANCE OF THIS SOFTWARE. * * Author: Jim Gettys, Compaq Computer Corporation, Inc. */ #include <stdio.h> #include <X11/Xlib.h> #include <X11/Xproto.h> #include <X11/extensions/Xrandr.h> #include <sys/types.h> #include <sys/stat.h> #include <fcntl.h> #include <unistd.h> #include <math.h> #include <stdlib.h> #include <sys/poll.h> #include <errno.h> #define ACC_DEV "/dev/backpaq/accel" #define ANGLE (.5 * 256) #define RMAX (.75 * 256) #define RMIN (-.75 * 256) #define POLLRATE 100 #define NSAMPS 4 static int err = 0; int main (int argc, char **argv) { Display *dpy; XRRScreenSize *sizes; XRRScreenConfiguration *sc; int nsize; Window root; int n = 0, i; Status status; Rotation rotation, current_rotation, rotations; XRRScreenChangeNotifyEvent event; static Rotation old = 0; int accfd; int nsamps = 0; struct { short x_accel; short y_accel; } racc; struct accel { int x_accel; int y_accel; }; struct accel acc[NSAMPS]; struct accel ave; struct accel cal = { 0, 0 }; int calibrated = 0; int product; int asq = ANGLE * ANGLE; pid_t pid; /* * this is coded to sleep for 10 seconds if either: * the X server isn't running, or: * the accelerometer isn't present, so it can run under any circumstance. */ pid = fork(); if (pid == -1) { fprintf(stderr, "Fork failed errno = %d\n", errno); } if (pid > 0) { printf("%d\n", pid); exit(0); } for (;;) { dpy = XOpenDisplay (NULL); if (dpy == NULL) { sleep(10); } root = DefaultRootWindow (dpy); sc = XRRGetScreenInfo (dpy, root); accfd = open(ACC_DEV, O_RDONLY); if (accfd == -1) { sleep(10); } else { while (1) { n += 1; if (n >= NSAMPS) n = 0; poll (NULL, 0, POLLRATE); if (read (accfd, &racc, sizeof(racc)) == -1) { goto leave; } nsamps += 1; /* compute running average, yes, ma, there are better ways */ acc[n].x_accel = (int) racc.x_accel; acc[n].y_accel = (int) racc.y_accel; ave.x_accel = 0; ave.y_accel = 0; for (i = 0; i < NSAMPS; i++) { ave.x_accel += acc[i].x_accel; ave.y_accel += acc[i].y_accel; } ave.x_accel = ave.x_accel / NSAMPS - cal.x_accel; ave.y_accel = ave.y_accel / NSAMPS - cal.y_accel; /* lame attempt at self calibration */ if (nsamps == NSAMPS) { cal = ave; if (argc > 1) printf("cal x = %d; cal y = %d\n", cal.x_accel, cal.y_accel ); } product = ave.x_accel * ave.x_accel + ave.y_accel * ave.y_accel; if (argc > 2) printf("ave x = %d, ave y = %d\n", ave.x_accel, ave.y_accel); if (product > asq ) { rotation = 0; /* * if(ave.x_accel > RMAX) rotation = 8; * if(ave.x_accel < RMIN) rotation = 2; * if(ave.y_accel > RMAX) rotation = 4; * if(ave.y_accel < RMIN) rotation = 1; */ if(ave.x_accel > RMAX) rotation = 1; if(ave.x_accel < RMIN) rotation = 4; if(ave.y_accel > RMAX) rotation = 8; if(ave.y_accel < RMIN) rotation = 2; /* don't try to rotate unless needed */ if (rotation && (rotation != old)) { status = XRRSetScreenConfig (dpy, sc, DefaultRootWindow (dpy), (SizeID) 0, 0, rotation, CurrentTime); if (status == RRSetConfigSuccess) old = rotation; } } } leave: XRRFreeScreenInfo(sc); } } }
July 18, 2011 at 1:38 pm |
Hi,
While the patent is outrageous, and I agree that some software patents are just plain ridiculous (including this one). Perhaps the ambiguity of this one is particularly confusing. It seems, by overlooking at the patent application, that they patented a gesture that rotates, overriding the acceleration sensors.
So rotation by sensors has been done, rotation with configuration (such as supported by windows) has done. But now the slippery slope comes with multitouch devices where Apple now wants to patent everything “done with a multi-touch gesture on a portable multifunction device”.
So the problem is not that they patented something that existed. The problem is that seems like an obvious extension of their product. If someone were to invent mind controlled computers, the “thought required for rotating the screen” could be also patented, because it hasn’t (of course) before.
That’s probably the ridiculous part of such patent application.
July 18, 2011 at 2:04 pm |
You are stating (in another useful way) exactly my problem with the patent system. Any simple combination of any feature seems to be patentable now, even though the result of that combination may have been blindingly obvious from the get-go This is causing a multiplicative explosion of microscopic patents, even ignoring prior art problems.
July 18, 2011 at 1:49 pm |
Don’t forget the work on Itsy. Here is a Compaq WRL paper from May 2000.
Click to access bartlett-rocknscroll.pdf
July 18, 2011 at 2:02 pm |
Yes, exactly, thanks for reminding me: I should have dug up that link. The Linux iPAQ work and the BackPAQ both were preceded by Itsy done by our colleagues at DEC/Compaq WRL. The biggest difference was the lack of wireless networking in Itsy. What I don’t know offhand (not having dug for it) if the code for the accelerometer was published or not for Itsy, and I don’t remember if the rotation was generalized (which it was in our X work slightly later). Also, the iPAQ + BackPAQ was much more widely available that Itsy’s were.
To make it very clear: the Itsy was the work that inspired Linux on the Compaq iPAQ in the first place, and Keith and I first ported X to the Itsy; I don’t remember if we started work on xrandr on Itsy, or if that occurred later. And it’s entirely possible the Itsy was really fully equivalent to what we did later on the iPAQ with the accelerometer; but memory after over a decade is faded enough I don’t remember that level of detail: I do remember writing the code to control xrandr on the iPAQ with the accelerometer.
July 19, 2011 at 11:55 pm |
I actually have a full Itsy development source tree laying around and there is an example photo album application in apps/rocknscroll that uses gestures to change orientation as well as gestures to flip to the previous and next picture.
Here is the top bit of Software/apps/apps/rocknscroll/album.c.
/*
Rock ‘n’ Scroll Picture Album
Copyright (C) 1998, 1999 Compaq Computer Corporation. All rights reserved.
Basic program structure:
1. Read each picture and build pixel map in memory.
2. Read each thumb nail and build menu pixel map in memory.
3. Open LCD and put up configuration text.
4. Configure RnS. Use rotates device in a vertical plane and
then holds it still at a good viewing angle.
5. Display menu
6. Control loop:
if pitch down gesture then
restore display position
lock display
wait for unlock / recalibrate / reverse scroll or
button push to quit
else
if still on edge then
re-orient
if scroll then
scroll pictures or menu
if either roll gesture then
if in menu then
save gesture as next picture gesture
display picture
else
if next picture gesture then
display the next picture
else
display the next picture
*/
July 20, 2011 at 9:34 am
Cool!
I have no memory of having played with the gesture stuff on the Itsy. But I’m not surprised that the Itsy folks did that in the slightest…
I was more motivated to get my favorite window system running on the Itsy (and then the iPAQ), and making rotation with the accelerometer work with all X applications by teaching it what to do.
July 18, 2011 at 2:01 pm |
The argument here isn’t so much as to whether it was obvious or not. The real problem here is that the patent examiners are not “skilled in the art” and apparently unaware of prior art. So sad. Now it will take millions of dollars for someone to try to invalidate the patent.
July 18, 2011 at 2:08 pm |
It’s both prior art and obviousness I’m having trouble with…
July 18, 2011 at 2:34 pm |
Europe, please hold on refusing software patents, please!
July 18, 2011 at 8:42 pm |
You’re flat out misrepresenting the patent, which you “proudly” state you haven’t read. Further you’re pretending like a sylus driven input is the same as a modern finger driven touch screen. That is flat out dishonest. The reason styluses were used in those devices is it is damn hard to figure out what someone is touching when the pixel is less than 1% of the surface area of the finger. That being damn hard, and never done before, is a big component of what Apple has invented.
What’s broken is the level of dishonesty and misinformation people are perpetuating. And the agenda is obvious– you want to deny Apple, the singular computer hardware company that still actually innovates– the fruits of their inventions so that you can get a ripoff android clone for a few bucks cheaper.
I once got a patent that was covered by slashdot. The patent was on keeping distributed clocks in sync on a realtime distributed database, yet the way slashdot argued it they claimed we had patented the “idea” of the MMO— and pointed to Ultima Online as “prior art” — a game that came out *after* the patent was filed.
Shame on you.
Your entire premise is based on a lie, and perpetuates the myth that these patents are over ideas like using an accelerometer to shift screen orientation.
July 19, 2011 at 9:00 am |
Not proudly: again, the patent system is the reason I can’t go read the patent, due to the possibility of treble damages.
July 18, 2011 at 9:18 pm |
Your assumption about what the patent covers is incorrect. Apple did not patent portrait-landscape flipping, and the prior art on that is even cited in plain language in the background section of the patent.
What Apple patented was a heuristic for conditionally overriding the input from the accelerometer in a way that is not included in any of the prior art you have cited.
If you would read the patent I am certain you would agree that it refers to an enhancement above and beyond what existed before. But as you seem disinclined to do so, I urge you instead to not make claims about patents that you have not read, as it encourages others to believe false interpretations. In essence, it is misinformation. Whether you intended to spread misinformation or not, you can’t trust people on the Internet to independently verify your claims.
July 19, 2011 at 8:59 am |
A third problem with the patent system is the treble damages in case of knowingly violating patents: this causes most company policies to be that engineers avoid reading patents. Since the original intent of patent system was intended to be trade between a limited monopoly in return for publication, this aspect is also broken.
So no, I can’t go read the patent. The patent system is broken.
And again, I’d have to ask if the enhancement is obvious or not: there are far too many patents on pretty obvious problems.
July 18, 2011 at 9:25 pm |
Could people please, please, PLEASE read the claims before declaring a patent obvious and trivial and the patent system broken? And if you “don’t read patents”, please refrain from making posts and comments thereon.
Maybe this will make you feel better about the USPTO: this patent’s claims involve gesture input for flipping, so the code above is not too relevant. If you need more details, the link is right there on the /. article.
PS: http://en.wikipedia.org/wiki/Software_patent#Europe
July 19, 2011 at 1:39 am |
I was lurking at the time on the IRC channel that people involved in developing Familiar were using. I remember this, as well as many other things.
July 19, 2011 at 9:25 am |
I’m going to suggest that Bell Labs has a patent librarian who can help you with researching patents (you obviously can’t read them yourself so every research lab has people like this as cutouts to protect you). As I’m sure you well know, it’s the exact combination of claims that is protected, so you have to know what they are to know if the patent is worthy or insultingly trivial. The argument would have to be be not on prior art but on whether this next step of over-riding the accelerometer rotation with a multi-touch gesture is obvious given prior art but without over-accurate hindsight… this is proving a tricky area and the reform on patent reexanimations is going to be interesting.
July 19, 2011 at 10:09 am |
Good point; I’ll see if I can get some help and take a look at the exact claim. It will likely be an lawyer, however.
So we have three problems here:
1) prior art, which may not apply to my example certainly we did not have a multi-touch screen to play with and did not explore that area.
2) Obviousness may be in the eye of the beholder, but certainly I’ve seen ideas which were non-obvious. The current broken patent system is encouraging filing of patents just for protection of every trivial idea.
3) the treble damages problem, which is why I did not go read the patent in the first place, and stifles actual innovation (independent of whether you thing software patents are a good or bad idea, being unable to know what is going on elsewhere defeats part of the fundamental thesis of why patents were granted in the first place.
I like my algorithm: the one on the iPAD is badly flawed and rotates the screen at times I really want it to stay put. As I remember it, my code only rotates the screen if the screen is within 45 degrees of the vertical; you end up with a very natural gesture (moving the entire device, not a gesture on the screen) to indicate when you want the screen to rotate, and it doesn’t flip when you have it approximately in the horizontal plane (which the iPad’s algorithm does). Overriding the rotation is seldom necessary.
July 19, 2011 at 4:18 pm |
When you write that you’re worried about treble damages, you’re basically concerned about the charge of willful infringement. A fairly recent ruling by the Federal Circuit significantly changed the ground rules for winning a claim of willful infringement.
http://www.irmi.com/expert/articles/2008/warren05-intellectual-property-law.aspx
http://www.americanbar.org/content/dam/aba/migrated/intelprop/magazine/LandslideJan09_LaFuze.authcheckdam.pdf
What this means is that the burden for showing willful infringement has shifted from a defendant needing to show they diligently investigated possible infringement to the plaintiff needing to show how the defendant was objectively reckless. This is a change in the rules for the better.
July 19, 2011 at 4:30 pm |
That’s really good to know. The willful infringement problem was one of the most egregious problems with the patent system.
It isn’t clear such word has spread through company policies as yet. I will certainly take it up with my company’s legal department.
February 3, 2012 at 2:53 pm |
I’m afraid I’m going to drive-by comment, but here goes.
As part of Algore’s “Reinventing Government” scheme, the charter of the USPO was changed from “issue valid patents” to “help our customers obtain patents”. Ever since then, there is NO presumption of validity attached to a patent. For that you need to hire lawyers to take things to the courts. Lots of lawyers.