Comcast and debugging….

For the last number of months (at least three months), I’ve been suffering with poor Internet service (intermittent high packet loss rates).  When my service has worked, it has worked pretty well, but minutes to an hour or so would go by when I’d see high loss rates (30-60%) and suffer accordingly.

Grrrrrr…

I need to give this story a bit of context; I won’t bore you with the shaggy dog story (though somewhat  interesting) of getting my cable Internet service to actually work after moving into this house.  To skip this history, it suffices to say that Comcast installed a new cable to the house, but the contractor they hired goofed and pulled the wrong cable for the length of the run, so I was warned that I might have trouble sometime in the future as a result of the cable not meeting their engineering standards.  Some sort of amplifier was installed to try to paper over the problem.  This was done with my permission, as I didn’t want my lawn torn up again, nor to waste Comcast’s money if it wasn’t really necessary.

This phase of the story started last summer, when we were struck by lightning.  A bunch of my gear in my house was damaged, including the Comcast provided cable modem, my router box, one of my ethernet switches, a couple ethernet cards in several machines, the irrigation system, and so on.  The cable modem was the most fun: its power supply literally blew up.

So off to the store I went, and over a period of few weeks, I got everything working again as time permitted and additional damage was discovered.

So far so good.  Almost all of my networking gear (with the exception of a single switch) was new at the end of this process.

But as I had had to go grab a little commercial firewall/router box for my point of presence (I want a system I can just unplug and replug in the case of problems when I’m traveling, so have avoided putting a little Linux box to do this, and to save the planet some joules…), and had no working Internet service, I had little time to see what was out there and just grabbed the first router box (a Linksys BEFSR41)  I could find at the store I went to.

It turned out to lack a feature I want: to disable the network for particular mac addresses at particular times of day (we got very tired of arguing with our sleep deprived kids to go to bed, etc…), so I went on line and ordered the follow-on to the dlink box I had originally had that had that feature, to replace the one I bought in my emergency.  I installed it, and all seemed well (except dlink had removed the other feature I liked: the ability to configure alternate name servers: at times I’ve been unimpressed by theirs….).

Somewhat later I noticed that our network had become intermittent; most of the time, it would work great, but other times, it would become unusable, possibly related to the amount of traffic we were generating, but this was never clear.

So I started the service call cycle.

After some number of pokes over a couple month period, Comcast noticed they had a seriously unhappy customer (and one who generally knows what he’s talking about) and escalated the issue to the supervisor in charge of network problems for my town and several adjacent towns.  After several more conversations and visits by him and technicians, we were coming to the conclusion either the cable from our house or inside the house might be at fault, and he had verified what I had been told after the cable had been installed that it was the incorrect grade of cable given the run.  Replacing the cable to the would be both expensive for Comcast, do another number on my lawn, and have to wait until the ground thaws, so both he and I were not happy campers.

But he had asked one of his technicians to monitor my cable modem, and after at least some tweaking, Comcast was seeing no further problems from their end….

Strange.  I thought about a scenario where the cable modem box might do monitoring traffic at higher priority than customer’s traffic, and still smelled a rat. But on the request of the nice Comcast supervisor, I repeated earlier experiments again, this time.  I also noted that running a traceroute while pinging the cable modem could induce a high packet loss rate just to the cable modem.  Ahah!, I could finally induce the problem more quickly, and do so before the packets going over the suspect cable.  An afternoon of heavy debugging, and it was clear the problem was in the ethernet between the cablemodem and the router; either could be at fault).

So I switched back to the Linksys BEFSR41 router I bought after the lightning strike.

End of high packet loss problem, and the culprit appeared to be the DLink EBR 2310, except…

An hour or so after resetting all the network gear (mine, and the cable modem), the ping to the cable modem would mysteriously stop without warning, which meant my traceroute test could not be run.  But over a nearly two week period, we saw no other network trouble.

At the suggestion of my helpful, Comcast service  supervisor, I tried resetting my router, and not resetting the rest of my network gear; the pings returned immediately, and the cycle repeated.  One last call to him and he suggested that since the cablemodem was so inherently dumb, I should go looking for trouble in the router as he said the cable modem has no filtering features.

Turns out he was right; there is a “Security” feature in the BEFSR41:  Block Anonymous Internet Requests, that was “on” for some reason; quite bizarrely, if on, it seems to have accounted for my returning ICMP ping responses evaporating after a while.  I would have thought it would have been all or nothing.

So at this point, I’m going to declare success, and my thanks to Comcast.

However, there were a couple of other things that I learned that seems unfortunate:

• I was told inside of Comcast’s network, they deliberately throw away ping/traceroute sorts of information, and that there was no place I could ping to help test the cable down my hill.  This made it MUCH harder to debug: ideally, there should be some way I could ping some place inside of Comcast’s network (ideally, very close to my local cable end); then, by noticing no difference in the packet loss problem going down the cable down the hill versus just pinging the cable modem, I would not have remained stuck on the possibility/probability of it being the cable itself  (which I certainly had reason to suspect, given the history and knowledge that the cable itself is not up to Comcast’s engineering standards).
• The Comcast technician told me they are not allowed to ping beyond their cable modem to to regulatory restrictions (even though I wanted them to); this would have made it much easier for them to help me determine it was on my side of the cable modem (by Comcast being able to ping my router box).  But given the “feature” of the Linksys, it isn’t clear this would have helped.

Conclusions:

1. It seems both network engineering and regulatory regulations are making debugging yet more difficult, and I have no clue how regular people would ever get to the bottom of a situation like this…  A bit of sanity on both fronts (regulatory, and ISP network design) be an improvement.
2. it isn’t always the ISP’s fault….  My thanks to Dave Dumais of Comcast for having the patience to get to the bottom of this with me.
3. my experience with dlink gear is now pretty bad (I had trouble with a wireless device of theirs dying as well a couple years ago).  They also removed two features I wanted from the previous generation of their product (I naively bought the successor product expecting the same functionality, and didn’t even fix their help files on the new model. “Value engineering” at its worst. I think I will avoid them in the future.

Time to do something else. (and typewriters aren’t so bad…)

As you have likely heard, OLPC is having a tough time. I’m one of the many casualties.  If you know of an opportunity that I’d be interested in, please do let me know.

Most of the last month since then went in a different direction having nothing to do with anything of general use and of limited interest:  filling out forms for my kid’s education, and helping my wife with the kids while she dealt with finishing up her master’s degree for middle school math teaching. Andi had Lyme disease in December,  so that had to get finished up, along with our taxes and all this paper work.  Ugh…

While these forms were primarily for my daughter’s secondary school education, I’m amazed that in 2009 the systems can possibly be so painful.  The only part of this that seems remotely standardized across schools are recommendation forms.

Since we’ve long since either given away or lost the one typewriter we had, filling out forms was a PITA.  Doing it by printing longhand wasn’t reasonable either, as both my daughter and I have terrible handwriting.  You might have thought schools would have automated this by now, or at least be using PDF forms so that you could fill them out electronically (like even the U.S. government has done for many of its tax forms….

I finally found a laborious (much more than what filling them out with a typewriter) method that was esthetically acceptable, even if time consuming.The other cookbooks didn’t do quite what I wanted. To help others out with similar pain (and so I wont’ forget myself in the future), I document it below…

I converted PDF’s of the forms to PNG images using ImageMagik

convert -density 300×300 -resize 2550×3300 $1 $2

This gets you a 300dpi png image (if the output file name is .png, anyway), without the horrible artifacts of scanning paper.

1. Then, in OpenOffice 2.4, I created a new document.
2. Now we set up styles.
3. Format->Styles and Formatting (F11); Brings up a dialog; select page style; the terrible dialog has a little page icon.
4. Create a new style; give it a name, such as page 1.
5. Select the background tab.
6. Background as graphics; browse and find the page image you want as a background and use it as the page background
7. Go to the page tab, and clear out the margins; otherwise your page image will get chopped to the margins
8. Repeat the process (and have page 1’s style then invoke page 2’s style using Organizer->Next Style

Then, go back to the document, and “Insert->Manual Break->Page break” for each page. Then you can use the graphics tools to put text fields wherever you need them; you can expect lots of fussing to try to get the paragraph spacing to match the underlying form.  And more fussing to make it neat.

So you then can finally have a presentable document that some other mortal might be able to read. If you have one of those old things called a “typewriter”, I recommend that over what I had to do…

There are possibly/probably other tools that might make this easier (scribus??), but this is the method I figured out.

Give one, Get many…

Here’s a way you can give something to everyone while getting something for yourself (other than our nice machines for your kids, or for you to use at the beach, or when traveling, ….).

Last time I blogged, it was about adopting and amplifying good ideas for the free desktop in general. One project I mentioned, the Façades idea, is of particular importance. Free software is often built by, and used by experts.  Since the Gimp is so easy a target, I’ll pick on it (though it has improved its usability over the years considerably, it has a good way to go).

One of the precepts of OLPC and the Sugar environment is “Low barrier to entry; no ceiling”. While I don’t think we succeeded at avoiding a number of gratuitous ceilings (our choice of window manager comes to mind, which we plan to fix), the fact is we have wonderful software like the Gimp which is too complicated for an 8 year old to use.  We do have nice things like TuxPaint, which is a (child) crowd pleaser; but those do not scale up as the child grows.

As Façades applied to the Gimp demonstrates, any application that is built using at-spi (the accessibility framework), in concert with Composite in the window system, alternate (in this case, simplified) user interfaces can be built.  More speculatively, one can see how to build applications that are literally “composited” together out of piece parts, if something like UNIX pipes can be arranged.

This would allow us to build much more tuned interfaces for particular paths; simpler interfaces for children, better use by the disabled, who may not be able to use certain UI elements effectively, etc, that are now not feasible.  While GUI builders have had this promise, in practice, they seem not to have actually succeeded at delivering this promise (you have to know too much to use them, and many applications have been built without them); maybe Facades can. As a child grows, (s)he could add features as they age, without having to switch tools altogether and start over.

Taking Facades ideas from research prototype  to the free desktop would be a real case of “Give one, Get Many”, for you personally, the children and any disabled people in your life.

Time to Lead….

The free desktop has been in “catchup” mode: catching up to first Windows and now nipping at the heels of the Mac.  Our path has been obvious to date. In some areas, our technology and applications lead; in others we still lag. From here on, progress becomes much less clear, though I’ll bet on the moving herd and natural selection of free software over directed closed commercial development any day.

How now to move from such a reactive strategy to true leadership in all areas? How do you set strategy, when our very culture is that of serendipity, discovery, sharing of ideas, and creation? where a single vision cannot rule?

I recently attended the UIST conference, and discovered there was essentially no overlap between the research community and our own. The conference (in free software terms) was expensive and much was uninteresting to me personally (though most would be interesting to someone in the free desktop community). But a couple of gems at UIST were there that made the trip worth while. For me, it was OctoPocus. For you, it will likely be something else.

For another concrete example, the Facades work built on Metisse is not well known in our community. I highly recommend watching the videos you find at those web sites. Facades, published several years ago, combines at-spi and composite to do things that aren’t possible anywhere else. You too, could edit the gimp in real time to have the UI you want! Peter Korn only became aware of Facades last week, after I had stumbled across it while investigating other work. My jaw dropped.  Peter’s jaw dropped. Please add your jaw to the collection on the floor. We could have deployed this by now! But we did not know it existed!

Dan Olsen notes in “Evaluating User Interface Systems Research“:

“This search for fatal flaws is devastating for systems research. It is virtually impossible for a small team of researchers to recreate all of the capabilities of existing systems or to completely examine all of the eventualities of new concepts. The farther such a team reaches into new territory the more compromises will be required and the more supporting ideas must be left unresolved.  If a new systems approach is attempted, the omissions of some important feature is guaranteed. The existence of a fatal flaw is a given. If the evaluation of the work is focused on “what does it not do” no research system will ever pass. Flaw analysis will frequently be a barrier to new systems research.”

As a result, UI systems research has stagnated due to combined effect of the middleware of the UI stack being closed (or in our case, until the recent flowering of the free desktop undid more than a decade of stagnation) and this effect.  Many good ideas of all kinds have been explored from hardware, to window system UI ideas, to UI ideas for specific classes of applications. But there is much less proof of which ideas actually works in a real system, for the reasons Dan lays out.  The resulting research experiments are usually “toy” systems, and toy applications, built in the limited environment of MacIntosh and Windows where much of the system is off-limits.  The research community has neither the resources nor (if they work on Mac and/or Windows) the ability to make advances “real”.  But we do! We have the resources, the open system, and the ability to and desire to innovate.

Our fundamental advantage we have is the ability to experiment  and modify all areas of the stack; from hardware, to the window system, though toolkits to applications.  Just as compositing has allowed a thousand flowers to bloom (most of which stink, but we’ve picked some that smell pretty sweet) in eye candy, accessibility and in other areas, compositing and other modern free software technologies can be used in new an unexpected ways. We are able to perform experiments, and have a large audience to test the experiments radically faster than commercial software.  Nice as the Web is, there are just some things you can’t do in the web until the underlying technologies have support.  20 years of frustration in the UI research community shows that pushing from research to commercial vendors does not and cannot work at more than a glacial pace. Let us pull people to free software by out innovating and becoming the system with the best user experience, by pulling with the best ideas of free software and research innovation.

How to lead?  I think some of what is going on now (e.g. the work to bring  journals to avoid file management among others) are fundamental.  But there are also almost 20 years of ideas lying fallow since the first flowering of UI work in the 1980’s. By mining research for the best ideas, fostering collaboration with researchers when possible, and combining with your own good ideas and production code now on millions of desktops, we can accelerate our progress.

Thankfully, the ACM is becoming more enlightened: conference proceedings and videos once locked up are now available. (Dan Olsen points out that videos done later found on the researcher’s web sites are often better than those presented at the conference; due to an oversight, the UIST 2008 videos are not yet up). The proceedings of UIST is one good place to start for the free desktop community. There is a list of other ACM conferences on many other topics. Go see which ideas are worth grabbing and running with!

This is a two way street: I had great fun demonstrating Compiz Fusion and some of our eye candy (as well as showing videos of Facades and Metisse) to a number of UIST attendees, who had been unaware that the free desktop had advanced beyond the commercial state of the art, already solving in a real system, problems they could not solve in their “toy” research prototypes on Windows and Mac.  When you have a chance, this is also very worth while: the more bright people working on free software, the better.

But it is up to you to figure out which ideas are good, which are bad (and why), which ideas really work, recombine them with your own great ideas, and get them into people’s hands (both literally, and on the desktop)!

DRM, Bazooka, two feet…

Boy, I wonder about these DRM crazies in the media industry…  Their sanity is definitely questionable…

I went to get an audio ebook for my son to study French (companion to a book in the public library….  Seemed like it should be an easy process, ne?  So after waiting for a few weeks for it to be our turn, we get notified we have 3 days to “check out” the ebook.

Of course, I find to do this, I have to go use my son’s windoze box.  This then causes me to have to install some software; we’ll have 2 weeks to actually download the files.

Said software then says I have to install a security update for Windows Media player.

I go do that…

I download the ebook.

The library web site says I should be able to burn CD’s.  And the license on the files says I should be able to as well.  So I try; the ebook companies software won’t burn the CD’s.

So I go try Windoze media player.

After futzing with it for 15 minutes (the copy to CD button is exquisitely well hidden), and many “adventures” with Windoze Media Player’s lousy UI, I manage to figure out how to burn the 8 CD’s (100Mb or so compressed data expands to fill those CD’s).

So, another hour or so fussing around to burn the CD’s.  I have the ebook for my son.

Now, at the end of all of this, I’ve (completely legally) wasted 8 CD’s of plastic, which I can turn around and rip into any format I like, should I desire to do so.

What did they think they were doing when they put these files under DRM?  “Protecting” their precious data?  Or just going out of their way to ensure I’ll never be a customer of theirs, should I want to get other books/ebooks in that series?  There are clearly some completely mindless people in that industry.  No wonder the media folks are hurting; they seem to actively hate their customers….

One wonders why they don’t put as much energy into making it easy to get royalties rather than making it as obnoxious as possible for their customers.

First post after a long absence. (and a Rant!)

Life’s been a bit crazy, including shipping a machine, and being in the clutches of a surgeon several times.  So I’ve not been blogging….

Here’s this morning’s rant: I just installed Kompozer (formerly nvu) to edit some html.  Here we are in 2008, and I still find that KDE and Gnome aren’t sharing simple settings; I end up with a different font on my screen.

Hey, folks, don’t we care about what people think who don’t care what project the application comes from, but are just looking for good tools to use?  I really don’t want to have to go find what magic knob to twist; I’ve already twisted one knob to my satisfaction… Let’s get together, all…

My other rant are for spammers and crackers…  Evil behavior, indeed. Someone broke into my home server. As time is short, I’ve decided to move my web log to to here where others can maintain the software for me; someday I’ll upload my old postings….