Bufferbloat and network neutrality – back to the past…

There are at least three topics in which I think bufferbloat intersects the currently hot topic of network neutrality. This is entirely personal opinion, and not that of my employer.

• by happenstance, broadband ISP’s are enjoying a serious competitive advantage with any other provider of telephony  and gaming services. I believe it unlikely this advantage put in place with malice aforethought, though I expect conspiracy theorists will enjoy trying to prove otherwise.  I think we’d have heard of this by now or there are a very few, very smart people out there who have managed to figure bufferbloat out and keep their mouths firmly shut. But if they were that smart, why did they not foresee the pain of the next bullet?
• the impact of bufferbloat on ISP’s needs to be well understood, to understand their motivations. I now believe bittorrent hit ISP’s and their customers much harder everyone understands; but that ISP’s diagnosis of root cause was flawed.
• to preserve future innovation for new applications and services.

We should not set public policy going forward without understanding what may actually have happened, rather than a possibly flawed understanding of technical problems.

Unfortunately, everyone has taken now very public positions based on a probably a flawed analysis of  the very real, painful problems they have experienced. Getting everyone to stop and revisit their presumptions, and rethink and possibly change their positions is going to be hard. Please help!

Sherman, please set the Wayback machine to when bittorrent first deployed (2004).

Whose house is of glasse, must not throw stones at another.

• George Herbert, Outlandish Proverbs, 1640

In my last post I outlined the general bufferbloat problem. This post attempts to explain what is going on, and how I started on this investigation, which resulted in (re)discovering that the Internet’s broadband connections are fundamentally broken (others have been there before me).  It is very likely that your broadband connection is badly broken as well; as is your home router; and even your home computer. And there are things you can do immediately to mitigate the brokenness in part, which will cause applications such as VOIP, Skype and gaming to work much, much better, that I’ll cover in more depth very soon. Coming also soon, how this affects the world wide dialog around “network neutrality.”

Bufferbloat is present in all of the broadband technologies, cable, DSL and FIOS alike. And bufferbloat is present in other parts in the Internet as well.

Home Router Puzzle Piece Two – Fun with wireless

I encourage you to perform this experiment, which many of you open source readers can likely perform as you read this entry.

By the end of this exercise, you’ll agree with my conclusion: Your home network can’t walk and chew gum at the same time.

Home Router Puzzle Piece One – Fun with your switch

As enough pieces have fallen into place to make actual predictions, (and the quarry’s spoor noticed), I decided to perform more deliberate experiments to see if I could capture more henchmen of the mastermind. I did.

I’ll start to provide puzzle pieces I’ve discovered here regularly, so you can help with the conviction of the criminal and repair of the damage they have caused.  Today’s simple experiments only involve your router’s switch. We’ll do some experiments on the wireless side of the router next.

Conclusion: something stinks in operating system’s network stacks.  Linux is often worst, with two different but related problems, followed by Mac OSX; Microsoft Windows manages to obfuscate much of it’s problems, but also demonstrably suffers. After mitigation, Linux may be able to perform much better than either.

Browsers and TCP revisited….

Over a decade ago, I was the editor of the HTTP spec (for both proposed and draft standard; thankfully I didn’t sign up to do the full standard work). Roy Fielding has had that headache for the full standard.

As will become clear as this blog unfolds, there is a problem that browsers are exacerbating.

First puzzle piece…

I started putting my network puzzle together from a random set of pieces starting this April.  It’s been a mystery that has been going on for a long time. It’s therefore appropriate to tell the story serially, as many detective stories have been published. It would be novel sized, and you, reader have to help assemble and write some of the the stories too.

Mixing metaphors is always fun,  so onwards to mix as many as we can…. Since blue and violet are very similar colors, and in a nod to the great master’s first work of Sherlock Holmes, we’ll entitle this installment:

A Study in Violet

I already had a number of puzzle pieces on hand for years, and had blogged about some of  them. I had no clue what puzzle they belonged to nor how they fit or even if they were part of the same puzzle.  I mis-assembled what pieces I had. Even lightning has a role in these stories, though indirectly. You have some pieces of the puzzle already, though you probably don’t know you do. Continually upgrading and changing my home network gear both ensured I would see  different pictures come and go, and be bedeviled more than most. Illustrations for the magazine, I guess.

I’m no Sherlock Holmes myself (I am a bear of little brain, and easily confuzled.), and more a jigsaw puzzle person.  And it would be too easy to give you the pieces in order, too; we’ll have to mix the pile of pieces up to make it more challenging for you.  But, to be somewhat fair, we’ll start at a beginning; but only of my own recent story. The actual much more important stories goes back to the invention of packet switching, and the pieces I’ve been responsible for are only a very few and I only enter the story at all a few times.  And, dear reader, you too will play Sherlock Holmes in this corpus, by finding pieces of the puzzle.  And you’ll help find and incarcerate the assassins in our midst, and take personal pleasure in doing so.

The Blue Box

Bell Labs has used a home grown tunnel device, called a “blue box”, developed years ago by our researchers to connect to the company cheaply. Our IT department would like the blue boxes to go away entirely; but there are times we need to be able to run complex networks at home. For example, I work from home and need to be able to use multiple IP cameras, not just a single  desktop or laptop that can run an IPSEC tunnel by itself. It seemed to me unlikely to me the old blue box would suffice for our teleconferencing project; and if not, we needed to get moving on a replacement. So I started to investigate whether the blue box would be adequate for our research.  To put some concrete requirements on a replacement, I did some simple performance testing to compare and contrast the blue boxes with IPSEC tunnels running on my Linux laptop.

Doing scp’s  were limited to not much more than 3-4 megabits/second, even in the downstream direction, where I had 20Mbps from Comcast. The blue box’s poor antique ARM brain just couldn’t do crypto fast enough.  3-4 mbps isn’t even a single HD stream of video. Linux,  on even my old, wimpy laptop was much, much faster than a blue box, though it showed some issues of its own.

This was not a surprise.

I also checked the blue box’s latency while doing the copy (audio and video conferencing cares about latency; ideally, you want zero latency in a device and network, as even the speed of light is too slow since human perceptible latency is measured in tens of milliseconds), and observed 1 to 1.4 second latency between Murray Hill and Massachusetts, a path which ought to be in the 20-30ms range; and even with crazy corporate network routing via Illinois or Texas, 80ms or so is the maximum it should have been. To add insult to the injury, the jitter (variation of latency) was just about as high as the latency.

Bandwidth much too low, and latency and jitter much, much, too high for a blue box to be possibly viable for our project…

“We should plan the funeral of the blue box, as it will die of old age soon, a natural death”, I thought in mid-April, as I looked at the soon to die box on my desk.

It is a very good for us all that the blue boxes ran as fast is they did: they were really fast for when they were designed. If the blue boxes had been half as fast, the criminal mastermind Moriarty would almost certainly have escaped detection entirely yet again. Moriarty’s assassin would have slipped though our fingers, and no trace of Moriarty’s presence or plan would have remained.

One last test I performed in April, before my blue box’s death by electrocution (by lightning) in June set me on Moriarty’s trail. The blue box had been poisoned, but with what? and by whom? What was this test that exposed the crime, you ask?  And what was the crime? And what is Moriarty’s master plan? Where will you find his henchmen?

And who is being robbed in stealth?

You are, dear reader,….  And you can help foil the mastermind Moriarty’s plan…

To be continued…

Interesting puzzle, surprising results

I don’t know about you, but there are times I enjoy puzzles, particularly on vacation.  I’ve spent hours assembling jigsaw puzzles at vacation places from sets, sometimes complete, sometimes incomplete, and sometimes puzzles that were put away in the wrong box.

Question and Exclamations!

Jigsaw puzzles are particularly interesting and challenging when you don’t really know the picture you are trying to assemble, and even more so if you find one or a few pieces of it yourself.  It helps if you know some of the pieces you are assembling pretty well even if you don’t know the picture.

Those of you who have suffered with me through this very occasional blog know that I’ve been having networking trouble. Over the last few months, a much more interesting and much, much larger picture has been unfolding, and in ways that to me are very unexpected, disturbing, and affecting all of us (without us realizing it).  It even explains much about the network problems I’ve blogged about in the past, to my surprise, though the explanation is very different than I would have understood then.

So I’ll guide you through assembling this puzzle, as more of the pieces fall into place and I can describe the picture better over the coming months.  You may be able to help find pieces and assemble the picture as well, as I’ll note in coming occasional installments, as much is not yet known about exactly what picture we’ll be assembling.

More later, maybe quite a bit later.

Comcast and debugging….

For the last number of months (at least three months), I’ve been suffering with poor Internet service (intermittent high packet loss rates).  When my service has worked, it has worked pretty well, but minutes to an hour or so would go by when I’d see high loss rates (30-60%) and suffer accordingly.

Grrrrrr…

I need to give this story a bit of context; I won’t bore you with the shaggy dog story (though somewhat  interesting) of getting my cable Internet service to actually work after moving into this house.  To skip this history, it suffices to say that Comcast installed a new cable to the house, but the contractor they hired goofed and pulled the wrong cable for the length of the run, so I was warned that I might have trouble sometime in the future as a result of the cable not meeting their engineering standards.  Some sort of amplifier was installed to try to paper over the problem.  This was done with my permission, as I didn’t want my lawn torn up again, nor to waste Comcast’s money if it wasn’t really necessary.

This phase of the story started last summer, when we were struck by lightning.  A bunch of my gear in my house was damaged, including the Comcast provided cable modem, my router box, one of my ethernet switches, a couple ethernet cards in several machines, the irrigation system, and so on.  The cable modem was the most fun: its power supply literally blew up.

So off to the store I went, and over a period of few weeks, I got everything working again as time permitted and additional damage was discovered.

So far so good.  Almost all of my networking gear (with the exception of a single switch) was new at the end of this process.

But as I had had to go grab a little commercial firewall/router box for my point of presence (I want a system I can just unplug and replug in the case of problems when I’m traveling, so have avoided putting a little Linux box to do this, and to save the planet some joules…), and had no working Internet service, I had little time to see what was out there and just grabbed the first router box (a Linksys BEFSR41)  I could find at the store I went to.

It turned out to lack a feature I want: to disable the network for particular mac addresses at particular times of day (we got very tired of arguing with our sleep deprived kids to go to bed, etc…), so I went on line and ordered the follow-on to the dlink box I had originally had that had that feature, to replace the one I bought in my emergency.  I installed it, and all seemed well (except dlink had removed the other feature I liked: the ability to configure alternate name servers: at times I’ve been unimpressed by theirs….).

Somewhat later I noticed that our network had become intermittent; most of the time, it would work great, but other times, it would become unusable, possibly related to the amount of traffic we were generating, but this was never clear.

So I started the service call cycle.

After some number of pokes over a couple month period, Comcast noticed they had a seriously unhappy customer (and one who generally knows what he’s talking about) and escalated the issue to the supervisor in charge of network problems for my town and several adjacent towns.  After several more conversations and visits by him and technicians, we were coming to the conclusion either the cable from our house or inside the house might be at fault, and he had verified what I had been told after the cable had been installed that it was the incorrect grade of cable given the run.  Replacing the cable to the would be both expensive for Comcast, do another number on my lawn, and have to wait until the ground thaws, so both he and I were not happy campers.

But he had asked one of his technicians to monitor my cable modem, and after at least some tweaking, Comcast was seeing no further problems from their end….

Strange.  I thought about a scenario where the cable modem box might do monitoring traffic at higher priority than customer’s traffic, and still smelled a rat. But on the request of the nice Comcast supervisor, I repeated earlier experiments again, this time.  I also noted that running a traceroute while pinging the cable modem could induce a high packet loss rate just to the cable modem.  Ahah!, I could finally induce the problem more quickly, and do so before the packets going over the suspect cable.  An afternoon of heavy debugging, and it was clear the problem was in the ethernet between the cablemodem and the router; either could be at fault).

So I switched back to the Linksys BEFSR41 router I bought after the lightning strike.

End of high packet loss problem, and the culprit appeared to be the DLink EBR 2310, except…

An hour or so after resetting all the network gear (mine, and the cable modem), the ping to the cable modem would mysteriously stop without warning, which meant my traceroute test could not be run.  But over a nearly two week period, we saw no other network trouble.

At the suggestion of my helpful, Comcast service  supervisor, I tried resetting my router, and not resetting the rest of my network gear; the pings returned immediately, and the cycle repeated.  One last call to him and he suggested that since the cablemodem was so inherently dumb, I should go looking for trouble in the router as he said the cable modem has no filtering features.

Turns out he was right; there is a “Security” feature in the BEFSR41:  Block Anonymous Internet Requests, that was “on” for some reason; quite bizarrely, if on, it seems to have accounted for my returning ICMP ping responses evaporating after a while.  I would have thought it would have been all or nothing.

So at this point, I’m going to declare success, and my thanks to Comcast.

However, there were a couple of other things that I learned that seems unfortunate:

• I was told inside of Comcast’s network, they deliberately throw away ping/traceroute sorts of information, and that there was no place I could ping to help test the cable down my hill.  This made it MUCH harder to debug: ideally, there should be some way I could ping some place inside of Comcast’s network (ideally, very close to my local cable end); then, by noticing no difference in the packet loss problem going down the cable down the hill versus just pinging the cable modem, I would not have remained stuck on the possibility/probability of it being the cable itself  (which I certainly had reason to suspect, given the history and knowledge that the cable itself is not up to Comcast’s engineering standards).
• The Comcast technician told me they are not allowed to ping beyond their cable modem to to regulatory restrictions (even though I wanted them to); this would have made it much easier for them to help me determine it was on my side of the cable modem (by Comcast being able to ping my router box).  But given the “feature” of the Linksys, it isn’t clear this would have helped.

Conclusions:

1. It seems both network engineering and regulatory regulations are making debugging yet more difficult, and I have no clue how regular people would ever get to the bottom of a situation like this…  A bit of sanity on both fronts (regulatory, and ISP network design) be an improvement.
2. it isn’t always the ISP’s fault….  My thanks to Dave Dumais of Comcast for having the patience to get to the bottom of this with me.
3. my experience with dlink gear is now pretty bad (I had trouble with a wireless device of theirs dying as well a couple years ago).  They also removed two features I wanted from the previous generation of their product (I naively bought the successor product expecting the same functionality, and didn’t even fix their help files on the new model. “Value engineering” at its worst. I think I will avoid them in the future.