Browsers and TCP revisited….

Over a decade ago, I was the editor of the HTTP spec (for both proposed and draft standard; thankfully I didn’t sign up to do the full standard work). Roy Fielding has had that headache for the full standard.

As will become clear as this blog unfolds, there is a problem that browsers are exacerbating.

One of the details in RFC 2068 and 2616 is that a single client is supposed to use no more than two TCP connections at once between a client to an a single server.  The reason for this was several fold:

  1. You can actually make a single TCP connection move data faster than many connections, if you do it properly, with less overhead, as we showed in our SIGCOMM paper (for which I’ve never seen other data showing otherwise).  Now, many people don’t understand how to do so, but we explained in that paper. For most TCP applications, doing this is very easy requiring only a slight amount of care; HTTP is quite ugly, and yo could suffer from “head of line” blocking in HTTP by having large embedded objects queued over one or two TCP connections. You have to use range requests, and similar hacks so that one big image doesn’t block everything else you might want to do with that server.
  2. At the time of those RFC’s, there were a lot of dial up routers out there with very little packet storage; if you ran a lot of TCP connections in parallel (and did GETs over them at the same time, like for a bunch of embedded images), it could very easily self-congest your dial up connection and get high packet loss/retransmission.  You could easily get enough packets in flight arriving on your connection that those routers could not hold the burst while dribbling them out over the dialup.

So over the last few years, I’ve noticed that various browsers have begun ignoring the HTTP spec, and didn’t think about it all that hard, as the reasons for the 2 connection limit have passed, though wondered why, given they’d already have done most of the more painful HTTP work in the past.

Here’s part of the answer I think: as broadband connections have increased,  XP’s very old TCP stack has become a serious limitation.  Over a 100ms delay path, you can’t get more than approximately 6 megabits/second out of XP, as it does not enable TCP window scaling by default (Vista, Win7, Linux and Mac all do).  To get TCP window scaling, you have to hack the XP registry, which is not for the faint of heart. So if you have a decent broadband connection (running at 20-50Mbits/second), still run Windows XP, and obey the HTTP spec, you’d lose, all because Windows XP long beyond its expiration date (but still has something like 60% share and mercifully dropping at last).

I think current Chrome is using 6 TCP connections, and Firefox seemed to be set to 15 when I went looking a week or so ago.

Unfortunately, I think this may be another puzzle piece in my puzzle that could be causing some serious headaches, so I’m going to take some data this week to find out.  Getting real data is so liberating; you start having to deal with the way things actually are rather than endure hand waving and obfuscation.  How important a puzzle piece even if I find what I expect to see in the traces won’t become clear until I also see some current HTTP traffic stats and take some measurements against today’s popular web sites: the web of 1997 is quite different from today. While it will be interesting to revisit the hack microscape web site we used for testing then, that won’t actually tell me all that much about how serious the problem may be on today’s Internet.

I’ll report back here once I have a bit of actual data…

I do have one question for those of you who have might IE 9 beta: how many connections is Microsoft using in in IE9?  I don’t have time right now to try to mess with Win7 and install IE9 myself….

24 Responses to “Browsers and TCP revisited….”

  1. danw Says:, click the “Network” tab.

    (short answer: 2)

  2. Pascal Says:

    I think that big websites may prefer having many connections, with proper load balancing the hundred of images of the page will be served by different random servers instead of having one hundred of queued requests on the same one.

  3. Ant Bryan Says:

    The httpbis spec (still a work in progress) has gotten rid of the 2 connection limit I believe.

    • gettys Says:

      It wouldn’t surprise me if it has upped the limit (I’ve been busy with my new job, and before that OLPC and not looked at HTTPbis in that area).

      As I said, until a month or so ago, my attitude had been that the historical limit of 2 connections was not really necessary anymore: that you might do better over fewer connections, (PITA though that may be due to the baroqueness of HTTP’s design), but it would not have a strong enough justification to keep the restriction at this date.

      However, I think there may be an important issue lurking that hasn’t been thought about in the discussions to date. But I’m not going to start such discussions until/unless I have good data showing the problem I think is lurking. Now off to take some data, to see if my suspicions are correct, and if so,
      how much a problem may be hidden.

  4. Anon Says:

    Perhaps a bit related: (and someone already mentioned browserscope)…

  5. Holger Freyther Says:

    Another piece is that many servers close the connection after one HTTP request forcing the clients to reopen a new connection. So in modern times if you have more connections you reduce latency with it? Sounds reasonable?

    • gettys Says:

      Not really; the TCP three way handshake needed to open a connection means another connection inevitably takes longer than continuing to use the same TCP connection (and a lot more work on the server’s part), even if we ignore slow start. You are always better off continuing to use the same connection if you can. There are highly loaded web servers which may want to/need to reclaim connections in a timely fashion to reclaim memory/sockets, etc, but the timeout on the server end being shorter than typical human interaction time for that site is, I suspect, a poor choice for both the servers and for the users under almost all circumstances.

  6. Paul Stone Says:

    Multiple requests over a single TCP stream is what HTTP pipelining is for, right? Firefox has supported this for years but unfortunately it can’t be turned on by default because of older servers and HTTP proxies that don’t support the standard correctly.

    Fortunately Mozilla are working on this problem by using a mixture of adaptive pipelining combined with a server blacklist. Unfortunately, I don’t think these patches will make it into Firefox 4.

    • gettys Says:

      Yes, there have been broken proxies. I’m surprised there are any remaining at this date, as we have known of them for more than a decade.

      I think that browsers (not just Firefox, but all of them) should take a more aggressive stance toward brokenness/non-standard conformance they detect, and complain bitterly when they see such problems. Silently inter-operating forever with brokenness does not put pressure on those responsible to fix them. I think had browsers complained when they saw such brokenness, we’d have long since been beyond this problem. I’m not saying to ignore backward inter-operation, just that it should be reported so the problems are routinely visible to people. If you don’t know of problems, they’ll never get fixed.

    • danw Says:

      Pipelining is a slightly different problem. There’s no difficulty with doing “request, (wait), response, request, (wait), response, request, (wait), response” on a single connection (assuming the server doesn’t insist on always closing the connection, as Holger notes that some do). The pipelining problem is that you’re *supposed* to be able to do “request, request, request, (wait), response, response, response”, but that tends to blow up some servers and many intermediaries.

      • gettys Says:

        As far as I know, pipelining has only been an issue on a very few, lame proxies, that were considered obsolete even in 1997. There might be more recent botched HTTP servers/proxies, but rather than spread such information, I’d like to some pointers to it.

        And yes, persistent connections without pipelining has never been an issue, as far as I know.

        • danw Says:

          I’m not up on the details of what does and doesn’t work with pipelining, but I know the browser vendors feel that it’s not quite usable on the public internet yet. covers some of the issues and has suggestions for working around them.

        • gettys Says:

          Thanks for the good reference; that’s exactly what I was looking for.

          I still think it unfortunate that browser vendors have been so lax at reporting brokenness (of all sorts) to users. I’m not saying that they should make a big deal of it; just flag misbehavior in someway slightly visible, and move on. If people don’t know of problems, they don’t get fixed, and a lot of this is is just upgrading broken proxies and other software.

  7. Stephane Loeuillet Says:

    What do you think about Google HTTP “replacement” proposal :

    Do you think it has real advantages over HTTP ?
    Could it succeed outside Chrome Browser/Android/ChromeOS and Google servers infrastructure ?

    • gettys Says:

      Well, it’s along what we tried to get people to do over a decade ago.

      I’m torn:

      1. On the one hand, it’s worth doing: I thought so when we tried to get people moving in that direction a decade ago.
      2. But I have been coming to the conclusion that the current protocols and very structure of our network protocols need rethinking from the ground up. It’s just too hard to make good progress in the current TCP/IP structure of today’s internet.

      I’m fascinated by projects like CCNx which are thinking through communications from scratch. In particular, the properties you get in that system by having authenticated content from the get-go are profound and simplifying in quite amazing ways.

      So we have a situation of “the enemy of the good is the perfect” on one hand (and something like spdy will indeed incrementally improve our current situation_, versus rethinking the whole structure of communications, which might get us to a fundamentally new and interesting place. The two are not necessarily in conflict; I just wonder if Spdy will be enough better for people to bother with it; they certainly didn’t want to over ten years ago.

      But HTTP/1.1 is already a giant kludge tower: a hack (the best we could do given the constraints) on top of HTTP/1.0, which is a hack on MIME, which is a hack on original email. If you don’t think the tower is teetering, you are blind….

      And I doubt if it could fundamentally change things everywhere the way content centric networking efforts might.

  8. Brolin Empey Says:

    Most major (GUI) Web browsers lack fundamental functionality at the application level: they implicitly set the last modified time of downloaded files to the current time instead of using the standard Last-Modified HTTP response header to preserve the last modified time. I do not want to start Yet Another™ debate over this behaviour; read the comments in

    for years of debate. However, I find it sad how implementing such fundamental functionality is so low priority for developers of major Web browsers, such as Mozilla, Microsoft, Opera Software, and Apple. I used to hire 2 programmers to finally fix this bug in Mozilla because no one else actually acted to fix it instead of only discussing it.

    • gettys Says:

      At least with an open source browser, you can go fix it if they won’t. Or better, you can join Mozilla and fix it as part of the project….

      • Brolin Empey Says:

        I actually did check out the Mozilla tree in order to try fixing the bug myself, but I decided my resources (time + money) would be used more efficiently by hiring a professional programmer than spending months or maybe even years of my free time to teach myself everything required for Mozilla development, then experimenting with the source code to fix the bug. Even if I taught myself, I would probably forget what I learned because I would not use the skills for anything else. For example, I have tried teaching myself Perl, but every time I try writing a Perl program I have to relearn Perl because usually bash is sufficient for my use, so I use and remember bash instead of Perl. In the case of Mozilla, why relearn what I taught myself about C++ years ago, for example, when C is sufficient for my own use?

  9. Scott Carey Says:

    Since HTTP pipelining requires that responses return in the same order as requests. This is a fundamental problem when building _applications_ over HTTP and not just web pages. An application often needs to mix requests that occur synchronously with user action with asynchronous background requests to the same server.

    As for web pages, If the only goal is to get all resources on a page done as fast as possible (time to last byte) then one connection with pipelining is superior.

    But that is not always the goal. User experience usually depends very heavily on the order of that content, and the browser can’t always infer what the best order is, and HTML doesn’t make it easy to specify what the order should be — and even if it did, it is now a burder on all site designers to get that order right. Concurrent connections ‘just work’ for most use cases without any effort on the part of site designers or browsers. If HTTP pipelining was a tagged queue and did not require responses in order, there would be very little justification to use more than a few connections per hostname. Load balancers could even ‘fan out’ pipeline requests to multiple servers without buffering.

    • gettys Says:

      I agree that the lack of a sequence number in HTTP to allow out of order deliver has made pipelining much less flexible.

      HTTP is a very ugly protocol, and should be replaced entirely: I thought so 13 years ago (when I was working on HTTP); but there was no interest (not that I’d do it the way we were proposing to do then now).

      The issue I have, as you’ll see, is not just the lack of pipeling; but the large number of connections browsers are now using, along with what certain web sites are doing.

  10. SPDY of the Future Might Blow Your Mind Today « Mikes Lookout | The Gilbert Family Says:

    […] Each of these connections needs a separate handshake to the server. Each of these connections occupies a slot in your ISP’s NAT table. Each of these connections needs to warm up the TCP SlowStart algorithm independently (Slow Start is how TCP learns how much data your Internet connection can handle). Eventually, the connections feed out onto the internet and on to the sites you’re visiting. Its impressive this system works very well at all, for it is certainly not a very inefficient use of TCP. Jim Gettys, one of the authors of HTTP has observed these inefficiencies and written about the effects of HTTP’s connection management with ‘bufferbloat’. […]

Leave a Reply

Fill in your details below or click an icon to log in: Logo

You are commenting using your account. Log Out /  Change )

Facebook photo

You are commenting using your Facebook account. Log Out /  Change )

Connecting to %s

%d bloggers like this: