Safari, Chrome, and Internet Explorer all reached new final versions of recently, while Firefox and Opera pushed their own web browsers into almost-there betas. We pulled out the digital stopwatch and testing kits to see how they measured up.
If you've never seen our browser speed test series before, you should know it's unscientific but, we believe, fairly thorough. We use a millisecond timer (Rob Keir's timer, to be specific) to manually clock the distance between launching a browser and seeing it fully loaded on a home page, then do the same timer testing while waiting for multiple tabs to load. All of the timing tests are performed three times and averaged, with far-off aberrations excluded and re-tested.
We use Mozilla's Dromaeo suite to test JavaScript performance, an increasingly important measurement for webmail, mapping, and other browser-based applications. There's no completely fair JavaScript test, as each engine is built with different inherent advantages and weaknesses for certain processes, but Mozilla's suite provides transparent results from the SunSpider test Apple leans on in its marketing, the V8 test Google uses to time Chrome's laps, and Mozilla's own set of calculation timers. We're skipping Cascading Style Sheet (CSS) testing this time around, both because the maker of our previous offline test has warned that WebKit-powered browsers (Safari and Chrome, in particular) don't test accurately in it, and because it seems like less of a relevant metric than JavaScript these days.
Finally, we pull up Windows' basic Task Manager window and measure how much system memory, or RAM, each browser is using to have just a home page open, and then have nine tabs open.
As with the most recent tests, this set was performed on a clean copy of Windows XP, with 2GB of RAM and a 2.0 GHz Intel Core 2 Duo processor. In every test, the browsers cache/"temporary internet files"/history was cleaned out before use, and a locally saved version of the plain vanilla Google search page was used as a home page. Finally, all the tests were performed on a cable-connected system, rather than Wi-Fi, to eliminate as many loading variables as possible for the nine-tab tests.
Now let's open up the gates and let Safari 4, Chrome 2.0, Firefox 3.5b99 (i.e. the Preview version), Opera 10 beta, and the final edition of Internet Explorer 8 chase the little white rabbit.
Test 1: Boot-up and Page Loading—Winner: Google Chrome!
It's reassuring to your humble tester to see his benchmarks carry over pretty consistently between different testing runs, even if it's less than exciting to declare the same winners and wild cards. Chrome continues to ace the competition, and maybe even gained a little, in loading time, both "cold" (right after a reboot, not having been loaded previously) and "warm" (re-loaded after opening and running around a bit). Firefox 3.5b99 seems to have tacked on a little more cold loading time than the 3.1b2 release. That's surprising, but my testing times ran at 9.4, 10.6, and 13.4 seconds, so there's certainly room for a streak of bad luck there.
Most people don't load up eight tabs all at once with a right-click on a bookmark folder, but it's how we perform a reasonable page loading test using a variety of pages. We used every browsers' home or download page (except for Internet Explorer's page, which caused problems with our last test), the reasonably heavy Gizmodo and Lifehacker pages, the Flash-loaded YouTube page, and Google's search page. Chrome edges out Firefox, both are followed surprisingly closely by IE 8, and Safari and Opera don't sell themselves on straight-up loading speed. One gets the impression that Firefox and Internet Explorer load a lot of things at start-up to be used later, while Chrome doesn't load much at all—then again, it's not holding an extension engine or significant customization, but let's not take up this space with that long and winding road. Here are the timings:
Test 2: JavaScript—Winner: Google Chrome!
The only real surprise in this latest round of tests was that the final release of Internet Explorer 8, which has serious problems with every JavaScript test we've put it through, performed even more sluggishly and with more "This page is running a script ..." errors than with its beta versions. The number used below is likely unreliable, and comes only from the SunSpider test, but it is, sadly, in line with other IE figures. Chrome and Safari continue to lead the pack, at least on a testing suite level, while Firefox and Opera stand apart.
Test 3: Memory Use—Winner: Firefox!
For our next browser speed tests, we'll have to come up with some longer-term memory testing. Firefox is the winner when you load up eight tabs (on top of the local home page already open, for a total of nine), but over time, many a Firefox veteran has noticed an uphill climb of memory usage. That might be extensions, that might be Flash or JavaScript remains. In any case, it's still the winner on a fresh load of eight tabs, while Opera and Safari remain intriguing choices. Chrome and IE 8 break each tab into a separate process for stability and security, so you'd expect to see bigger memory commitments, but IE 8 seems to be grabbing a good deal more than in previous betas.
What did we learn from this round? That, in most cases, final versions won't have that much difference in strict performance metrics from their betas, except in at least one notable, Redmond-derived case. Chrome retains its crown as the speed king, but we really can't wait to run it through its paces after it has built out its support architecture a bit more.
What surprised or reassured you in our tests? What would you like to see tested in our next go-round? Tell us your take in the comments.