The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What?s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.

Vladimir Putin's victory in the Duma elections on Sunday was widely foreseen, but the result has been grudgingly received in the West. The official election-observing worthies wagged their fingers: there were rumours of ballot-stuffing, or of intimidation in workplaces, and they complained that the Russian media had presented an almost uniformly glowing report of the Government for weeks before the elections. No doubt there are elements of truth in this.

A Russian electorate is still rather a strange animal, a hybrid of old and new. In the old Soviet days, elections were an excuse for a party - an event in some provincial, dead place. Little old ladies turned out in great numbers, and voted fairly resolutely for the one-party candidate who was offering the tea and cakes (and music, generally heavily amplified). The same still happens, and there was also, again from Soviet times, an element of the rotten borough about the election: people will vote for the boss or his candidate, much as vast estate-owners used to do in England.

But there is also a new feature in Russian political life, the emergence of a real public opinion, and no amount of criticism will sweep that away. President Putin is popular, and from a Russian perspective, you can easily see why. Indeed, the outcome of his recent election more than slightly resembles General de Gaulle's success in 1958.

Russia, like the France of that era, is emerging from a crisis that could have been deadly. In 1958 France was torn apart by the foul Algerian war. It was attended by savagery of lesser degree but in the same class as we have seen over the past decade or so in Chechnya. The Government collapsed; de Gaulle took over; a referendum on the new constitution gave him 80 per cent of the vote, and the political parties he patronised took two thirds of it. This is not very different from the figures attached to Mr Putin's success, and even the figure for turnout - again, more than three fifths — do him some credit, because one chief feature of elections held in “newly emerging democracies” has been profound apathy.

The background to this is that Mr Putin has been doing well in defiance of a historic trend that, if continued, might have broken up Russia altogether. Nowadays all of Russia's graphs are moving up. This includes two vital matters. The first is the birth rate. Under communism, or at any rate in its last three decades, the birth rate went down and down; abortion was a gruesome epidemic, as contraception was primitive, yet women would rather go to the abortion clinics than bring children into that Soviet world. Now the birth rate is again at replacement level; a vote of confidence in the future.

There is another important pointer. Russia has lost its empire, but it is still a country with a great many minorities, chiefly Turco-Tatar peoples. It is a measure of something that they do not support separatist-nationalist parties, with or without some Islamic connection: they too have firmly followed Mr Putin. He seems, in other words, to have found some modus vivendi and somewhere like Kazan, the capital of Tatarstan, nowadays is quite a prosperous place.

It's not hard to understand President Putin's popularity. Russia is, in parts, booming, and not just because of oil prices. It is making things, and in well-run towns like Archangel there is a level of wealth that is quite new. Even a Tory MP observing the elections, angry at being cold-shouldered, interspersed his criticism of Mr Putin with the remark that some of the housing estates he had seen might have been in Surrey.

His victory also reflects a will on the part of Russians never to have to go through the horrors of the Chechen war ever again. Remember the Moscow theatre siege, and the gassing of the audience? Or the Beslan schoolchildren massacred in their own urine? That these things are in the past is to Mr Putin's credit, and so they deserve to be.

More generally, the Russians feel at last that someone is standing up for them. In the days of Boris Yeltsin, their Government seemed to be clownish — but, as Arthur Koestler remarked, “the face of a clown, seen close to, can seem sinister”. It was an era when huge amounts of corrupt money went abroad, when Russian living standards became dismal, when anyone who could, emigrated. The country was treated with scant regard by foreigners —and there has been much resentment that the British gave asylum to characters whom the Russians saw as criminal.

Worse, Russians sensed that there was an American strategy to take over much of the old Soviet Union, with bases in Central Asia, oil interests extending towards the Caspian, Poland and Georgia (where the police wear Turkish uniforms) being turned into Nato agencies. If in the lands of the old USSR, as for instance in Ukraine, governments resisted these things, there was a machine to topple them: demonstrations in the street, pro-Western non-governmental organisations brandishing this or that ostensibly impeccable cause, while profiteers waited in the wings for the cheap privatisations that would follow “democratisation”. There were the “Rose” and “Orange” revolutions, complete, in Kiev, with bearded Euro-parliamentarians skulking in tents in the main square as if they were about to be charged by Cossacks. It was farce, but rather sinister farce, and all in Russia's backyard.

The same Western diplomats and NGOs have groomed their own people in Moscow — some of them unquestionably decent and serious, but all of them, in the eyes of public opinion, agents of the West, of a potential “Cabbage Revolution”.

Vladimir Putin has saved Russia from the turmoil of Ukraine or Georgia. As with de Gaulle, he has not been popular with many journalists (of whom de Gaulle incidentally imprisoned 300). As with de Gaulle, he has proved that he can, in foreign affairs, be difficult, even a pain in the neck. But if Russians see him as the best hope, they should be understood.

Norman Stone's most recent book is World War One: A Short History