The Internet Archive discovers and captures web pages through many different web crawls. At any given time several distinct crawls are running, some for months, and some every day or longer. View the web archive through the Wayback Machine.

Collection: Wide Crawl started March 2011

Web wide crawl with initial seedlist and crawler configuration from March 2011. This uses the new HQ software for distributed crawling by Kenji Nagahashi.

What?s in the data set:

Crawl start date: 09 March, 2011
Crawl end date: 23 December, 2011
Number of captures: 2,713,676,341
Number of unique URLs: 2,273,840,159
Number of hosts: 29,032,069

The seed list for this crawl was a list of Alexa?s top 1 million web sites, retrieved close to the crawl start date. We used Heritrix (3.1.1-SNAPSHOT) crawler software and respected robots.txt directives. The scope of the crawl was not limited except for a few manually excluded sites.

However this was a somewhat experimental crawl for us, as we were using newly minted software to feed URLs to the crawlers, and we know there were some operational issues with it. For example, in many cases we may not have crawled all of the embedded and linked objects in a page since the URLs for these resources were added into queues that quickly grew bigger than the intended size of the crawl (and therefore we never got to them). We also included repeated crawls of some Argentinian government sites, so looking at results by country will be somewhat skewed.

We have made many changes to how we do these wide crawls since this particular example, but we wanted to make the data available ?warts and all? for people to experiment with. We have also done some further analysis of the content.

If you would like access to this set of crawl data, please contact us at info at archive dot org and let us know who you are and what you?re hoping to do with it. We may not be able to say ?yes? to all requests, since we?re just figuring out whether this is a good idea, but everyone will be considered.

Vladimir Putin is considering an ambitious project to invite millions of ethnic Russians home from the outlying states of the former Soviet Union, where they have been left stranded by generations of emigration and enforced resettlement.

The Russian President is planning to tackle a shrinking population and a shortage of skilled workers by offering ethnic Russians voluntary repatriation, according to the Russian business newspaper, Vedomosti.

The newspaper said that Mr Putin has ordered officials in the presidential administration and several other state bodies to draft a programme by June 1 which might attract Russians who have been scattered by work and exile across Central Asia and the Baltic states.

Since the mid-19th century and tsarist attempts to colonise Siberia and modern-day Kazakhstan in the east, Ukraine in the south and the Baltic states of Latvia, Lithuania and Estonia, Russians have been enticed to wider reaches of the former USSR with offers of land and work.

Under Stalin, "Russification" acquired a more compulsory character, and Russians became powerful minority populations across the Soviet Union. By the time the USSR collapsed in 1991, between 20 and 25 million ethnic Russians were living outside the borders of Russia proper, many in prosperity.

But in the last 15 years, Russians abroad, especially those stranded under budding nationalist regimes in Turkmenistan and Kyrgyzstan, have suffered.

In countries such as Kazakhstan, Ukraine and Latvia, where ethnic Russians make up to a third of the population, communities have watched as the Russian language has been eroded from government and schools. Many do not have passports, only Soviet-era residence permits, and feel they will never be fully accepted as citizens.

Vedomosti said that Mr Putin's plan is based on Russia's declining population. Citing recent UN figures, the newspaper said Russia’s current population of 144 million could decline to 112 million by 2050 if the fall in birth rates continues, posing a threat to the country’s long-term economic viability.

Since the collapse of communism, the average birth rate in Russia has plunged from 2.5 million per year between 1992 and 1995 to around 500,000 a year between 2001 and 2004, Vedomosti said.

Policy analysts say that Mr Putin's project will only attract takers if it offers considerable incentives. Despite feelings of rootlessness and, in some countries, outright discrimination, many ethnic Russians still have better jobs and enjoy higher standards of living than they feel are available in Russia.

Valery Tishkov, director of the Institute of Ethnic Studies and Anthropology at the Russian Academy of Sciences, told reporters in Moscow that he thought an ethnic-based scheme to attract workers to Russia was "absurd".

"The repatriation of ethnic Russians from neighbouring countries that has been talked about recently is absurd, although the coming here of all who want to come, temporarily or to remains, is an extremely desirable thing," he said.